Extracting a React-Native App Out of an Old, Clunky Website

Introduction

There is a website that my sailing club provides which must have been written ten or fifteen years ago that is miserable to use. It's especially bad on my iPhone. The purpose of this website is to reserve sailboats, and sometimes I want to whip out my phone and reserve a boat right away. But then I think about panning around the page and zooming in and out and trying to remember my password (that it won't save!) and trying to deal with the twitchy and buggy JavaScript controls. And I think: "Oh, I'll wait until I get home."

But that's a hassle. And anyway – why do we accept that the performance on a brand new mobile device is inferior to a ten-year-old windows PC running Internet Explorer version 6? I want a dedicated app on my phone which just deals with showing and making reservations. Let me just tap on an app icon, see my current reservations and five or six taps later, make a new one without ever touching a browser.

If there is an app like this in your life, and if you are a self-respecting JavaScript developer, there is no reason you can't have the experience you want. There is a new breed of JavaScript frameworks (such as React Native, NativeScript, and Tabris.js) which allow you to sit down and take that lousy web app and make a simple, attractive, and easy-to-use native app that provides just the interface that you want.

This blog post is going to focus on using React Native to accomplish this task, but it could have been done just as easily using one of these other frameworks.

But isn't what you're suggesting in this article stupid?

Yes, in some ways it is. Interfacing with a web app by pretending to be a client is somewhat horrible. If you own or manage the website in question, you should create a jsonapi interface for it instead of doing what I suggest in this article.

But what if you don't own the website? Or what if the code is so gross and horrible that you don't even want to think about touching it? In that case, this is a perfectly reasonable approach, especially if that website isn't going to change.

Pre-requisites

This article is oriented towards building an iOS app using React Native. So that means that you're going to have to run XCode, which means you're going to have to be using a Mac. If you're not using a Mac, you can either develop an Android app or use [Tabris.js]. Anyway, for the rest of the article, I'm just going to go ahead and assume you've got a Mac because that's easier for me.

First, you're going to want to install the following tools on your Mac:

- homebrew - for managing linux-like packages
- curl (brew install curl) (once you've installed homebrew) - for testing interaction with websites
- betwixt - for validating that your native app is working
- Google Chrome - for capturing traffic to the website you want to reverse-engineer

Reverse-engineering the website using Chrome and cURL

1. Capture an interaction with the website

The first step in building a native app replacement is to figure out exactly how the web app works. For this, you're going to want to start with the Chrome developer tools. If you're logged in to the web app, log out of it and begin by looking at the login screen. Here's an example of the app I'm trying to deal with shown here:

Open the developer tools and log in to the app. Once you've made a successful login attempt, go to the developer tools and find the HTTP request that was responsible for you logging in successfully to the site (see screen-cap)



The successful login request will likely be a `POST` and will likely contain some body information that has your username and password. In my case, the login attempt was the very first call to the website to the URL: `URL:http://www.tradewindssailing.com/wsdl/Logon-action.php` I can see from the Chrome developer tools that the post contains my clear-text username and password (over http, this is a pretty bad practice, but we already know this website is old and lousy, and poor security shouldn't surprise us at this point).

2. Convert the website interaction to cURL

Next what I want to do is to capture this login attempt as a [cURL] command. Why? Because `cURL` is a simple and succinct way to programmatically interact with a website. `cURL` allows you to capture an interaction with a website and quickly replay it, allowing you to verify that you have all the information you need to produce the same kind of request from your React-Native app.

Luckily Chrome makes this amazingly easy. All you do is right-click on the request which contains the interaction with the website and select Copy as cURL. Then paste the resulting data into the nearest text-editor. When I did this, this is what I got:

curl 'http://www.tradewindssailing.com/wsdl/Logon-action.php' -H 'Cookie: userid=8637900; username=Peter+Banka; _gat=1; PHPSESSID=01cjast5kev4k98oqamhejrhg6; _ga=GA1.2.115674046.1452703527' -H 'Origin: http://www.tradewindssailing.com' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'Referer: http://www.tradewindssailing.com/wsdl/Logon.php' -H 'Connection: keep-alive' --data 'userid=8637900&pwd=4zBDkV1Agi&Submit=Submit' --compressed

3. Trying out the curl command

You should feel free to experiment with your curl command on the command-line. Open up a terminal. Paste that sucker in. See what it does!

NOTE Use your own curl commands that you got from **your own** website. Don't try to use the examples that I provided above. They contain bogus credentials, and they will not work for you. While they are based on a real use-case, I don't want to publish my personal credentials for an online website on the Internet, so I have modified them. They are provided as examples only!

When I paste that thing into a terminal, I get the following:

<html></html>

Which is somewhat disappointing :( . Did it work? How do we know?

In order to get better results out of curl, you're usually going to want to focus on the headers and not the actual response html. To do this, add the -v flag to the end of your curl command. When you do this, you'll get something like the following:

*   Trying 198.171.78.101...
* Connected to www.tradewindssailing.com (198.171.78.101) port 80 (#0)
> POST /wsdl/Logon-action.php HTTP/1.1
> Host: www.tradewindssailing.com
> Cookie: userid=8637900; username=Peter+Banka; _gat=1; PHPSESSID=01cjast5kev4k98oqamhejrhg6; _ga=GA1.2.115674046.1452703527
> Origin: http://www.tradewindssailing.com
> Accept-Encoding: gzip, deflate
> Accept-Language: en-US,en;q=0.8
> Upgrade-Insecure-Requests: 1
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36
> Content-Type: application/x-www-form-urlencoded
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> Cache-Control: max-age=0
> Referer: http://www.tradewindssailing.com/wsdl/Logon.php
> Connection: keep-alive
> Content-Length: 43
>
* upload completely sent off: 43 out of 43 bytes
< HTTP/1.1 302 Moved Temporarily
< Date: Tue, 23 Feb 2016 04:32:17 GMT
< Server: Apache
< X-Powered-By: PHP/5.4.23
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Set-Cookie: userid=8637900; expires=Mon, 23-May-2016 04:32:17 GMT
< Set-Cookie: username=Peter+Banka; expires=Mon, 23-May-2016 04:32:17 GMT
< Location: Reservations.php
< Keep-Alive: timeout=5, max=100
< Connection: Keep-Alive
< Transfer-Encoding: chunked
< Content-Type: text/html
<

There are a few clues here that this command resulted in a successful login attempt:

• the line HTTP/1.1 302 Moved Temporarily tells us that the website wants us to go to a new location, and
• the line < Location: Reservations.php tells us that it wants us to go look at the reservations page. If the website had told us to go back to the `Logon.php` location, that would be a pretty-good indication that our attempt did *not* work.
• The line < Set-Cookie: userid=8637900; expires=Mon, 23-May-2016 04:32:17 GMT tells us that the website is trying to set cookies on our browser. This typically means that the website trusts us and wants our browser to identify itself as our proper user self in the future.

These all indicate that the website probably accepts the request that we made and that we are in.

4. Experiment with the curl command

When you look at that command that you pasted in, there's a lot of stuff there, and probably much of it you don't need. We should pull it apart and understand what parts of it are absolutely necessary and what parts can be dispensed with.

If you refer to the original curl command in the previous section, you'll note that for each `-H` flag, curl is setting a header. Let's experiment with getting rid of these and see which ones are really necessary.

It turns out that – at least in my case – I can pare this command down to the following:

curl 'http://www.tradewindssailing.com/wsdl/Logon-action.php' --data 'userid=8637900&pwd=4zBDkV1Agi&Submit=Submit' -v

This is good, because we're going to be converting this call to a fetch call later, and the less we have to convert, the better.

Converting the cURL calls to fetch

So now we have a simple call to a website. I know that my userid is 8637900, and I know that my password is 4zBDkV1Agi, so I feel confident that I can create a curl request for anyone's username and password. Now I need to convert this request to something that can be used in React Native.

If you'd like to see an example of how this curl call is converted to React-Native, it can be seen here:

Essentially it boils down to the following:

funciton logIn(userData) {
  const params = {
    method: 'POST',
    body: `userid=${userData.username}&pwd=${userData.password}&Submit=Submit`,
  }
  return fetch('http://www.tradewindssailing.com/wsdl/Logon-action.php', params)
    .then(logonResponse => {
      if (logonResponse.url === 'http://www.tradewindssailing.com/wsdl/Logon.php') {
        console.log('Bad username or password'})
      } else {
        console.log('You are totally logged in right now')
      }
      return fetch('http://www.tradewindssailing.com/wsdl/Reservations.php')
    .catch(error => {
      console.log('something went terribly wrong');
    })

You're seeing here the new fetch api in action. It's built into React Native and it works great!

Dealing with cookies

You probably already understand cookies and web authentication, but for those of you who do not, here is a quick primer: when the web server wants to keep track of a user, the server sends an instruction for the browser to set a cookie. The browser is then required to send that cookie whenever it sends a request to the server until the server tells it to stop.

Often a server will send an instruction to the browser to keep track of a session-id. So, when curl tells you that the response from the web server contains a line like the following:

< Set-Cookie: PHPSESSID=taamci2tit4j4thoaq8sqnvl24; path=/

that tells you that your browser needs to set a session-id cookie which will identify your session with that server from now on. If you log in to the server but fail to send that cookie with every request, then the server will not recognize that you have logged in and will treat any further requests from your React Native app as coming from an un-authenticated user.

When using a program like curl, you tend to expect cookies to come in through headers and to be set with headers. However, this is not how cookies work on the React-Native platform!

Instead, there is an npm module called react-native-cookies which examines fetch requests and responses and manages cookies for you.

To retrieve the cookie from the last `fetch` API call, you'll use CookieManager.getAll(). Then to ensure that `fetch` uses your new cookie for every future message to that server, you'll use CookieManager.set() as shown here:

        CookieManager.getAll((cookies, status) => {
          const newCookie = {
            name: 'PHPSESSID',
            value: cookies.PHPSESSID.value,
            domain: 'www.tradewindssailing.com',
            origin: 'www.tradewindssailing.com',
            path: '/',
            version: '1',
            expiration: '2016-05-30T12:30:00.00-05:00'
          }
          CookieManager.set(newCookie, (err, res) => {
            if (!err) {
              console.log('cookie set!')
            }
          })
        });

Debugging your React-Native app using Betwixt

Converting curl calls to fetch calls, while fairly straightforward, can still be problematic: did you get your cookies set properly? Did you get your data formatted properly in the POST? How exactly did the server respond?

Answering questions like this on the React Native platform can be difficult because you can't use the Chrome Debugger tools to capture your fetch traffic like you can when using a browser. Therefore you can be flying blind when trying to determine exactly what your web requests and responses look like that are coming from your React-Native app.

Luckily, there is a new project called [betwixt] which provides all the Chrome Developer tools for you to analyze *all* calls coming to and from your Mac. And, since your iOS simulator on your Mac is just another Mac application, you can see all network calls coming from your React Native application and see exactly how they compare to the ones that you captured in the beginning of your reverse-engineering session.

Here is an example of a betwixt session in-action, where we can see and inspect the calls being made by the fetch API from the React Native system on the iOS simulator:

The final product

When finished, you should be able to have a simple, effective, and intuitive app which runs on your phone that uses the web app as a backend. You may decide to send this out to the app store, or you may just decide that you want to use it for yourself only. Either way, you have proven that you can make something simple and useful for yourself and that you don't have to be at the mercy of lousy web apps on your phone any longer!

Getting started with MMStats in your application

I have been working on a large Python/Django based web application that has a web-server component and a job-scheduling component. We recently transitioned from "Early-Deployment" to "General-Deployment" of our application, and management instantly wanted to know how many people were logging in, how many records were being created and when.

The first solution: Logging

Logging is the first solution to this problem. We've had logs since the beginning, using Python's logging library. This is indispensable, and we'd be lost without our logs. So the first thing we tried when the boss wanted data was to perform special log entries that could be parsed out by a log-mining script. This lead to log-pollution, ugly logging code, and a significant time-delay between events occurring and getting reporting data.

The second solution: Email

Next, we fired email events when certain important events occurred. Fire an email to a distribution list whenever a traceback event is trapped. Fire another email whenever a record has invalid data. This quickly lead to bloated inboxes and very poor metrics on how many important events occurred and when. It also lead to developers getting desensitized to "normal errors."

The correct solution: mmstats

We loaded the mmstats package into our application a month ago with the intention of trying to transition to it gradually. Today I had a chance to kick the tires. This will enable us to log exactly the kind of data we want to see, and get an understanding for how this data is changing over time.

In our application, we have an automated process that reaches out to Cisco routers to make configuration changes. Each router configuration job is dispatched into its own thread, and I wanted to start gathering statistics on the amount of time each command to the router took and how many commands were issued to each router.

We have a class which encapsulates all the interaction with the router called RouterConnection:

 class RouterConnection(AbstractConnection):
   "Connection to a router"  
   def __init__(self, router_config, mongodb):  
     AbstractConnection.__init__(self, router_config)  
     self.mongodb = mongodb  
     self.connection = None  
     self.enable_prompt = "#"  
     self.disable_prompt = ">"  
     self.config_prompt = "\(config.*\)#"  
     ...   

Next, the trick is to instrument this class with mmstats, so the first step is to create another class in the same module that keeps track of the items I care about:

 class RouterStats(mmstats.MmStats):  
   wait_for_prompt_timer = mmstats.TimerField(label="wait_for_prompt")  
   try_to_read_lines_timer = mmstats.TimerField(label="try_to_read_lines")  
   line_timeout_counter = mmstats.CounterField(label="line_timeout")  
   test_counter = mmstats.CounterField(label="test_counter")  
   to_router_counter = mmstats.CounterField(label="to_router")  

There's the new monitoring class... add a line to instantiate it within the RouterConnection class...

 class RouterConnection(AbstractConnection): # pylint: disable=R0902  
   "Connection to a router"  
   def __init__(self, router_config, mongodb):  
     AbstractConnection.__init__(self, router_config)  
     self.mongodb = mongodb  
     self.connection = None  
     self.enable_prompt = "#"  
     self.disable_prompt = ">"  
     self.config_prompt = "\(config.*\)#"  
     self.stats = RouterStats(  
       label_prefix='router.stats.',  
       filename="/tmp/mmstats-%s" % router_config.name  
     ) 

Since we'll be instantiating several of these objects, I am customizing the filename based on the name of the router being connected to. Next, whenever there are items of interest in the code, use the stat collector. For example:

     while '\n' not in read_buffer:  
       try:  
         new_buffer = self.connection.read_nonblocking(timeout=MIN_LINE_TIMEOUT)  
         read_buffer += new_buffer  
       except pexpect.TIMEOUT:  
         self.stats.line_timeout_counter.inc()  

and...

     while not (found_index == exit_index or self.exit_interact):  
       try:  
         with self.stats.try_to_read_lines_timer:  
           read_lines = self._try_to_read_lines()  
       except pexpect.EOF as p_eof:  
         if eof_action:  
           eof_action()  
         else:  
           raise p_eof  

Reading the data that you've collected