Appcelerator Blog

The Leading Resource for All Things Mobile

Arrow Web Scraping

27 Flares 27 Flares ×

Web scraping is a technique used to extract information from websites. For mobile apps, it should be considered a last resort. Instead, try to get access to the underlying data via a documented REST web service API. However, if you find that a REST or SOAP API is not available, you may need to use web scraping in order to get the data you need.

If you are going web scrape, then don’t do it in the mobile app. Use a microservices platform, like Arrow. By implementing the screen scraping in an Arrow middle tier server, when the website changes you can change your scraping algorithm without needing to publish a new version of your mobile app.

This blog post will show a simple example of using Arrow Builder to build an API that utilizes web scraping.

The basic steps are:

  1. Create an Arrow project
  2. Install the x-ray npm
  3. Create a custom API (i.e. not based on a connector or model)
  4. Follow the documentation at x-ray

Create an Arrow project

Execute the following command from the command line:

appc new

Install the x-ray npm

In your project folder, execute the following command from the command line:

sudo npm install x-ray

Create a custom API

In the project /api/ folder, create a new file. My simple example, geta.js, is shown below:

var Arrow = require(‘arrow’);
var Xray = require(‘x-ray’);
var x = Xray();

var GetA = Arrow.API.extend({
    group: ‘xrayapis’,
    path: ‘/api/geta’,
    method: ‘GET’,
    description: ‘this is an api that shows how to web scrape using x-ray npm’,
    parameters: {},
    action: function (req, resp, next) {
        x(‘’, [‘a’])(function(err, a) {
            if(err) {
                resp.send({“error”: “cannot reach url”});
            } else {
                resp.send({data: a});

module.exports = GetA;

Follow the documentation at x-ray

In my example above, I am extracting all ‘a’ tags on the web site using the following:

x('', ['a'])

When I run this API, it returns the following:

  "data": [
    "More »",
    "Web History",
    "Sign in",
    "Get Google Chrome",
    "Advanced search",
    "Language tools",
    "Advertising Programs",
    "Business Solutions",
    "About Google",


In this example, we saw how we can leverage the x-ray npm and Arrow to perform web scraping and expose the data as a mobile-optimized REST API. Furthermore, when the web site changes, you can modify your scraping logic in the Arrow API and avoid the need to re-publish a new version of the mobile app.

27 Flares Twitter 0 Facebook 0 Google+ 0 LinkedIn 27 Email -- 27 Flares ×


  1. abada

    Nice post ,

    There are best tools for web scraping and i tried them into Arrow

    • Leor Brenman

      Thanks abada.

      X-Ray is created by the author of cheerio and has cheerio as a dependency and I use request all the time for making web service calls from node. Cleaner than Node’s built in http request.

Comments are closed.

Sign up for updates!

Become a mobile leader. Take the first step to scale mobile innovation throughout your enterprise.
Get in touch
computer and tablet showing Appcelerator software
Start free, grow from there.
27 Flares Twitter 0 Facebook 0 Google+ 0 LinkedIn 27 Email -- 27 Flares ×