HTML5 Zone is brought to you in partnership with:

Kristof Degrave is a Microsoft Certified Technology Specialist ASP.NET and has a broad interest in web development. He works as a software engineer for RealDolmen, one of Belgium's biggest ICT companies, where he takes on the role of squadleader for the web squad, a group of web-enthousiasts within the Microsoft Division. He mainly focuses on HTML5 and related technologies, such as CSS3, jQuery, ... He is currently working on a library on top of the IndexedDB API called Linq2indexedDB. The goal of this library is an easy-to-use query interface with extended filter, sort and select capabilities. The library can be found on http://linq2indexeddb.codeplex.com Kristof is a DZone MVB and is not an employee of DZone and has posted 14 posts at DZone. You can read more from them at their website. View Full User Profile

Cross-browser Testing Using BrowserStack

09.30.2013
| 15534 views |
  • submit to reddit
 

Introduction

When building mobile web applications, the goal is to support as many browsers and mobile devices as possible. To achieve this goal, testing all these environments will be necessary. But what are the options to do so? You could go to the store and buy all the devices you would like to support, but that would be pretty expensive. Another option would be to create a virtual machine for every supported platform and install the supported browsers, and for mobile devices, you could install the corresponding emulator. But what if I could save you from all the work of finding the correct emulators, installing them, setting up virtual machines and everything else? Then let me introduce you to a third option: Cross-browser testing platforms. Some of these products include SauceLabs, Browserling and BrowserStack. My personal favorite is BrowserStack.

BrowserStack is a cross-browser testing platform in the cloud. The reason why I chose this platform is because it provides me a JavaScript Testing service that integrates nicely with most of the large JavaScript Testing frameworks. Besides that, it provides a free license for all open-source JavaScript frameworks and libraries. A second reason is the fact that major open-source web development leaders like John Resig (jQuery founder), Chris Coyier (CSS Tricks) and Paul Irish (HTML5 Boilerplate) embraced it, and are using it to test their own projects. Last is the fact that I’m a developer with a .Net background, and BrowserStack provides nice integration into Visual Studio IDE and is recommended in the modern.ie site.

I also want to note upfront that I’m not involved at all with the BrowserStack project. The following article is written having only the free trial account, and an open-source license for the JavaScript Testing. Alternatives to BrowserStack are:

  • Browserling
    • Live testing (without plug-in, uses the canvas element)
    • Only desktop browsers
  • Browsertap
    • Live testing
    • In alpha (free for now, but doesn’t support mobile devices yet)

BrowserStack

BrowserStack provides a web-based, live-testing platform to test all your web applications. It has a large set of virtual machines already configured for every possible platform and browser combination. For mobile devices, they have set up their virtual machines with the official mobile emulators. A complete list with all the possible combinations can be found here.

Not only is the browser pre-installed on these virtual machines, but there are also plugins needed for Rich Internet Applications (like Java, Flash and Silverlight), and developer tools, so you can start debugging right away. A complete list of all installed developer tools can be found here.

With these virtual machines, BrowserStack provides three ways of testing:

  • Screenshots
  • Live testing
  • JavaScript testing

Screenshots

The first method of testing BrowserStack provides is by taking screenshots. These screenshots are made after a configurable amount of time and come in handy if you only want to test the design of your website, and not its functionality. The only thing you need to do is provide the location of your web application and select all the browsers and platforms you want to test.

When you have a paid account, you can go even further. Besides choosing the platform and browsers, you can set the timeout when the screenshots should be made, select the resolution of the platform, and for mobile devices, choose the orientation of the device (portrait or landscape). If you don’t like waiting, you can also choose to receive an e-mail notification when all screenshots are made. Additionally, all screenshots you ever make will stay available for the lifetime of the account, or for one month with the free plan.

Responsive

Using the responsive screenshot service, you can test the web application with actual device sizes. As the screenshot service generates a batch of screenshots for which you can choose the browsers and platforms, the responsive screenshot service gives you a predefined list of devices. The difference is that you can see the web application as you would see it on the screen of your mobile device in the orientation you want. Another difference is that you will only see the visible part of your web application. The screenshot service provides you with a screenshot of the entire page, not limited to the size of the viewport (as far as is technically possible).

With the screenshot service, all screenshots made will be saved in your screenshot history. The only disadvantage of the responsive screenshot service is that you can only view them at the actual device resolution without the device skin.

Automation

With the paid plans, there is also the ability to automate testing using the Screenshot API. The Screenshot API is a small HTTP-based API, which you can call to start and monitor your tests. In every call, you need to authenticate, which is simply done using your username and password within the HTTP request.

The Screenshots API exposes the following methods:

  • GET (http://www.browserstack.com/screenshots/browser.json) 
    • Retrieves all available browser and platform combinations that can be addressed
    • Returns a JSON file containing an array of all available browsers and platforms with the following information:
      • os (e.g., Windows)
      • os_version (e.g., XP)
      • browser (e.g., Internet Explorer)
      • browser_version (e.g., 8, or null if the version is not applicable)
      • device (e.g., null if the device is a desktop, filled in with the mobile device when it’s a mobile device.)
  • POST (http://www.browserstack.com/screenshots) 
    • Starts a new job to create screenshots
    • POST data must be in JSON format with the following parameters:
      • url (String, the location of the webpage you want to test)
      • mac_res (String, optional, the resolution of Mac machines: 1024x768, 1280x960, 1280x1024, 1600x1200, 1920x1080)
      • win_res (String, optional, the resolution of Windows machines: 1024x768, 1280x1024)
      • quality (String, optional, quality of the screenshots, Original or Compressed)
      • tunnel (String, optional, uses local testing, true or false)
      • wait_time (Integer, optional, the timeout when the screenshot is made in seconds: 2, 5, 10, 15, 20)
      • callback_url (String, optional, a valid URL where the data will be posted once the job is done. The list will return a list of all screenshots generated. The posted data is in the same structure of the response of this request.)
      • orientation (String, optional, the orientation of the device: portrait or landscape)
      • browsers (Array of objects, List of all browser and platform combinations you want to test)
        • browser (String, the targeted browser)
        • browser_version (String, optional, the targeted browser version)
        • os (String, the targeted operating system)
        • os_version (String, the targeted operating system version)
        • device (String, optional, the targeted device)
    • Response is a JSON object containing the same information as posted with some extra properties:
      • Job_id (unique ID for the job, can be used to retrieve status information and the result of the job.)
      • Browsers (The following properties are added to the objects in the browser array)
        • Id (unique ID for the screenshot)
        • State (state of the screenshot: pending, done or failed)
        • thumb_url (URL of the thumbnail of the screenshot)
        • Image_url (URL of the screenshot)
        • Created_at (The time the screenshot was created)
  • GET (http://www.browserstack.com/screenshots/<jobid>.json)  
    • Gets status information about the job with the given ID.
    • Response is the same as the result of the job request.

More information and language bindings can be found here.

Live

The second provided way of testing is live testing. With this type of testing, a chosen virtual machine is started and navigates the browser to the provided URL. In the browser, you gain access to the virtual machine using a Flash plug-in. Once the virtual machine is fully loaded, you can start testing your web application. You can navigate through the entire application and use development tools to debug your application.

Responsive

At the moment, it isn’t possible to have responsive live testing, but this will change in the future.

Automation

The Live testing service also provides test automation. This can be done, for now, only by using Selenium. Selenium is an open-source tool that is used to automate browser tasks, such as testing, as well as boring, web-based administration tasks. Browser tasks can be automated by making Selenium a native part of the browser, which most of the large vendor browsers support, or are taking steps toward supporting. More information about selenium can be found here and, for integration with BrowserStack, here.

JavaScript Testing

The last way of testing your web application is JavaScript testing. Using this HTTP-based API, you can cross-browser test your JavaScript files across over 200 browsers and mobile devices. You can also run tests by integrating your favorite testing framework, like QUnit or Jasmine, and by using tools such as Yeti, TestSwarm (a tool used by jQuery) and JS-Test Drive to automate the tests.

Good news for all open-source developers of JavaScript frameworks and libraries: BrowserStack makes automated JavaScript testing free for open-source projects. Just send them an e-mail regarding your project, and they will give you access to this API.

Every call you make to the JavaScript testing API needs to be authenticated, which is simply done using your username and password within the HTTP request. The JavaScript testing API exposes the following methods:

  • GET (http://api.browserstack.com/3/browsers)
    • Provides a list with all available browsers
    • Returns a JSON object with the following information:
      • Operating System (Windows, OS X, iOS ,etc.)
        • Operating System version (7, XP, Snow Leopard, Lion, 4.0, 6.0, etc.)
          • Browser (IE, Firefox, Chrome, etc.)
          • browser_version (8.0, 2.0, 23.0, etc.)
    • Example:
{
  'Windows':
    {
      '8': 
        [
            { browser: 'ie', browser_version: 10.0 },
            { browser: 'firefox', browser_version: 3.0 },
            { browser: 'chrome', browser_version: 23.0 }
        ],
       'XP': 
        [
            { browser: 'ie', browser_version: 8.0 }
        ],
    }
   'ios':
    {
      '4.0':
        [
            { browser: "Mobile Safari", browser_version: null, devices: ["iPhone 4"] }
        ],
      '6.0':
        [
            { browser: "Mobile Safari", _version: null, devices: ["iPhone 4S (6.0)", "iPhone 5", "iPad 3rd (6.0)"] }
        ]
    },
}
    • When you add ?flat=true to the query string you get the result back in a flat structure (an object for every browser/platform combination)
      • os (e.g., Windows)
      • os_version (e.g., XP)
      • browser (e.g., Internet Explorer)
      • browser_version (e.g., 8 or null if the version is not applicable)
      • device (e.g., null if the device is a desktop, or filled in with the mobile device when it’s a mobile device.)
  • POST (http://api.browserstack.com/3/worker)
    • Creates a new worker to run the JavaScript code. You can start multiple works at once and these will be queued and run when their turn comes.
    • POST data must be in JSON format with the following parameters:
      • url (String, the location of the webpage you want to test)
      • browser (String, the targeted browser)
      • browser_version (String, optional, the targeted browser version)
      • os (String, the targeted operating system)
      • os_version (String, the targeted operating system version)
      • device (String, optional, the targeted device)
      • timeout (Integer, optional, number of seconds before the worker gets terminated. Default 300 and maximum 3600
    • Response is a JSON object containing the id of the worker.
  • DELETE (http://api.browserstack.com/3/worker/)
    • Terminates a worker
  • GET (http://api.browserstack.com/3/worker/)
    • Retrieves the status of the worker
    • When empty, the worker is terminated; otherwise you receive a JSON object with the following information:
      • Status (The status of the worker: running or queue)
      • os (e.g., Windows)
      • os_version (e.g., XP)
      • browser (e.g., Internet Explorer)
      • browser_version (e.g., 8, or null if the version is not applicable)
      • device (e.g., null if the device is a desktop, filled in with mobile device when it’s a mobile device.)
  • GET (http://api.browserstack.com/3/workers)
    • Retrieves the status of all the workers
    • Returns an array of status objects like the worker status
  • GET (http://api.browserstack.com/3/status)
    • Returns information about the status of your API
    • Returns a JSON object with the following information or a message when you run out of API time
      • used_time (The time the API has already used in seconds)
      • total_available_time (The time that’s available for your account in a month)
      • running_window_sessions (The number of concurrent Windows sessions running)
      • windows_sessions_limit (The maximum number of concurrent Windows sessions)
      • running_mac_sessions (The number of concurrent Mac sessions running)
      • mac_sessions_limit (The maximum number of concurrent Mac sessions)

Local Testing

Not only public web applications can be tested; by using the local testing functionality, you can make a tunnel to your local machine to locally test your web application. More information about setting up local testing can be found here.

Integrations

BrowserStack also provides integrations with development tools. For Microsoft developers, there is a Visual Studio and Web Matrix plug-in available, making cross-browser testing even easier. Scott Hanselman wrote a great blog post about how he created the plug-in.

Conclusion

BrowserStack is a great service for cross-browser testing. It gives you access to more than 200 browsers and mobile devices already configured and ready to use. Using the screenshot service, you can check your design on multiple browsers and platforms in a matter of seconds. Using live testing, you can test your functionality, and you can use the JavaScript testing service to test all your JavaScript logic using your favorite JavaScript testing framework.

This article was commissioned by Intel and the Tizen Mobile Web Development team.

Published at DZone with permission of Kristof Degrave, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)