Agile Zone is brought to you in partnership with:

Matt is the Group Leader of Research Application Development in the Research Informatics Division of Information Sciences at St. Jude Children's Research Hospital in Memphis, Tennessee. Matt has been developing and supporting enterprise Java applications in support of life sciences research for St. Jude since 2001. Matt is a committer to multiple open source projects and is the founding member of the Memphis/Mid-South Java User Group. Matt is also a regular speaker on the No Fluff Just Stuff symposium series tour (as well as other major conferences), and his articles have appeared in GroovyMag and NFJS the Magazine. His current areas of interest include lean/agile software development, modularity and OSGi, mobile application development (iPhone/iPad/Android), web development (HTML5, etc.), and Groovy/Grails. Matt has posted 44 posts at DZone. You can read more from them at their website. View Full User Profile

To Selenese or not to Test? That Seems to be the Question.

  • submit to reddit
We had a very good turnout for the Automated Browser Testing Poll this week. Before I get into my musings about what happened, here's a quick summary of the results. With 304 total votes counted:

- 58% of respondents are using some variant of Selenium, with Selenium 2/WebDriver garnering the highest percentage of any testing tool. I also included in this percentage those that are using a Selenium wrapper framework such as Tellurium, Geb, or Cucumber.
- 31% of respondents are simply not doing automated browser testing
- 11% of respondents are using "something else," and every category received at least two votes.

These three summary points lead me to three major conclusions about the state of the automated browser testing world.

Selenium is still "King"

Three out of five developers are using Selenium in some form or fashion, representing a pretty solid majority. Selenium 2 uptake is moving quickly as well, with the majority of raw Selenium users (57%) already migrating to Selenium 2 while it is still at its alpha release stage. I can think of several reasons why this may be the case:

  • Selenium is a mature, battle-tested framework
  • It has a great deal of commercial backing (ThoughtWorks, Sauce Labs, etc.)
  • Google contributes to and uses it
  • It can target a huge number of browsers (Firefox, IE, Safari, Chrome, iPhone, Android, Blackberry, etc..)
  • Tests can be written in nearly any target language

We still have a lot of work to do...

All that gushing over Selenium aside, no tool garnered a higher percentage of the vote (31%) than "I don't do automated browser testing!" What's amazing to me is that these folks still took the time to click on the poll and then vote. I honestly wouldn't have expected them to be reading this stuff! At any rate, I'm still curious as to why this is the case. No one posted up any relevant comments, so I'm left to speculate.

  • No time? I would argue that you don't have time not to automate your browser tests. If you're building an application of any size and you're doing incremental releases, the only professional and responsible thing to do is to test every single feature of the application on every supported browser platform every time you release the software. I don't know about you, but I don't know of anyone that has enough folks sitting around with the time to do that *manually.* Add to this the fact that you really ought to be testing everything following any change to the source code, and you really can't get it done. But that's the only way you're going to minimize the time to discovery (and thus the impact) of bugs in your system.
  • Too hard? I would argue that it's never been easier to write automated browser tests. Pick your programming language, pick your style, pick your browser: something has support. Selenium 1 and Watir have freely available recording tools that reduce the barrier to entry. If you're doing automated unit testing, you can do automated browser testing.

The ecosystem is only getting bigger!

It seems like new automated browser testing tools/frameworks are popping up every day, and as fast as they are appearing, folks are starting to use them. One very recent arrival, Geb, is a Groovy wrapper around the WebDriver API that uses a very "jQuery-esque" syntax to interact with the DOM. Coupled with Spock, the popular BDD/unit testing framework for Groovy, one can get very close to the idea of executable specifications for web applications. I think the combination of Geb and Spock is quite compelling, and so I've decided to build a session on it for the Rich Web Experience. What was incredibly surprising to me is that Geb garnered NINE VOTES in our poll.

So there you have it. Thanks to everyone that voted!
Published at DZone with permission of its author, Matt Stine.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Sadsds Dsadsadsa replied on Mon, 2010/10/25 - 3:27am

I agree that automated browser testing is needed, but you have to be careful when you make statements like:

 "If you're building an application of any size and you're doing incremental releases, the only professional and responsible thing to do is to test every single feature of the application on every supported browser platform every time you release the software."

 I've worked with companies with massive test suites of automated tests and they invariably suffer from 2 problems:

1. Maintenance of this suite of tests is a huge burden and time that could be spent on business functionality.

2. The time taken to run the tests can soon be prohibitive so teams run them overnight. This means the feedback cycle is extended which is not good.

So I think that blindly testing all your functionality through the GUI is a bad idea. For example, imagine a calculator app. You could test addition/subtraction/multiplication/etc through thousands and thousands of tests through the GUI. Or you could make sure each button ends up sending a character to the backend and back (which is what the GUI does) but have the actual calculation logic tests in unit tests.

You also didn't mention that these browser tests are fragile by their very nature. Changes to the UI can lead to a lot of tests breaking.

The use of the correct patterns with these tools is also important. The page object pattern is a good one.

I recommend looking at the work of Gojko Adzic(, he has a book in the pipeline were he addresses some of these issues.


Esko Luontola replied on Wed, 2010/10/27 - 6:13am

The questionare was missing one option: "I don't make web applications." So it's not possible to say that how many of the 31% are not doing automated web testing because it is too hard or time consuming, and how many do not do it because they have no need for it.

Also none of this says how thoroughly the application is tested below the web UI. It could be that the application is fully automatically tested that way and the UI is only a thin wrapper above the application, without things which could break, so they have decided it to not be worth testing the UI. And if the application is tested only through the UI, then that has its own set of problems.

Erik Pragt replied on Sun, 2010/11/21 - 2:27pm

Hi Matt,

I agree that (automated) testing is needed in any serious "Enterprise" application, since testing things manually is just too slow and therefor to expensive. I, however, do not agree with the statement that the browser tests have to be automated. It's a tradeoff. Having complex logic? Focus more on the business logic tests. Need to support 28 different browsers? Automated browser test is your friend here!

But 'it depends' (needed to say that) on the system your building. IMHE, user interfaces are easy to break, and easy to fix. No database schemas need to be updated, no integration with other systems is needed, just fix some WPF, Javascript, HTML things, and you're done. However, when creating complex business logic, and your calculation works only 'most of the time', you might have a more serious issue, and an expensive one to fix. So my recommendation is to focus on that, before automating the UI.

(And, yes, I'm biased, but Fitnesse might just be a tool to support automating these complex business tests.)

Danil Sviazev replied on Tue, 2010/11/23 - 9:02am


Selenium 2.0 is actually pretty good, but missing some required features. Also we found one more tool which seems pretty usefull in web testing (also based on selenium 2.0)

It's called Screenster, still in active developement, but the main idea is that it uses screenshots of web page for validation, comparing them to base line, so you don't need to spend time on assert rules, which might still pass invalid rendered page.

David Wagstaff replied on Fri, 2011/01/14 - 7:22pm

I interpret the 31% as confessing, i.e. they KNOW they should. You then speculate and destroy them as the two strawmen of no time and too hard.

Where's the empathy? You describe those not following your advice with 4 words? You'd put more thought into a single test than you did in your speculation about non-testers. Until you understand where they are coming from, don't preach.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.