Agile Zone is brought to you in partnership with:

Matt is the Group Leader of Research Application Development in the Research Informatics Division of Information Sciences at St. Jude Children's Research Hospital in Memphis, Tennessee. Matt has been developing and supporting enterprise Java applications in support of life sciences research for St. Jude since 2001. Matt is a committer to multiple open source projects and is the founding member of the Memphis/Mid-South Java User Group. Matt is also a regular speaker on the No Fluff Just Stuff symposium series tour (as well as other major conferences), and his articles have appeared in GroovyMag and NFJS the Magazine. His current areas of interest include lean/agile software development, modularity and OSGi, mobile application development (iPhone/iPad/Android), web development (HTML5, etc.), and Groovy/Grails. Matt has posted 44 posts at DZone. You can read more from them at their website. View Full User Profile

Automation for the People (because Everybody Doing It Manually Hurts)

10.04.2010
| 11515 views |
  • submit to reddit
Today I'm kicking off a new emphasis here at Agile Zone on software project automation. The astute reader will recognize the double reference to R.E.M.'s 1992 4X platinum album "Automatic for the People." That album contained their hit single "Everybody Hurts." And that's precisely what I want to talk about in this kickoff article - the pain that all of us suffer from "doing things manually" on software projects. In the coming days we'll be looking at practical ways that you can eliminate this pain from your development efforts.

To start off, I'd like to address the immediate objection that this type of article just isn't needed. I mean, anybody that's a competent software developer has this down already, right? Wrong. While I have absolutely no data to back this up (Data? We don't need no stinkin' data!), it's my firm belief that nearly every developer out there is today doing something manually that they could automate. The fact of the matter is, they just haven't taken the time to do it.

In his book The Productive Programmer, Neal Ford recounts [1] the story of needing to parse a large legacy SQL file. In order to ease their task, Neal and his coworker decided to break the file into 1,000 line chunks. They quickly decided to automate the task rather than doing it by hand. The automation task took them roughly one hour, and they estimated that completing the task manually would have taken about ten minutes. Mathematically this looks like a loss until you find out that they eventually had to complete the same task five more times, almost reclaiming all of the time spent in automation. But according to Neal, that's not the important point. Instead, he asserts:

Performing simple, repetitive tasks by hand makes you dumber, and it steals part of your concentration, which is your most productive asset.

And:

Figuring out a clever way to automate the task makes you smarter because you learn something along the way.

I've realized the value of these points many times over. Recently I was faced with the necessity of loading multiple large spreadsheets of client data into one of our applications. For various reasons, the straightforward route of loading directly into the database tables was going to be a painful one. It quite likely would have taken me more than a week to get right, and I needed to get these data into the application within twenty-four hours. Shortly before biting the bullet and tackling the task via brute force (becoming "dumber" along the way), it hit me that I had a couple of tools in my toolbox that might just work.

The Selenium framework was developed at ThoughtWorks out of the need to test an internally developed Time and Expenses application. It has rapidly grown into one of the most heavily used tools for automated cross-browser testing of modern web applications. Part of what makes Selenium so powerful is its ability to record user interaction scripts via its Selenium IDE plugin for Firefox and then export those scripts to your programming language of choice (at this writing Java, Groovy, C#, Python, Ruby, Perl, and PHP are explicitly supported). A developer can then leverage the Selenium Remote Control (RC) server to drive most modern web browsers from these exported scripts. But that isn't the best part.

Because these scripts are now written in real programming languages, you can bring all of your best development techniques and third-party libraries to bear on your problem!

My thought process was as follows:

  1. Record a Selenium script of me loading the first row of data into our application.
  2. Export that script as a GroovySeleneseTestCase (a subclass of groovy.util.GroovyTestCase).
  3. Leverage Apache POI's support (http://poi.apache.org/spreadsheet/index.html) for parsing Microsoft Excel spreadsheets to write a parser for the client data in Excel.
  4. Wrap the originally exported Selenium logic in a loop over the spreadsheet's rows, substituting the parsed values for the manually typed data.

It took me a few hours to grasp some of the eccentricities of Selenium with which I wasn't familiar, and I had to dive head first into some rather ugly XPath expressions, but eventually I had a working solution. About twenty minutes after my success, all of the data were loaded. But as in Neal's case, the story didn't end there. I've been called upon to repeat the said data load several times. Unfortunately my problem shifted around a bit, as none of the spreadsheets were identical. However, every time I was only a few minutes away from a new working solution. I could then go back to real knowledge work while my computer happily performed the mundane work.

Not only had I built a reusable automated solution that saved me vast amounts of otherwise unproductive time, I also rapidly ramped up my skill level with Selenium. Double win!

So today we've looked at an isolated, small success story that saved one developer (me) a great deal of time and made him smarter along the way. Is it possible to scale that kind of success up to the entire project team? Absolutely. And that's just what we'll be doing in the coming weeks. I'm hopeful that you'll pick up some nuggets along the way and incorporate them into your own projects. Trust me, once you automate the mundane, sweetness follows.

References

[1] Ford, Neal. The Productive Programmer. O'Reilly, July 2008.
Published at DZone with permission of its author, Matt Stine.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Xebia Labs replied on Thu, 2010/10/14 - 9:51am

Matt, we couldn’t agree more with your new series about the importance of automation! Automating any processes you can may take time to set up initially, but will pay off in the long run by making tasks easier and less time consuming. Automating deployments can save time by preventing repetitive tasks, as you mention, but this also has a residual effect. By eliminating more of the manual input, the probability of human error is also greatly reduced and thus saves more time that may have otherwise been spent figuring out what had gone wrong and how to fix it. We look forward to hearing more in the future of this series!

Matt Stine replied on Fri, 2010/10/15 - 11:09pm

@XebiaLabs - thanks for the positive feedback! I hope to get back to this topic again next week.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.