Jakub is a Java EE developer since 2005 and occasionally a project manager, working currently with Iterate AS. He's highly interested in developer productivity (and tools like Maven and AOP/AspectJ), web frameworks, Java portals, testing and performance and works a lot with IBM technologies. A native to Czech Republic, he lives now in Oslo, Norway. Jakub is a DZone MVB and is not an employee of DZone and has posted 154 posts at DZone. You can read more from them at their website. View Full User Profile

How to Create Maintainable Acceptance Tests

  • submit to reddit

This post summarizes what I’ve learned from various sources about making acceptance or black-box tests maintainable. This topic is of great interest to me because I believe in the benefits that acceptance tests can bring (such as living documentation) but I’m also very much aware that it is all too easy to create an unmaintainable monster whose weight eventually crushes you. So the question is how to navigate the minefield to get to the golden apple?

The key elements that contribute to the maintainability of acceptance tests are:

  1. Aligned business, software, and test models => small change in business requires only a similarly small change in the software and a small change in tests (Gojko Adzic explains that very well in his JavaZone 2012 talk Long-term value of acceptance tests)
    • The key to gaining the alignment is to use business language in all the three models from the very start, building them around business concepts and relationships
  2. Testing under the surface level, if possible
    • Prefer to test your application via the service layer or at worst the servlet layer; only test on the UI level if you really have to and only as little as possible for UI is much more brittle (and also difficult to test)
    • The more you want to test the more you have to pay for it in the terms of maintenance effort. Usually you decide so that you cover the part(s) of the application where the most risk is – the best thing is to do cost-benefit evaluation.
  3. Isolating tests from implementation by layers of test abstraction
    • Top layer: Acceptance tests should only describe “what” is tested and never “how” to test it. You must avoid writing scripts instead of specifications.
    • Layer 2: Instrumentation – right below the acceptance test is an instrumentation layer, which extracts input/output data from the test and defines how to perform the test via a high-level API, provided by the next level (we could say a test DSL) such as “logInUser(X); openAccountPage();”
    • Layer 3: High-level test DSL: This layer contains all the implementation details and exposes to the higher layer high-level primitives that it can use to compose the tests without depending on implementation details (ex.: logInUser may use HtmlUnit to load a page, fill a form, post it)

(And of course many, if not all, of the rules for creating maintainable unit tests apply as well.)

What others have to say

On the three layers of test isolation:

  1. Business rule or functionality level: what is this test demonstrating or exercising. Ideally illustrated with realistic key examples. For example: Free delivery is offered to customers who order two or more books, illustrated with an example of a customer who orders one book and doesn’t get free delivery and an example of a customer who orders two books and gets free delivery.
  2. User interface workflow level: what does a user have to do to exercise the functionality through the UI, on a higher activity level. For example, put the specified number of books in a shopping cart, enter address details, verify that delivery options include or not include free delivery as expected.
  3. Technical activity level: what are the technical steps required to exercise the functionality. For example, open the shop homepage, log in with testuser and testpassword go to the /book page, click on the first image with the book CSS class, wait for page to load, click on the ‘Buy now’ link and so on.

(The Secret Ninja Cucumber Scrolls, page 109 – 110)

On aligned business, software and test models:

The principle of symmetric change
The best software is the one where the technical software model is aligned with the relevant business domain model. This ensures that one small change in the business domain (new requirements or changes to existing features) results in one small change in software. The same is true for the other artefacts produced for software — tests and documentation. When the tests are aligned with the business domain model, one small change in business will result in one small change in tests, making the tests easy to maintain.
Tests described at a low level technical detail, in the language of technical UI interactions, are everything but aligned with a business model. If anything, they are aligned with the current design and layout of user interfaces. A small change in business requirements can have a huge impact on such tests, requiring hours of updates to tests after just a few minutes of changes to the code.

(The Secret Ninja Cucumber Scrolls, page 111)


This post is very much based on the following resources:

  1. The Secret Ninja Cucumber Scrolls (free pdf) by David de Florinier and Gojko Adzic, 2011-03-16
  2. http://concordion.org/ by David Peterson
  3. Specification by example, Gojko Adzic, 2011, Manning, ISBN 978-1617290084
  4. Long-term value of acceptance tests, Gojko Adzic, a talk at JavaZone 2012


From http://theholyjava.wordpress.com/2012/01/18/how-to-create-maintainable-acceptance-tests/

Published at DZone with permission of Jakub Holý, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)



Steven Baker replied on Mon, 2012/01/30 - 6:52pm

Mistype. Sorry.

Dean Pehrsson-c... replied on Wed, 2012/02/01 - 6:32am

"Prefer to test your application via the service layer or at worst the servlet layer; only test on the UI level if you really have to and only as little as possible for UI is much more brittle (and also difficult to test)" Can you expand on this? I can see how this helps keep the tests maintainable, but surely the UI is part of the software - unless a working UI is not part of what a user deems acceptable, it should be in there, brittle or not.

Jakub Holý replied on Fri, 2012/02/10 - 5:24pm in response to: Dean Pehrsson-chapman

Hi Dean, good question!

We know that we can never test everything - and thus we must decide what to test and what not to test. How to make that decision?  Based on economics, of course - what do we gain by having the tests? When does the cost of creating and maintaining them exceed that gain? In this case the gain is the mitigation of risk that something will break - and thus the question is wher, in which part and layers of the particular application is most of the risk? How much of the risk do we need to and can afford to eliminate? 50%, 60%, 90%? (Though I can't really claim that tests can eliminate it.)

UI is usually a rather low-risk area (most of the risk being in the business logic and rules, occasionally in integration and cooperation of the components) while it is also extremely expensive to test (and also quite volatile).

For a typical application it thus isn't worth testing the Ui thoroughly, it is much more important to test the logic (and by testing it at the service layer, we can get it covered much better and cheaply then through the UI). You only need few UI tests to verify the key/happy flows through the app and some manual/exploratory testing.

Of course there are special cases where there is the need to test the UI ore extensively whatever the costs but quite rarely.

Acceptance tests, despite their name, haven't only the goal to prove the app is acceptable for the user - and I'd actually say that it even isn't one of their main goals. After all, tests cannot prove that something is correct - they can only ever prove that something is not correct. The key benefit is improved maintainability and, occasionally, "living documentation". For users/customers to accept the app you'll anyway need a manual user acceptance tests. Thus if automated acc. tests aren't sufficient as a replacement of a manuak UAT, you can drop the UI part and concentrate on their other benefits :-) 

Carla Brian replied on Thu, 2012/03/29 - 5:18pm

Every change in the system of the business should be tested after. This is to ensure that if the software is efficient and if it follow the business flow in the company. - Dr Paul Perito

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.