I'm software developer and microISV, with a lot of passion to technologies.. having a fun writing code, writing blog posts and giving public speeches. For about 10 years of total experince I've been working in many branches of software development, including management and QA. I try to follow and adopt agile practices and signed up for software craftsmenship manifesto. Alexander is a DZone MVB and is not an employee of DZone and has posted 66 posts at DZone. You can read more from them at their website. View Full User Profile

Functional tests must not be done first

07.16.2011
| 4384 views |
  • submit to reddit

First of all, by functional tests I understand tests of application features primarily from user (UI) perspective, with different set of frameworks - Seleminum, Watin, FuncUnit or whatever.

I've seen the cases that functional tests are being implemented either first or immediately after some particular feature is implemented. I think this is a just wrong practice. Even the UI testing could definitely provide value you should think twice before getting to deep with functional testing and apply as development process rule. Functional tests have several serious drawbacks:

  • Fragility. Functional tests are extremely fragile. You could run test suite and got green results and re-run them in 5 seconds and got completely red sheet. And there could be thousands of reasons why it happened.
  • Speed. They are simply slow. As much tests you have as slow they are. As much slow tests you have as long feedback cycle is. As long feedback cycle is as low value you are getting from tests.
  • Bad isolation. Because of functional tests have integration nature, it is just not possible to run them in isolated environment. Tests became depended from each other, that in particular leads to fragility.
  • Support. Implementation of UI test is not trivial thing. Even with so cool frameworks like FuncUnit or WebDrivers that have strict and clear interface, it could be hard to test some UI feature of application. But support is much more harder - really small change in could turn many tests to red state. In fact, implementation of change that costs you 1 hour, could cost 1 day for correction of tests.
  • Changes. We are trying to adopt agile into development world, one of the aspect that makes development really agile is reacting on change. Reacting on change have to be fast. Changes are integral part of development, even if developers hate changes. But as I said above, even small change could ruin UI tests and it is not fast to fix it. It means that cost of change is getting more and more expensive.

So, trying to implement UI tests in early development stages or cover all applications features with UI tests is a waste. Ratio between effort / value would be just go to zero with time.

What I think as recommendation for UI testing:

  • Rely more on unit tests - all server side code must be covered with a different level of testing. Units have to be isolated and fast.
  • Keep view as thin as possible - in ideal you should not have logic in view at all, but modern web application heavily using client side (javascript) logic that affects view. Javascript code have to be unit tested with qUnit or similar frameworks.
  • Don't start with UI tests - UI testing have to be done last. If the feature is just implemented it does not mean it has to be immediately tested. It has to pass PO approval, QA. Ideally it has to be released to beta group and confirmed that "this is exactly what we need". After feature stabilized test could be added.
  • Don't try to test everything - since UI testing cost more, you should be more careful. It has sense to automate smoke test, critical parts of application or parts that have high regression.
  • Clean up tests regularly - peer tests regularly, if you have bunch of tests for some area that works OK for a long time you might consider to remove those tests, to speedup whole tests process.

Keeping focus only of primary features of application and avoid of testing small stuff should be key for functional tests.

 

From http://www.beletsky.net/2011/04/functional-tests-must-not-be-done-first.html

Published at DZone with permission of Alexander Beletsky, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

John Ferguson Smart replied on Sun, 2011/07/17 - 4:39am

Yes and no. For most apps, UI tests are too fragile to implement first (I have seen exceptions to this rule). However, if you use an ATDD/BDD approach, you would automate a set of pending high-level functional tests before doing any code, and implement the tests once the UI is reasonably stable. Note that I'm talking about high-level, specification-by-example style tests, which shouldn't go into implementation details but remain very high level (in other words, focus on 'what', not 'how'). This gives you your goal posts for the story and even for an iteration, and helps massively with team communication. It also gives you the flexibility to implement the UI as best you see fit, without compromising the tests. Gojko Adzic's 'Specification By Example' book is a good introduction to this approach.

Alexander Beletsky replied on Mon, 2011/07/18 - 1:37am in response to: John Ferguson Smart

John, of cause ATDD/BDD are very important stuff! I personally want to use both as much as possible. What I'm saying is that I saw ATDD used a little wrong, namely just starting with that cases. It make more problems, than benefits as for me :)

John Ferguson Smart replied on Mon, 2011/07/18 - 11:01pm

Fair point, and I totally agree with the danger of going too far and too deep with high-level functional tests (including implementing the details too soon when the UI is not yet stable) instead of more comprehensive unit tests.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.