Should Programming Languages Support Unit Testing Natively?

I used to be strongly opposed to this idea but I started changing my mind recently. Here is what happened.

 

The bad

Production and test code can be integrated at various levels:

1. Supported by the language.
2. Not supported by the language but mixing production and test code in the same classes.
3. Production and test code live in different classes but in the same directory.
4. Production and test code live in different directories.

I have always thought that options 2) and 3) are a bad idea because they make it hard to read and review the code, they contribute to the creation of huge classes and they negatively impact your build infrastructure (which must now be able to strip out the test code when you want to create a shippable binary). We (Google) feel strongly about these points, so we are strictly enforcing option 4) (although we often put our tests in the same package as the production code).

I think this practice is the most common out there and it works very well.

With that in mind, wouldn't a language that natively supports unit testing be the worst case scenario?

 

The epiphany

Not long ago, I reflected on my testing habits for the past ten years, and I made a couple of interesting observations:

• I feel the need to write tests for the code that I write very often.
• Just as often, that need is thwarted by environmental constraints, so I end up not writing these tests.

My experience is with large software, large teams and huge code bases. When you work in this kind of environment, it's very common for the company to have developed its own testing infrastructure. Sure, the code remains code, but how you run it and how you deploy it will vary greatly from company to company (and sometimes even from project to project).

Typically, I code a feature, iterate over it a few times and I reach a point when I'm pretty happy with its shape: it's looking decent, it gets the job done and while there is obviously more work to be done on it, it's mature enough that writing tests for it at this point will not be a waste.

The code to write these tests is usually pretty obvious, so I can form it in my head pretty quickly and materialize it in code not long after that. Now I need to find a way to actually run this test and make it part of our bigger testing infrastructure, and this is where things usually get ugly. I typically find myself having to change or update my environment, invoke different tools, pull out various wiki/HTML pages to brush up on what's required to integrate my tests to the big picture.

The worst part is that I will probably have to relearn everything from scratch when I switch to the next project or the next job. Again, I will write the test (which is pretty easy since it's the same language I used to write the production code) and I will find myself having to learn a brand new environment to run that test.

The environmental hurdle is not easy to address, but if the language that I am coding in supported unit tests natively, I would probably be much more tempted to write these tests since 1) there is now an obvious location where they should go and 2) it's very likely that the test infrastructure in place knows how to run these tests that I will be writing.

The main gain here is that the developer and the testing infrastructure now share a common knowledge: the developer knows how to write tests and the infrastructure knows how to access these tests. And since this mechanism is part of the language, it will remain the same regardless of the project or the company.

How do we do it?

So what would a language that natively supports unit tests look like?

I know first hand that writing a test framework is not easy, so it's important to make sure that the test feature remains reasonably scoped and that it doesn't impact the language complexity too much. You will notice that throughout this entire article, I make a point of saying "unit test" and not just "test". As much as TestNG is focused on enabling the entire spectrum of testing, I think it's important for a language to only support unit testing, or more precisely, to only make it easy to test the compilation unit that the test is found in.

Interestingly, very few modern languages support unit testing, and the only one that I found among the "recent" ones is D (I'm sure commenters will be happy to point me to more languages).

D's approach is pretty minimal: you can declare a unittest section in your class. This keyword acts as a method and you simply write your tests inside:

//
// D
//
class Sum
{
  int add(int x, int y) { return x + y; }
 
  unittest
  {
    Sum sum = new Sum;
    assert(sum.add(3,4) == 7);
    assert(sum.add(-2,0) == -2);
 }
}

This is as barebones as it can get. The advantage is that the impact on the language itself is minimal, but I'm wondering if I might want to be able to write different unit test methods instead of having just one that contains all my tests. And if we're going down that path, why not make the unittest keyword be the equivalent of a class instead of just a method?

//
// Pseudo-Java
//
public class Sum {
  public int add(int x, int y) { return x + y; }
}
 
unittest {
  public void positiveNumbers() {
    Sum sum = new Sum();
    assert(sum.add(3,4) == 7);
  }
 
  public void negativeNumbers() {
    Sum sum = new Sum();
    assert(sum.add(-2,0) == -2);
  }
}

 

As I said earlier, I think it's very important for this feature to remain as simple as possible, so what features from sophisticated testing frameworks should we remove and which ones should we keep?

If we stick with D's approach, there is probably little we can add, but if we go toward a class keyword, then there are probably two features that I think would be useful:

• Method setUp/tearDown (which would already be useful in the example above, where we create a new Sum object in both test methods.
• Exception testing.

At this point, I'm already feeling abit uncomfortable with the extra complexity, so maybe D's simplistic approach is where we should draw the line.

What do you think about native support for unit testing in programming languages? And what features would you like to see?

From http://beust.com/weblog