I am a software engineer at Google on the Android project and the creator of the Java testing framework TestNG. When I'm not updating this weblog with various software-related posts or speaking at conferences, I am busy snowboarding, playing squash, tennis, golf or volleyball or scuba diving. Cedric is a DZone MVB and is not an employee of DZone and has posted 90 posts at DZone. You can read more from them at their website. View Full User Profile

Should Programming Languages Support Unit Testing Natively?

11.09.2009
| 6418 views |
  • submit to reddit

I used to be strongly opposed to this idea but I started changing my mind recently. Here is what happened.

 

The bad

Production and test code can be integrated at various levels:

  1. Supported by the language.
  2. Not supported by the language but mixing production and test code in the same classes.
  3. Production and test code live in different classes but in the same directory.
  4. Production and test code live in different directories.
I have always thought that options 2) and 3) are a bad idea because they make it hard to read and review the code, they contribute to the creation of huge classes and they negatively impact your build infrastructure (which must now be able to strip out the test code when you want to create a shippable binary). We (Google) feel strongly about these points, so we are strictly enforcing option 4) (although we often put our tests in the same package as the production code).

I think this practice is the most common out there and it works very well.

With that in mind, wouldn't a language that natively supports unit testing be the worst case scenario?

 

The epiphany

Not long ago, I reflected on my testing habits for the past ten years, and I made a couple of interesting observations:

  • I feel the need to write tests for the code that I write very often.
  • Just as often, that need is thwarted by environmental constraints, so I end up not writing these tests.
My experience is with large software, large teams and huge code bases. When you work in this kind of environment, it's very common for the company to have developed its own testing infrastructure. Sure, the code remains code, but how you run it and how you deploy it will vary greatly from company to company (and sometimes even from project to project).

Typically, I code a feature, iterate over it a few times and I reach a point when I'm pretty happy with its shape: it's looking decent, it gets the job done and while there is obviously more work to be done on it, it's mature enough that writing tests for it at this point will not be a waste.

The code to write these tests is usually pretty obvious, so I can form it in my head pretty quickly and materialize it in code not long after that. Now I need to find a way to actually run this test and make it part of our bigger testing infrastructure, and this is where things usually get ugly. I typically find myself having to change or update my environment, invoke different tools, pull out various wiki/HTML pages to brush up on what's required to integrate my tests to the big picture.

The worst part is that I will probably have to relearn everything from scratch when I switch to the next project or the next job. Again, I will write the test (which is pretty easy since it's the same language I used to write the production code) and I will find myself having to learn a brand new environment to run that test.

The environmental hurdle is not easy to address, but if the language that I am coding in supported unit tests natively, I would probably be much more tempted to write these tests since 1) there is now an obvious location where they should go and 2) it's very likely that the test infrastructure in place knows how to run these tests that I will be writing.

The main gain here is that the developer and the testing infrastructure now share a common knowledge: the developer knows how to write tests and the infrastructure knows how to access these tests. And since this mechanism is part of the language, it will remain the same regardless of the project or the company.

How do we do it?

So what would a language that natively supports unit tests look like?

I know first hand that writing a test framework is not easy, so it's important to make sure that the test feature remains reasonably scoped and that it doesn't impact the language complexity too much. You will notice that throughout this entire article, I make a point of saying "unit test" and not just "test". As much as TestNG is focused on enabling the entire spectrum of testing, I think it's important for a language to only support unit testing, or more precisely, to only make it easy to test the compilation unit that the test is found in.

Interestingly, very few modern languages support unit testing, and the only one that I found among the "recent" ones is D (I'm sure commenters will be happy to point me to more languages).

D's approach is pretty minimal: you can declare a unittest section in your class. This keyword acts as a method and you simply write your tests inside:

//
// D
//
class Sum
{
int add(int x, int y) { return x + y; }

unittest
{
Sum sum = new Sum;
assert(sum.add(3,4) == 7);
assert(sum.add(-2,0) == -2);
}
}

 

This is as barebones as it can get. The advantage is that the impact on the language itself is minimal, but I'm wondering if I might want to be able to write different unit test methods instead of having just one that contains all my tests. And if we're going down that path, why not make the unittest keyword be the equivalent of a class instead of just a method?

//
// Pseudo-Java
//
public class Sum {
public int add(int x, int y) { return x + y; }
}

unittest {
public void positiveNumbers() {
Sum sum = new Sum();
assert(sum.add(3,4) == 7);
}

public void negativeNumbers() {
Sum sum = new Sum();
assert(sum.add(-2,0) == -2);
}
}

 

 
As I said earlier, I think it's very important for this feature to remain as simple as possible, so what features from sophisticated testing frameworks should we remove and which ones should we keep?

If we stick with D's approach, there is probably little we can add, but if we go toward a class keyword, then there are probably two features that I think would be useful:

  • Method setUp/tearDown (which would already be useful in the example above, where we create a new Sum object in both test methods.
  • Exception testing.

At this point, I'm already feeling abit uncomfortable with the extra complexity, so maybe D's simplistic approach is where we should draw the line.

What do you think about native support for unit testing in programming languages? And what features would you like to see?

From http://beust.com/weblog

Published at DZone with permission of Cedric Beust, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Thomas Courant replied on Mon, 2009/11/09 - 6:03am

Hello,

i'm not sure unit testing in native language is a good thing.

I think that production code and unit testing must be separated, except for basic or light unit testing, otherwise it can produce large code file where all is mixed.

The philosophy behind unit testing is that we write them as if we don't know the implementation, so it doesn't influence our testing code (black box concept).

Maybe, development environment (such as Eclipse and plugins) can constraint us to write or warn us for testing classes we write.

As for environment issue, maybe using mock objects or simplify coupling may solve it.

Best regards 

 

 

 

Endre Varga replied on Mon, 2009/11/09 - 7:07am

I think it should be part of the language. Unit testing is something that you need high discipline to do regularly. If there is any help that could make writing of test cases quicker and less pain, we should have it. I think language integration could greatly reduce much of the pain. Also, I do not agree with black box testing. Testing an interface is important, and It should be reusable to test many implementations. But I do not agree on abandoning testing of individual implementations.

Tracy Nelson replied on Mon, 2009/11/09 - 10:22am

I don't think it should be part of the language.  From the article, it sounds like the big problem is integrating the unit tests into the build & test environment in the project.  Building support into the language won't help much.  Sure, if the environment leverages the built-in test facilities of the languages (as you suggest), it could help, but on bigger projects with more complex test environments is exactly where you're going to run into test frameworks that have to accomodate multi-language or extended resource testing (e.g., XML validation, database schema checking).  Such frameworks will be more likely to ignore language-specific features and instead rely on external tools/techniques, which is where your problems crop up.

I think that testing is part of the software development process, and languages shouldn't be expected to provide explicit support for certain practices.  Or worse, dictate how certain practices should be performed.

JeffS replied on Mon, 2009/11/09 - 1:03pm

No, programming languages should not support unit testing natively. Languages don't need the feature bloat, and external tools/libraries handle the job just fine. Also, unit testing, just like many other techniques/tools, should be used judiciously, not as an automatic knee jerk strategy.

kent tong replied on Tue, 2009/11/10 - 12:25am

Possible to write a simple tool that runs your test? This tool should be specific to the way the tests are organized and run in your company/project. Ultimately it might support a plug-in system to allow it to be used in different enviroments.

Armin Ehrenreich replied on Tue, 2009/11/10 - 5:41am

It is most of the time a very bad idea to include something in the language. Testing is a typical task of an API. If e.g. someone comes up with a better idea for a testing framework/strategy you are stuck when you included it in the language or you have at least unnecessary feature bloat in your language. This is similar to scripting languages that often include some data structures in the language. But if you need a different implementation of an included data structure...

Mark Thornton replied on Tue, 2009/11/10 - 9:49am

It appears to me that what you really want is for someone to anoint a single framework as the one true way to do unit testing. It is certainly annoying working with one framework that has argments expected, actual while another uses actual, expected. On the other hand we should remember that adding logging to Java didn't end the logging debate.

I think it is too late to add unit testing into the language --- it would just become another way and not even the dominant method.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.