Tests that last.

Roy Osherove asks on his blog if you have ever written throw away tests (TATs) and whether you have since learned to write tests that last (TTLs).

Sure, I have written TATs and learned from the mistake.  Now I mostly write TTLs and they stay that way because I maintain them.  Sadly there have also been a couple of situations where I've transferred code ownership to others who did not practice unit testing and allowed the test suites to rot... *sigh*

My approach to writing TTLs is pretty simple.

Rule 1: Don't overcommit to your unit testing effort early on.

Ok...  Before the Agile folks put my head on a platter please let me clarify.

I am often engaged in designing architectural concerns such as APIs, processes, conventions and infrastructure.  These concerns all have associated contracts.  I wouldn't sign a contract in blood without careful negotiation first; would you?  You would do more research and you would haggle.  So don't overcommit testing resources to a hastily sketched out system that might change tomorrow.

Architectural concerns have a very high initial design burden.  The cost of changing your mind can be huge!  On the other hand, big design up front is risky.  This seems almost at odds with Agile development.  It isn't.

Ideally we want to partition the design process so that we can obtain confirmation of the soundness of core ideas incrementally.  However, because the core ideas all need to work together smoothly we must reserve the right to change our minds.  To get the feedback we need, it helps to sketch out sample clients that exercise a representative subset of the current prototype.  It's even better if the sample clients are executable because it provides more concrete evidence that we're on the right track.  Oh look, we just wrote integration tests!

What about unit tests for a new API?  Well, eventually we'll want heaps of those.  But early on, they aren't all that helpful.  Moreover since the design is still rather volatile, you run a risk of generating tons of TATs.  This is especially true for white-box tests.  Defer writing expensive and risky tests until you're confident they won't need be thrown away.  (But do remember to write them as soon as you're sure! In many situations that just means waiting a couple hours until you're done thrashing with ReSharper.)

Rule 1 (corollary): Write integration tests before most unit tests.

Unit tests are usually very tightly coupled to the code you're writing.  Volatility in the code under test can translate into major headaches and wasted effort.  In particular, reserve white box tests for last.

Rule 2: Don't aim for 100% code coverage of your first draft.

It's almost never right the first time.  Wait until well after you've first integrated a component into the system and has been thoroughly proven out.  Code coverage metrics are good for gaining more confidence that your implementation is correct; however, they are not of much use figuring out whether your design satisfies the requirements.  You'll probably end up refactoring your first draft beyond recognition so don't waste time checking it too deeply.

Rule 2 (corollary): Don't write white box tests until the implementation has stabilized.

White box tests are useful.  Don't let them turn into TATs because you wrote them prematurely.

Rule 3: Write your tests clearly.

All of the usual tricks apply:

• Test code should be soluble.
• UseVeryClearNamesThatMightEndUpBeingALittleLong.
• Don't verify more than one behavior in each test.  (Try not to have too many assertions in a single test.)
• Limit the usage of mock objects to generating stubs and setting up relatively straightforward expectations for dependencies.
• Clearly document how the test harness is being set up and what interactions are taking place.
• Include detailed messages in your assertions to describe expected behavior.
• Don't do anything surprising in your tests.
• Avoid introducing side-effects across test runs.
• Try to isolate the tests as much as possible from one another.
• Don't depend on too many white box implementation details.  Do cover the corner cases of your algorithms but try not to depend on exactly how the computation will proceed or in what order certain intermediate results will be produced.
• Avoid hardcoding exact string equality comparisons except in trivial cases.  If you are calling a function that formats a text message containing some information then there can be a fair amount of looseness in the formatting.  Look for the presence of expected information by searching for matching substrings.
• Avoid hardcoding hidden policy configuration that might change.  For example, today an anonymous email service might only let you send 5 emails per hour before a spam killing rule is activated; tomorrow it might be 2.  Instead of hardcoding "5" into your test, try to obtain the constant from the same source as the actual implementation.

Rule 4: Invent your own rules!

Conventions are good.  Learn what works and apply that insight routinely!  Then you'll be sure to get tests that *really* last.

MbUnit Gallio, for example

I'm currently architecting MbUnit Gallio incrementally.  If you were to download the source now, you'd notice there aren't all that many unit tests.  Two of the tests currently fail!  Have I lost my mind?  Shouldn't I develop a test framework using TDD?  Perhaps...

The problem is that I ran out of storage space in my brain.  There are too many things going on inside Gallio for me to have any confidence getting it right the first time as a pure gedankenexperiment.  Pretty diagrams and written notes were helpful at the high level but not so much below that.  I needed to write actual code (or at least design interfaces) so that I could observe the interplay between the various high level concepts and adjust them accordingly.  It's been very rewarding.  As the design process progresses, I've managed to isolate and solidify several components to enable others to work in parallel while I iterate over other fuzzy details.  It's getting there!

I've written integration tests at several different levels of integration within the system.  There's enough meat to get confidence that the design is moving in the right direction.  Those integration test failures I mentioned earlier were fully expected because parts of the system are still stubbed out!  Meanwhile I've been trying to progressively write unit tests for components that have solidified.  But here I've encountered the limits of my ability.  I can't do architecture, design, code, tests and project coordination all at the same time.  So as a team we shall work together to increase our testing velocity.  (Which leaves me feeling irrationally guilty because I prefer to take personal responsibility for thoroughly testing everything that I code.  Actually, I need to learn how to delegate more of the detailed design and coding work. Despite being intensely interested in these tasks I need to release my investment in them so that I can focus on maintaining the velocity of the project as a whole.  I am learning to scale up to larger projects.  It's not easy.)

I'm confident that almost all of these tests will last because we are working to understand the implications of the system that we are designing and building.  And we care enough to get it right (eventually)!