Release early, release often.

The reaction to James' Newkirk's announcement of XUnit (looks interesting!) has reminded me of a very important principle: Release early, release often.

Sometime last year I created some spiffy data-driven extensions for MbUnit that took it far beyond RowTest.  Actually, MbUnit supports a whole suite of data-driven testing features but there were still a few things I wanted to have that didn't exist.

Keep it simple

One of goals was to provide simplified support for pulling data from files.  I wanted a simple declarative form that would load data from CSV or Xml files with a minimum of fuss.  Ideally it would map values based on the actual field, property or parameter names in the tests so that very little in the way of data binding glue would be required in the common case.  It should also be easy to define new data sources (and it was...).

Here's an example using CSV:

[DataTestFixture]
[TestFixture]
public class AuthenticationTest
{
    [CsvResource("server-config.csv")]
    public string ServerName;

    [CsvResource("server-config.csv")]
    public string ServerPort;

    [DataTest]
    [CsvResource("credentials.csv")]
    public void Login(string userName, string password, bool isValid)
    {
        Server server = new Server(ServerName, ServerPort);
        bool success = server.Login(userName, password);
        Assert.AreEqual(isValid, success);
    }
}

Where we have two embedded resources.
server-config.csv

ServerName, ServerPort, Owner, Domain
mymachine.test.example.com, 9999, operations, test

credentials.csv

UserName, Password, IsValid
jeff, letmein, true
jeff, knockknock, false
jim, letmein, false
jim, knockknock, true
"", "", false

Execution of the test is pretty straightforward.  The test fixture's ServerName and Password fields are initialized with the contents of the matching columns of the server-config.csv file.  Then the Login test method gets run once for each row in credentials.csv.  Data binding occurs automatically including conversions from strings to other types (like booleans).

Integrate!

A second goal was to simplify retargeting of tests to multiple environments without recompilation or manual munging of files.  Like most enterprises, we have separate development, test, staging and production environments.  The configuration of each environment varies in a few particulars such as in domain names and in the pre-populated data set.

To capture environment-based configuration, I created a new batch of attributes like this:

[DevCsvResource("server-config.dev.csv")]
[TestCsvResource("server-config.test.csv")]
[StageCsvResource("server-config.stage.csv")]
public string ServerName;

Each new file now specifies configuration for a particular environment.

The environment to target is selected at runtime either by popping up a dialog box for the user to choose one (the selection is saved for the duration of the test session) or by grabbing it from an environment variable.  (Remark: MbUnit Gallio will include support for passing external values to tests directly from the command-line or from the GUI test runner to handle just this sort of case with less hackery.)

The moral of the story...

I thought this API was pretty cute so I published the (non-proprietary) code to the MbUnit mailing list.  However, since I was already thinking about how to perfect the feature in MbUnit Gallio, I refrained from polishing the code and contributing it back to the MbUnit v2 framework itself.  That was a mistake.

If I had polished the code, other people would be using it now and no doubt I'd have received plenty of useful feedback to apply towards MbUnit Gallio.  Oh well.  So here is the code for these simple data-driven extensions for MbUnit v2, completely unretouched.

With that in mind, we're gearing up to cut the first alpha release of MbUnit Gallio soon...  Release early, release often.