Monday, March 19, 2007

First MbUnit v3 checkin.

Well, I finally found time to work on MbUnit v3. After tossing around many Object Model drafts, I decided to leave off for a bit and work on some of the easier bits in the framework. So here's the first checkin log message pretty much verbatim.

First MbUnit v3 checkin.
I've been focusing on the runtime framework side of things for a bit because there were a few ideas I wanted to try out there which weren't quite as crazy as the rest so they seemed like a good jumping off point. The test cases are lacking in this checkin. Don't panic!

This checkin exhibits several new design elements on the framework side of things. A common theme is the expansion of MbUnit's range of applicability to include functional system testing. Such tests place a higher burden on the ability of a framework to schedule, monitor, and report on tests / work-units.

1. Framework components are resolved indirectly through services. This makes it possible to mock out, configure or substitute components that would otherwise have been hardcoded in. Moreover, it affords a greater degree of separation of concerns between the framework and the core. The primary use-case for this approach is to facilitate testing but there are certainly other possibilities involving non-default component implementations that might be useful for particular framework extensions or embeddings.

2. The runtime context of the test is exposed to the test framework (and the test) to provide for reflection of the test execution scope, management of scoped test objects, and for interception of scope-related events. The primary use-case is for specialized test framework extensions that perform advanced tracing of tests or that require robust resource reclamation.

3. Multiple concurrent runtime contexts are maintained. Moreover, contexts can be passed among threads (and are usually expected to be inherited by default via the .Net CallContext mechanism). The objective is to provide a firm foundation for two forms of concurrency: a) one test involving multiple threads that must be coordinated and monitoring and that may need to be killed when the test ends, b) multiple tests running in parallel such as for load-test fixtures.

4. Tests have access to greatly expanded reporting facilities. This has come from the observation that many test writers really want to be able to dump out more log information and classify it. However, a more interesting use-case is to support embedding images and other rich content within the test results. That's one of the top features my office's QA team has requested for our little web test framework since they spend hours tracing faults. If this works out, MbUnit v3's test reports have the potential to be much sexier than anything else out there.

5. Assertions are described by Assertion and AssertionResult objects. Each Assertion has a unique id that identifies the kind of test being performed. Moreover it also captures the parameters of the assertion and the user-supplied detail message. This information may be used by a smart GUI application to visualize a failure intelligently such as by displaying actual/expected results side by side.

Assertions are no longer always one-shot deals that throw AssertionExceptions. It is now possible to evaluate a block of code and collect zero or more assertion failures along the way with the _option_ of terminating execution after the first one by throwing an AssertionException (typical behavior). Consequently assertions are now expressed using 3 primitives: Run() runs a Block and collects AssertionResults it generated, Evaluate() evaluates an AssertionCondition for a given Assertion and returns its AssertionResult, and Verify() tallies and examines an AssertionResult and throws an AssertionException on failures unless multiple assertions are allows.

Assert.Multiple runs a Block and groups multiple logically related assertions into a single unit so that more diagnostic information can be gathered about the state of the objects involved. This takes advantage of the ability to capture multiple assertion failures without throwing AssertionExceptions.

Assert.Ignore runs a Block and disables the verification phase of all contained assertions. This is intended as a debugging tool to allow the code to run longer than it might otherwise and so that more diagnostic information can be captured.

Assert.ExpectedException runs a Block and yields a failure unless an exception of the expected type was thrown.

Assert.ExpectedFailure runs a Block and yields a failure unless an assertion failure occurred within the block which effectively inverts the meaning of the contained assertions (well, not quite). This one may be a little gimmicky but is very straightforward. I'm curious to see what people do with it.

Lots more stuff to come!

Today I'm working on the design doc so I don't get too carried away... There's a lot of stuff in my head that I need to dump now that I have some time.

Microsoft's Web Client Software Factory

A coworker brought up Microsoft's Web Client Software Factory bundle from MS Patterns and Practices as a possible foundation for rearchitecting our web apps.

I hadn't looked at it much before and while it does seem like a step up from our ad hoc solutions, it just doesn't seem to have the elegance of Castle.MonoRail and particularly Castle.Igloo. Moreover, ObjectBuilder does not seem as powerful as Castle.Windsor (which I've been using lots on new projects!)

The bonus for WCSF is that it's Microsoft code so it instantly acquires more legitimacy to some people. (The downside of course is also that it's Microsoft code...) Given that I know MonoRail doesn't have a snowball's chance of cracking into our tangled and legacy-encumbred Platform perhaps WCSF would be able to make a small measure of improvement. Our architecture already includes the concept of a PageHandler which is a pretty close fit for an MVP-ish Presenter. It also includes the concept of a (Page)Flow and does cookie-based persistence and state management via reflection. Formalizing these notions and setting them on a firmer foundation seems helpful overall. Given WCSF's service model, I'd also expect the code would become less coupled to the concrete implementation and therefore more testable.

Any opinions from the peanut gallery?

Friday, March 16, 2007

How do you deploy?

Ayende Rahien asked: How do you deploy?

In our case the CI build server produces a ZIP file containing the build image. Then we run a command on a master deployment machine which pulls the latest build ZIP (or a specified version) and uses the WSHController COM object to fire off commands on the individual servers to deploy that version, also providing additional information about the environment.

I wrote a prototype batch deployment tool using Ruby on Rails two weeks ago which will eliminate the step of remoting into the master deployment machine. This still needs to be under partial manual control because we use the same process in production but it's convenient to have all the commands in one place with some status screens. The next step is to investigate whethere more of the workflow should be integrated into it because production deployments require a significant amount of communication among teams.

The CI integration deployment eliminates the manual step and does not perform remote command execution. The CCNet instance on the build server fires a message over to force the CCNet instance on the integration server to deploy the latest ZIP'ed build image to the local machine and run the tests with MbUnit and NCover. Pretty simple.

Where it all breaks down...

The remote deployment trick with WSHController has worked fine for 5 years. Unfortunately we started having reliability issues with WSHController several months ago where it occasionally crashes or hangs. It appears to be due to a buffer overflow in its handling of environment variables. After many hours of trying to figure out how to fix or work around it, I just decided to rewrite the script using a different remote command execution tool. So yesterday I spent 2 hours rewriting the remote deployment script to use SysInternals ' PsExec instead. Then I spent the next 14 hours trying to figure out why it also crashes (C0000409) or hangs (sometimes, unless the output of the remote command is redirected to a file instead). As I can't see anything obviously wrong with my wrapper script (in Iron Python) I'm left wondering which patch, upgrade, or service pack might have caused my current woes. *sigh*

So... now I'm going to take a break, have lunch on the patio, watch the fish swim in the pond, try to do something about my sinus headache and get down to working on MbUnit v3.

Wednesday, March 14, 2007

Mixins: Separation of Concerns in Test Fixtures

I was working with a tester yesterday to provide access to STAF from within a system test written in C# with MbUnit. Recently we started a project to use STAF to coordinate system tests across our test farm so that we can run tests early, often, reliably and in parallel.

As is common with large stateful systems, there are a limited number of resources that system tests much share among themselves. For instance, our system contains a Hammer box with a limited number of ports that must be shared. Likewise, functional tests often share common test user accounts so those tests must never be permitted to run concurrently or else a pool of suitable test user accounts must be provided. Consequently there must be rules governing which test is permitted to use which resources at which times. Up to now, we have settled those rules statically by hardcoding reservations for certain resources or by trying to time-boxing test schedules to discourage concurrent execution of conflicting tests. As expected, this process is extremely brittle. So moving forward we decided to perform such reservations dynamically using the STAF Resource Pool.

So we started off with some code by Rohit Krishna Kumar to provide a primitive C# wrapper for STAF based on P/Invokes. But we wanted to go further! We wanted the test framework to automatically manage STAF Handle registration and disposal and to support a declarative notation for allocating resources from STAF resource pools.

Today's Problem: How to achieve separation of concerns in a test fixture?

So here's the kind of code we want to write in the test:

public class MyTest : STAFTest
    [STAFResource("master-machine", "HammerPortResourcePool")]
    public int HammerPort;

    public void MakeACall()
        // do something involving the HammerPort resource we just obtained.

STAF integration is really cool, but how can we reconcile it with our in-house Web test framework (similar to Watir but written in C#). Both the STAF integration and the Web test framework integration want to be super-classes of the tests! Both of them need to contribute test fixture setup and framework code to manage their lifecycle so it's going to get tangled.

public class TangledSTAFAndWebTest : SimpleWebTest
    public STAFMixin STAF;

    public override void TestFixtureSetUp()

        STAF = new STAFMixin(this);

    public override void TestFixtureTearDown()
        if (STAF != null)


Yuck! So we picked SimpleWebTest to be the superclass and to inject STAF support somewhat ungracefully so we could also use it elsewhere with other superclasses. MbUnit v2 only allows one [TestFixtureSetUp] or [TestFixtureTearDown] attribute each per test fixture so they have to be declared in the base class and made virtual. So in general a few changes had to occur across the class hierarchy to ensure this scenario would work. Awww... if only we had mixins...

public class STAFAndWebTest
    public STAFMixin STAF;

    public WebTestFrameworkMixin Web;

Well we don't really have mixins, but we can certainly simulate them. And with a little framework support it becomes quite easy.

public class STAFMixin
    private object fixture;
    private STAFHandle handle;

    public STAFMixin(object fixture)
        this.fixture = fixture;

    public STAFHandle STAFHandle
            if (handle == null)
                handle = new STAFHandle(GetType().FullName);
            return handle;

    public void TestFixtureSetUp()
        Type fixtureType = fixture.GetType();

        foreach (FieldInfo field in fixtureType.GetFields(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance))
            foreach (STAFResourceAttribute attrib in field.GetCustomAttributes(typeof(STAFResourceAttribute), true))
                string value = AllocateResource(attrib.Endpoint, attrib.PoolName, true).Value;
                field.SetValue(fixture, Convert.ChangeType(value, field.FieldType));

        // etc...

    public void TestFixtureTearDown()
        // similar to above, release resources.

        if (handle != null)
            handle = null;

    /* Code for submitting STAF requests, allocating resources, etc... */

Ooooh shiny! The test framework instantiates our mixins for us and assigns them to fields on the test fixture then incorporates them into the test fixture's lifecycle. In principle, a mixin could participate in test fixture setup/teardown, in test setup/teardown and in resolution of data-bound test parameters (like any fields decorated by STAFResourceAttribute).

There's a lot more though. Why stop at just one level of mixin inclusion? Why not just think of the test fixture mixin as a component that has been resolved, instantiated and installed for us by the test framework acting in the guise of an inversion of control container...

Edit: Made a few cosmetic and grammatical changes to the content.

Down the Rabbit Hole

My name is Jeff Brown and I'm a programmer. I'm originally from Ottawa, Ontario but somehow found my way to San Francisco, California where the weather is much finer.

I'm fond of programming languages, software architecture, system verification, distributed systems, generative code, tooling, automation, twisted metaphors, friends, good times, spicy food, raw fish, and beer, eh? Lately I've been working primarily with the .Net platform using Adobe Flex, ASP.Net, MbUnit, Rhino.Mocks, Castle, IronPython and other goodies. I confess to have had a secret love affair with Ruby on Rails. I turned my nose up at Java then learned to love it for a few years and then just felt indifferent about it. Not so long ago I found C++ template meta-programming quite useful in everyday work. I used to think in Motorola 68k assembler and in TMS9900 before that. I grew up with TI BASIC. I'd love to eat Monads for breakfast but Haskell O's and .Net Flakes go together like milk and orange juice (at least when consumed at the office) so alas that project has been put on hold. I'll skip over the rest, perhaps leave that to a retrospective post... Or maybe not.

Here there shall be much ado about programming, philosophy, abstraction, patterns, separation of concerns, composability, testability, static verification, robustness, maintenance, tooling and whatever else happens to catch my fancy.

So now that the introductions are over, let's dig into the code...