Friday, October 26, 2007

A Loopback Mail Server for Functional Regression Testing

So for functional regression testing purposes, we need to be able to create tons of user accounts on our system.  Every user account needs an email address.  And of course, our system sends out notification emails to users so we need to test that stuff too.

No Frills

Ok, so I want to write tests that can submit an email address to the application during user registration and then go read that user's mailbox for any notifications that have been received.

Here's the constraint: I can't instrument our application and stub out its email sending code with a stub.  Remember, I'm writing functional regression tests here.  Some of those will be smoke tests that we want to run in a live deployed environment.  So the emails have to be real.

I don't want to use the corporate mail because of administrative overhead.  I am certainly not interested in asking Ops to create thousands of test email accounts and ensuring their contents get purged on a regular basis.  So we've got to have our own SMTP and POP3 server just for testing purposes.

And there are other requirements:

  • Zero-administration.  I can install the server on some machine and forget about it.
  • Zero-configuration.  I never have to worry about creating user mail boxes.
  • Self-cleaning.  It'll purge stale data automatically.
  • XCOPY deployable.  I can just drop it on a server and maybe run a script to install it as an NT Server and I'm done.
  • Lightweight.  It has such a negligible footprint that it's well below the radar in terms of resource utilization.
  • Robust.  It keeps going, and going, and going...
  • Cheap.  No per-user licensing, please!

Buy or Build?

A little Googling turned up nothing useful in this vein.  (Which surprised me!)  So we can't buy our way out of the problem.  Darn.

Ok, build it.  I'm sure we can cobble such a solution together using sendmail and a mess of perl scripts but I just don't have the time or inclination to dive into that kind of arcana.  (Besides, they both make my eyes bleed.)

Can I do it in a couple hours with off-the-shell components?  Yes!  It's not free, but it's definitely cheap enough.

Introducing the Loopback Mail Server

Last night I spent a couple of hours and put together a simple SMTP / POP3 mail server as described above.

Get the code here:  As mentioned above it does depend on the CleverComponents .Net suite so it's not quite free but it does the job.

Here's what it does:

  • The Loopback Mail Server stores mail messages in a folder called Mail within the application directory to contain the user mail boxes.  Messages are kept around for a maximum of 7 days and are deleted automatically during periodic cleanup sweeps every 4 hours.  (These parameters can all be configured if desired.)
  • The SMTP server accept email sent by any user (no authentication is performed) to any recipient.  Mail for the recipient is stored as flat files in an automatically created folder named based on the recipient's email address.  The recipient's email address is only lightly validated to ensure that it is a valid file name.
  • The POP3 server accepts connections from any user (no authentication is performed) and provides the ability to retrieve and delete mail messages.  The POP3 user's name must be a valid email address which gets mapped back to the user's mail box folder on disk.  If there is no such folder then the user's mail box is considered empty.  Otherwise the files within the folder are enumerated and presented as the mail box contents for that user.
  • The server can be run standalone by running the Standalone.bat file.  In that case a cheesy little debugging dialog appears.
  • More commonly, the server is installed or uninstalled as an NT Service by running the Install.bat or Uninstall.bat file.

The code archive includes a Readme.txt file to explain how the program is used.  It also includes some rough sample code for sending/retrieving emails for testing purposes.

Using the Loopback Mail Server in a test

Here's what a typical test might do:

  • Create a WatiN IE browser instance.
  • Using WatiN, register a new user with a unique email address like where the domain has appropriates CNAME and MX records to point at the host where the Loopback Mail Server is running.
  • Using a POP3 client within the test (there's one included in the CleverComponents library), retrieve all emails for
  • Verify that the user received a notification from the system regarding registration.
  • Extract the registration verification link from the body of the email notification.
  • Using WatiN, navigate back to the site and complete the registration process.

You'll notice we didn't need to tell the Loopback Mail Server anything about  The system just sent an email to the account and it just worked.  Likewise we won't need to do anything to delete the account when we're done with it.  The Loopback Mail Server will take care of purging emails and mail folders 7 days after they are created (by default).


Experimenting with the Designer

Just for kicks since it was such a simple project I decided to see how much I could use the designer to build it in a RAD-friendly way (shh!).  The experience was something of a mixed bag.

Overall I found the discoverability and responsiveness of the designer to be quite poor.  I was often left scratching my head as to how to accomplish quite simple data binding tasks or as to why the components I needed weren't showing up in the toolbox.

Overall, using the designer was often more of a hindrance than a help.  Even on the GUI layout of the debugging dialog, I encountered a lot of friction.  I still can't imagine using it for anything but the simplest applications.  *sigh*

Free and Open-Source Alternatives

Unfortunately I wasn't able to find any actively developed and maintained free open-source SMTP and POP3 server implementations for .Net.  It's possible I just missed them.  Moreover it didn't need to be .Net at all.  It could have been Java, or C, or even Perl for all I care.

If you are interested in writing a mail server like this without any commercial dependencies or if you know of another one that is already available, please let me know!  I'd love to use it and bundle it up as some kind of function regression testing package add-on for MbUnit or STAF.

kick it on

Tuesday, October 23, 2007

But the xUnit Book Says...

The xUnit Book says a lot of things.  But it's not the last word on automated testing, not by a long stretch.

Steven Micham posted this comment in relation to my blog post about using dependency injection in a test framework:

I've also just skimmed through the xUnit Test Design Patterns book and the criticism of NUnit and therefore MbUnit with the [TestFixutreSetup] is exactly that the shared state is a 'bad thing' for constructing isolated test environments on a per-test basis.

Since this injection is basically shared state, how do you answer that?

You've got a good point.  So how do I answer that?

All that glitters is not unit testing.

Well, [TestFixtureSetUp] is obviously not a great foundation for building isolated test environments on a per-test basis.  We have [SetUp] for that.  However, there are cases when fixture-level isolation is desirable.

Here are a few:

  • The tests use resources that are expensive to initialize or to dispose.  For example, the tests might need to start / stop a partially stubbed out service with a high initialization overhead.
  • The tests involve a common computation that results in a lot of state that we want to verify systematically.  For example, the tests all consume the same input data that is produced by compilation.  Each test might examine this data in a different way so you get soluble, focused assertions across independent portions of the state instead of a single massive test with dozens of unrelated assertions within.
  • Or, when you're not writing unit tests...

Smells like integration testing.

Mmm... the musty odor of integration tests.  Reminds me of mothballs, old sweaters and damp wool.

Every test you write integrates some parts of the system.  The more you integrate into the set of components under tests, the more expensive your test will be, and the more difficult it will be to write.  Eventually you find yourself writing system tests (functional regression tests) and you wonder where all of the colour in the world went.

There's a bit of a change of scenery as you leave the comfortable realm of pure unit testing with mocks, stubs, and hard lines.  Integration testing is a vast desert: raw, wild, dangerous.  You can get hurt out there until you learn to adapt.

It's remarkably easy to encounter situations writing unit tests where you cannot exercise as much control over the environment as you'd like.  For example, the component under test might do some heavy lifting with the filesystem.  Often you can refactor your way out of these situations by stubbing out behavior or moving the heavy-weight concerns elsewhere where they are easier to test.  However, there is a point of diminishing returns.

So at some time you'll look at your suite of mocks and in desperation say: "I guess I'll need to create a real file on disk and make sure it gets deleted after the test finishes."  Ok.  So you do that and put the necessary code in [SetUp] and [TearDown].  And then you need a few more files...  Oh no, you've crossed over into integration testing.

Maybe you can refactor to improve the design.  Go ahead and try it!  Or... maybe not.  Crud.

Now what?

If you're going to write integration tests, do them for real!

It's a fact: integration tests are qualitatively different from unit tests.  The most dangerous thing you can do is to look at your tests and think: "these are just slightly ugly unit tests."  They're not really unit tests anymore.  If you try to treat them that way you'll just end up with "really ugly integration tests" and which is really bad.

So what do you do?

Well, first you want to be using a testing framework that degrades nicely when you reach the limits of white box unit testing.  As you write integration tests you're going to end up breaking a lot of unit testing rules.  You need a tool that can keep up with this transition.  Or you get a completely different tool.

In practice, MbUnit and NUnit both degrade fairly well for integration testing.  That's where being able to selectively violate test-level isolation by introducing shared fixture-level concerns is really handy.  Using a completely different tool is probably not worth the pain until you start doing something very different, like functional regression testing.

But... to help keep things straight, be sure to keep your unit tests and integration tests isolated.  Spend extra effort documenting those integration tests.  Intent revealing test method names might not be enough.  You'll need real documentation: possibly even paragraphs!

And that's ok.  We're not really doing unit testing anymore.  Even if we used a unit testing framework to implement the tests.  We crossed over (and it didn't hurt much).

I do care what the xUnit Book says.

Yes, I do care.  But the xUnit book is about unit testing.

The extra features MbUnit provides that other frameworks like xUnit.Net do not are there precisely to give programmers the freedom to selectively break the unit testing rules but not the tool.

MbUnit is intended to be used for more than just by-the-book unit testing.  So we offer support and guidance for those activities too.  We just don't expect the tool to enforce our recommendations to the exclusion of all else.

There's other good stuff out there.  Trust your instincts.

Constructor dependency injection in a test framework!

Nonsense?  Hardly!  (Translation: I'm doing it already and it's very useful.)

After reading the plans for xUnit.Net beta 2 that involve a cumbersome IUseFixture<T> interface type, I decided to pass on some of my ideas for MbUnit v3 in hopes that they will be of value to others.

Defining the problem

Problem: I've got some common state or behavior that I want to share across multiple tests within a fixture (or perhaps across multiple fixtures).


  • With MbUnit v2 or NUnit, just put the state on your fixture class and use [TestFixtureSetUp] and [TestFixtureTearDown].
  • With MbUnit v3, you can do that or you can create "mixins" that are defined using dependency injection on the fixture class and are first-class participants in the test suite.  Mixins are standalone classes that augment fixtures with custom behavior via attributes like [TestFixtureSetUp] or [SetUp] as usual but they can also contribute new tests, data sets, and other goodies.  Using a mixin is as simple as declaring a property, field, or constructor parameter of the mixin type on your fixture (or on another mixin).  Look for this feature in MbUnit v3 - Alpha 2 in a month or two.
  • With xUnit beta 2, your test class should implement IUseFixture<T> and put the necessary logic in T's constructor and Dispose.  But there are drawbacks to this approach.  One of them is that it results in additional boilerplate for providing setter methods to inject the shared instance.  Another is that the setter methods are obviously called after the constructor runs so you can't consume the shared instance as part of any constructor-based initialization.
  • Something else?

Constructor dependency injection to the rescue!

Another way to think of the problem is to imagine we are trying to inject services into the test fixture.  The service initialization and disposal can be taken care of by the framework per-test or per-fixture as required.  Let's see what constructor dependency injection can do here...

// Just an ordinary test fixture...
public class SomeTests
    private readonly IE ie;
    private readonly FixtureGlue glue;

    // The framework can use constructor dependency injection
    // to provide instances of a WatiN IE browser wrapper
    // and some custom FixtureGlue type as needed since they
    // are concrete types.  They also both happen to implement
    // IDisposable so that the framework can take care of
    // disposing them too.
    // So to inject concrete components like this into a test,
    // all we really need is to declare a constructor with the
    // right signature.
    public SomeTests(IE ie, FixtureGlue glue)
    { = ie;
        this.glue = glue;

    // Some ordinary test...
    public void MyTest()

        Assert.AreEqual("", ie.Url);

// Encapsulates some common behavior we want to run before
// and after all tests.
public class FixtureGlue : IDisposable
    public FixtureGlue()
        // configure something in the environment

    public void Dispose()
        // tear down whatever we did

However, we haven't said anything about when we get new instances of IE and FixtureGlue.

Fine-tuning the semantics

Should we create new instances of the IE and FixtureGlue types for each test or share them across all tests within the fixture?

In MbUnit, the expected answer would be probably to share these services because the fixture instance is reused all tests so its constructor will only run once (although in principle we could make this configurable).

In xUnit.Net, the situation is a little different.  xUnit.Net always constructs a new instance for each test.  So by default the test author will probably assume that the injected services are not reused.  However, that's precisely what we want...

To facilitate local reasoning about the tests, we can offer a custom attribute that is applied to the constructor's parameters to explicity state what's going on.  For xUnit.Net, we can add a [Fixture] attribute that states that the value of the associated parameter should be reused for each test in the fixture.  This usage of the word "fixture" seems pretty consistent with xUnit.Net nomenclature.

So our revised constructor declaration looks like this:

public class SomeTests
    // Initialized with a new instance of IE that is created
    // and disposed for each test and a shared instance of
    // FixtureGlue that is reused across all tests in the
    // fixture and disposed when finished.
    // So we can see that by default services are recreated
    // each for each test unless the [Fixture] attribute appears.
    public SomeTests(IE ie, [Fixture] FixtureGlue glue)
        // as before...

    // as before...

Obviously this idea can be taken much further.  Dependency injection can be used for parameterizing tests in plenty of other interesting ways and with more or less customization of the underlying process...

Spoiler: MbUnit v3 does all sorts of nifty stuff with dependency injection, particularly related to data-driven testing and to increasing the reusability of common test fixture extensions well beyond what can be achieved by subclassing.  Dependency injection has already proven to be immensely beneficial to me in the form of custom MbUnit v2 extensions for data-driven testing.  The plan is to go much much further.

kick it on

Thursday, October 18, 2007

Pain points

Roy wrote a great post about an anonymous company that apparently has serious communication, leadership and accountability problems.

Generally speaking, there are various “pain points” that currently prohibit the departments from either getting the full data they need, influence the current releases so they contain the correct feature set or be more productive in their work.

In a strange turn of events, yesterday I met someone on the way home who seem to have similar problem in her line of work.  The discussion centered primarily around the cost and apparent ineffectiveness of manual regression testing being performed "over there" in the outsourced QA departments.  I found it somewhat distressing overall.  Dollars and units of time were being weighed against an apparent need for more leadership, better internal processes, and improved flow of communication.  There may also be room for architectural improvement since it sounded like routine changes to products, unit price, and supply chain were requiring excessive regression testing and had a high defect rate.  I have too little technical information to guess what's going on there.

I'm all for work smarter not harder but I don't quite understand how work cheaper and faster is supposed to play out.  Fortunately the person I encountered was interested in improving internal processes to get there.  However I received a somewhat neutral response when I suggested leadership might also be required.  Moreover it's pretty clear that the outsourced workers are not really considered full members of the organization.  Not good but not too surprising.

All in all, I enjoyed listening to a 10,000ft summary of one person's internal perspective on the organizational difficulties of a large multi-national company.  I found that my outlook was somewhat more nuanced and biased towards the people involved whereas hers was rather categorical and biased towards the business demands.  That in itself was interesting.

My intention is not to judge, but these observations bear more thought.  I'm not accustomed to thinking on such scales.

I want an office with a sofa!

Monday's enhancement to my home office was a sofa.  It's just like the one pictured on the right but I opted for a deep red fabric instead of the cream color.  The ends fold down into sofa, chaise and cot configuration.

Anyways, I'm sure I can find more attractive sofas but this one was pretty much what I was looking for and at a price I was willing to pay.

No more clutter pile of boxes and dust bunnies in the corner of the room!

Now I've got a comfortable place to sit and think or to snuggle up with my wife and watch a movie on the flatscreen.  Ahh!  Now if only I could get one of these at the office...

Once upon a time I worked in an environment with lots of natural light, open space, and a central lounge that was actually in the middle of the space rather than tucked off in some undesirable corner.  Surprise, surprise, some of the most insightful conversations would occur on or around the couches in this conveniently located lounge space...

False sharing opportunities across modules.

I guess I'm really just writing this for me.

In one of my current projects, I have a case where two contributors on separate occasions have moved a class from one module to another (to increase reuse).  In each case I provided an explanation of why it was in its original location and requested that it be put back there again.  (I'm considering sticking a remark on the class to prevent further tampering...)

The explanation is a little long-winded.  Essentially it boils down to the fact that any apparent reusable potential of this particular class is illusory.  It declares the command-line options accepted by a tool along with help text and whatnot.  This is true despite the fact that several tools may offer similar options.  The problem is that the options aren't really identical.  The actual parsing code has been factored out to a common utility class and is reused.


Anyways, this got me thinking.  How could two very smart guys independently make the same mistake.  I think it has to do with modular design considerations that aren't always so obvious.

When new modules are created, it always happens that some things must reside inside the module and others must reside outside.  So module boundaries do constrain reuse and for good reason: it's to protect the encapsulation.  Opportunity for sharing the insides of modules should be viewed with suspicion until proven worthwhile lest they introduce undesirable coupling.  Perhaps more subtle is the case where the sharing opportunity exists only because of accidentally and rather arbitrary circumstances.  Watch out for those.

It's worth keeping in mind that redundancy in the small scale is often not very relevant to the overall architecture.  Outright duplication can exist across and within modules due to such diverse reasons as: isolated development groups, greatly varying rate of change or maturity of modules, historical carry-overs, inefficient cross-team communication, licensing constraints, conformance requirements, political tension, inexperience, carelessness.

Respecting my promise...

I had prepared a longer diatribe on this topic entitled DRY is not DAMP (Don't Repeat Yourself is not Disregard All Modular design Principles).  But I remember my promise to try not to be antagonistic...

Just the same, I'd like to point out one pet peeve of mine that recurs frequently in UI toolkits.  It's common for the Check Box and Radio Button controls to be represented by classes derived from a common ancestor like CheckBoxOrRadioButton.  After all, both controls essentially consist of a toggle-button and a label.  Right?

No, wrong. I believe the sharing in this case is illusory.

At best, I believe the shared implementation concerns should be captured by delegation rather than inheritance so they don't clutter up the API.  After all, Check Boxes and Radio Buttons have widely varying semantics.  The controls are not really substitutable anyways.  This is part of why I feel the common strategy of using inheritance to express composition of UI components is generally misguided.  (Or maybe I just like SWT more than WinForms.)

Interfaces should not leak implementation details to the surface simply by virtue of having exploited some opportunity to reuse a little code.  Each module should be considered a cohesive unit with a clear relationship to the whole and unburdened by inherited concerns that are of no relevance to its users.

In fact, sometimes there are very good reasons for skipping over an apparent opportunity to reuse code because it would do more harm than good.

Saturday, October 13, 2007

Gallio Progress

There's something deeply satisfying about this screenshot. Here we can see MbUnit v3, MbUnit v2, NUnit and xUnit.Net all side-by-side in the same test runner. Not bad.

Edit: I just noticed that I mislabeled the xUnit.Net framework in there. It should be using the official name of the project rather than Xunit. That's fixed now.

Friday, October 12, 2007

Software Automation Wizard in San Francisco

Ingenio is hiring software automation wizards.

Self-Operating Napkin (Rube Goldberg) On a selfish note, I'm looking for someone to work with me to use and grow our test automation platform and solve challenging integration testing problems.  There is a lot of creative programming involved as we write and maintain these tests and the systems that run them.

Software Automation Wizard in San Francisco

Do you enjoy writing high-quality code using the latest tools?  Do you hate bugs?  Do you like solving challenging system integration problems?

Ingenio's automation team is looking for engineers to develop a comprehensive and maintainable test suite for Ingenio's many products and diverse features.  We'll bring you up to speed with the latest techniques writing automated tests in C# with MbUnit, WatiN, and other tools.

Our system consists of a mixture of web, telephony and data platforms.  It takes a lot of creativity to make sure all of the pieces work together to provide a compelling end-user experience.  There is no out-of-the-box solution.  In fact, we have contributed extensively to several open source projects to help solve this very problem.

What we will look for in you:

  • You love to program!  You have most likely written stuff in various languages.  You already know at least one object-oriented programming language and are interested in learning C# (if you don't already know it and a dozen others).  You have opinions on why one language can be better than another for some task.  You’re not afraid of reading the manual and experimenting to learn about more powerful language features.
  • You love to program so much that you have a 4 year degree in computer science or engineering.
  • You have a passion for software testing.  Breaking things to figure out how they work is something you have done for a long time – even before you thought you could get paid for it.  You have probably sent complaints to the various web sites you use when you do something odd on the site and it doesn’t work as expected.  Maybe you already have some experience writing tests using MbUnit, JUnit, WatiN, WatiR, Selenium or other tools.
  • You have great communication skills.  You will often work with other engineers, program managers, and business people.  Conversation comes easy to you no matter who you’re talking to.
  • You are highly motivated.  In fact, perhaps you haven’t even heard of WaitN but by the time you have contacted us you have already written some cool stuff with it.

The Social Enterprise

A couple of days ago, Ayende wrote:

Enterprise Software - A common term used to refer to systems that usually run a core part of a business.  Often those are mission critical systems, which interact with many other systems in the organization in many interesting ways.  Enterprise software is considered hard because it often need to mix technological capabilities with the business logic, and be flexible to handle changing conditions as the business evolves.

I feel there's something very important missing from this definition.

Where are the people?

Enterprise software is hard not just because of gnarly technical system integration issues related to core business processes but because it has to be very carefully tailored to fit the social environment of the enterprise.

I'd argue that the social challenges of enterprise software greatly outweigh the technical considerations.  If the system does not somehow relate to people, it is not an enterprise system.

Enterprise systems generally incorporate a very large installed base of internal and external stakeholders that have widely varying perspectives and needs.  These people are accustomed to a particular conception of the business operating environment.  They have developed a specialized vocabulary to describe their doings.  They have adopted work habits to get them through the day.  To observers they often appear to behave in strange, mysterious and chaotic ways.  Changes run the risk of inducing systematic paralysis, shock, mass-hysteria and cannibalism.

We're going to build systems for them?!?  You must be out of your mind.

The stakeholders of enterprise systems are tightly coupled to the system itself.  They are a part of the machine and they exhibit a sensitive dependence on its behavior (and are often quite resistant to change).  Building and maintaining enterprise systems involves performing a carefully unscripted dance around the social, political, financial, legal and quasi-religious spheres of the organization.

The actual code seems almost irrelevant!  "Business logic" is a formalized contortion of the whimsical demands of a fractious collective of fanatics, individualists, providers, consumers, and parasites.

The true horror of enterprise systems can only be revealed in the actions and sentiments of people.  They are your business.  That's what's so hard.

Sometime I'll get around to posting my ALT.Net reflections...

Friday, October 5, 2007

ALT.Net Conference.

I'll be heading down to Austin, Texas for the ALT.Net Conference this weekend.  I'm very much looking forward to meeting everyone and seeing what you're all up to.  I'm also planning to give a little demonstration of MbUnit Gallio work in progress.

Should be lots of fun!