Thursday, April 10, 2008

Gallio Archimedes

What do you do when you have lots of automated integration tests?

Integration tests are different from unit tests.  They are developed by different people for different purposes with different priorities, different runtime characteristics, different maintenance overhead, different reporting needs, and different management demands.

Unit tests should be fast, sleek and isolated.  They're the Formula 1 racers of the test world.  They whiz around the track faster than you can say Continuous Integration!

Integration tests should be comprehensive, robust and realistic.  They're methodical and avoid taking shortcuts.  If it takes 5 minutes to simulate a particular kind of transaction then they will spend 5 minutes doing it for real.  (To be fair, I'm talking about black box functional regression tests here.)

The Right Tool for the Job

Where I work, we have lots of automated integration tests.  These are built by the QA Automation Team in C# using MbUnit, Gallio and WatiN.

These tests are largely intended to displace manual QA activities for acceptance testing purposes.  Consequently they are integration tests of the highest order: they interact with the web site and other system components as nearly as real people would.  But much faster and more accurately.

Still, they take a long time to run.  Running them interactively is completely out of the question.  Running them serially is painfully inefficient.  Running them only on demand wastes a valuable opportunity to generate Continuous Feedback from the system and to spot non-deterministic bugs.

Moreover these tests really belong to the QA arm of our organization.  The operators are QA, Operations and Engineering staff.  The consumers are all of the above plus Program Management and other stakeholders.

So all told, our integration testing needs are vastly different from our unit testing needs.  So we need different tools.

If There Is No Suitable Tool, Build It

For the past few months I have been thinking about building a tool to assist with deploying, running and managing automated test cases.

There are a lot of very interesting problems to solve, particularly in the user interface and overall workflow model.  Consider that many users of the system will not be test authors themselves.  Results must be meaningful, reliable, and easy to interpret.

Here's a sketch of what I have in mind for this new addition to the Gallio tool chain: Archimedes.

Archimedes will incorporate:

  • A distributed test runner.
  • A web-based management console.
  • A repository for historical test data.
  • A deployment engine.
  • A scheduler.
  • A notification service.
  • A reporting, trending and statistics module.
  • A data feed and web service for mash ups.
  • Documentation.

Archimedes will leverage and extend the Gallio platform.  It will work with all supported test frameworks such as MbUnit, NUnit, xUnit.Net, MSTest, NBehave, and others yet to come.

Sound Like Fun?

Reply in comments or send an email to the Gallio-Dev Mailing List.

Technorati Tags: ,


Julián Hidalgo said...

Sounds great!
Well, and ambitious too, but I'm sure you'll make it come true. I hope to be able to collaborate with a few features at least....

Anonymous said...

This sounds absolutely wonderful, and your list looks good so far except for perhaps on addition that should (IMHO) be made explicit:

Test server provisioning hooks and, if possible, standard implementations (eg. against VMWare Server)

Most of the QA teams I've worked with would very much like to see the build deployed to and tested against one or more baseline VM images, not to a physical or virtual machine that has (or ever had) the previous build installed on it. By "one or more" I mean to suggest supporting configurations such as are required to test across multiple operating system version, multiple framework versions, and/or client on one machine with server on another, etc.

These kinds of permutations get pretty complicated pretty fast, so it would be completely acceptable to only provide hooks and to have provisioning accounted for in the various other features. As such, each team could hook in and provision out the permutation they require for each test, using their virtualization technology (and/or true physical machine provisioning) of choice.

Grumpy Grandma said...

Wow! that list is awesome!

MbUnit is Unit testing on crack, steriods and ice at the same time!

I can't wait to see the involuntarily twitching, borderline psychopathic beta in all it's hepped up glory.

Jeroen said...

Hi Jeff,

This would speed up live as we know it! Just today I came across

Looks like an attempt to do this as well. Just for your info.

I'll see if I have some time to help out cause it is very tempting.

Andres Gonzalo said...


Unknown said...

I'm very interested in this project. I would like to know what status of Archimedes is? When do you plan to release it?