What do you do when you have lots of automated integration tests?
Integration tests are different from unit tests. They are developed by different people for different purposes with different priorities, different runtime characteristics, different maintenance overhead, different reporting needs, and different management demands.
Unit tests should be fast, sleek and isolated. They're the Formula 1 racers of the test world. They whiz around the track faster than you can say Continuous Integration!
Integration tests should be comprehensive, robust and realistic. They're methodical and avoid taking shortcuts. If it takes 5 minutes to simulate a particular kind of transaction then they will spend 5 minutes doing it for real. (To be fair, I'm talking about black box functional regression tests here.)
The Right Tool for the Job
These tests are largely intended to displace manual QA activities for acceptance testing purposes. Consequently they are integration tests of the highest order: they interact with the web site and other system components as nearly as real people would. But much faster and more accurately.
Still, they take a long time to run. Running them interactively is completely out of the question. Running them serially is painfully inefficient. Running them only on demand wastes a valuable opportunity to generate Continuous Feedback from the system and to spot non-deterministic bugs.
Moreover these tests really belong to the QA arm of our organization. The operators are QA, Operations and Engineering staff. The consumers are all of the above plus Program Management and other stakeholders.
So all told, our integration testing needs are vastly different from our unit testing needs. So we need different tools.
If There Is No Suitable Tool, Build It
For the past few months I have been thinking about building a tool to assist with deploying, running and managing automated test cases.
There are a lot of very interesting problems to solve, particularly in the user interface and overall workflow model. Consider that many users of the system will not be test authors themselves. Results must be meaningful, reliable, and easy to interpret.
Here's a sketch of what I have in mind for this new addition to the Gallio tool chain: Archimedes.
Archimedes will incorporate:
- A distributed test runner.
- A web-based management console.
- A repository for historical test data.
- A deployment engine.
- A scheduler.
- A notification service.
- A reporting, trending and statistics module.
- A data feed and web service for mash ups.
Archimedes will leverage and extend the Gallio platform. It will work with all supported test frameworks such as MbUnit, NUnit, xUnit.Net, MSTest, NBehave, and others yet to come.
Sound Like Fun?
Reply in comments or send an email to the Gallio-Dev Mailing List.