Today we have released Gallio v3.0 Alpha 3, including a preview of MbUnit v3. 
The installer is available here: GallioBundle-3.0.0.247-Setup.exe (temporary link)
It will also be available from the Gallio website shortly: www.gallio.org.
There has been quite a lot of progress in this release including new features, bug fixes, and various internal changes to make it easier for runner and framework authors to build new cool stuff on top of Gallio.
Here is a quick summary of these changes.
New and Notable
New Features
1. Annotations
Annotations are a means for test frameworks to communicate errors, warnings and informational messages to the user in association with particular code elements.
MbUnit v3 uses annotations to inform the user about improperly used attributes and to highlight points of interest such as ignored or pending tests.
Annotations are displayed as messages in test reports.
They manifest as error/warning stripes in R#.
Graham Hay has also added an Annotation view in the Icarus GUI. Double-clicking on the annotation displays the source code.
2. Report Enhancements
HTML reports now include navigation stripes to provide a quick overview of the contents of the report and to navigate to failed, inconclusive and skipped tests.
When JavaScript is enabled, test results are automatically collapsed for ease of reading. They are automatically expanded during navigation.
HTML reports can grow to become quite large when there are many tests. Accordingly, we have added a "Condensed" version of this report that omits passing tests. This is particularly useful in combination with CruiseControl.Net.
3. Icarus Test Execution Log
Graham Hay has added a new Execution Log view to Icarus. You can view test results here while tests execute. If you periodically refresh the view then you can watch test execution in near real-time including output from long-running tests that are still in flight.
4. Test Assembly Configuration Files
Gallio now provides all of the machinery required to support test assembly configuration files. All frameworks and adapters can benefit from this support, including MbUnit v3. Just create a configuration file named: "MyTestAssembly.dll.config" and put it next to your test assembly (assuming it is called MyTestAssembly).
5. MSTest Adapter
Julian Hidalgo implemented an adapter for MSTest. This feature enables all Gallio runners to run MSTest tests. One of the interesting consequences of this design is that R# v3.1 is now capable of viewing and running MSTest tests via Gallio.
6. Visual Studio Team System Integration (Experimental)
Gallio now includes an experimental ITip provider for integration with Visual Studio Team System 2008.
To use it, be sure to select the feature in the Gallio Installer. It is currently excluded from installation by default because of its experimental nature.
Then just load up your favorite test project in Visual Studio 2008 and run a Build. The tests should now appear in the Visual Studio Test View. You can then run them from there. It should work with all supported test frameworks.
As far as I know, we are the first Open Source testing project to provide this kind of integration.
If would like to contribute to the development of this feature please let me know!
7. Test Runner Extensions
Many new features can be added to Gallio by implementing plug-ins. However, some users have requested the ability to extend test runners to change how test results are reported.
Gallio Alpha 2 included an ITestMonitor interface that roughly served this purposes by making it possible to inject additional behavior into the test runner's event handlers. In Gallio Alpha 3, we have introduced the ITestRunnerExtension interface and provided a mechanism for loading extensions into the runners by means of an additional argument.
Implementing a test runner extension is straightforward:
- Create a new assembly for your extension.
- Add a reference to Gallio.dll.
- Create a subclass of TestRunnerExtension.
- In the Initialize method, register event handlers on the Events object provided to the extension.
- Build your assembly.
- Copy the file wherever you like.
Using a test runner extension is similar to registering an MSBuild logger.
- If you are using Gallio.Echo.exe (the command-line test runner), add an argument like: "/extension:MyFancyExtension,MyFancyExtension.dll".
- If you are using the Gallio NAnt or MSBuild tasks, the process is similar. Refer to the documentation of these tasks for details.
- There is a built-in extension for debugging. Add "/extension:DebugExtension,Gallio" or an equivalent argument to enable it. Be sure to also set the message verbosity to Debug.
The plan is to open up more of the platform to lightweight extensions of this sort.
8. Parameterized Test Enhancements
MbUnit v3 now fully supports generic test fixtures and test methods and various other parameterizations.
A test can now be parameterized in any of the following ways:
- Adding a method parameter to a test method.
- Adding a generic method parameter to a test method.
- Adding a constructor parameter to a test fixture.
- Adding a field to a test fixture.
- Adding a writable property to a test fixture.
- Adding a generic type parameter to a test fixture.
Correspondingly data-source attributes can be specified on any of the following code elements:
- Test methods.
- Test method parameters.
- Test method generic parameters.
- Test fixture types.
- Test fixture constructors.
- Test fixture constructor parameters.
- Test fixture fields.
- Test fixture properties.
- Test fixture generic parameters.
So there are many choices, but each choice is quite simple.
Here are a few interesting combinations.
A basic "Row"-test.
Note: The [RowTest] attribute is not needed anymore and has been eliminated.
[TestFixture]
public class Fixture
{
[Test]
[Row(3, 4, 5, true)]
[Row(1, 2, 3, false)]
public void Test(int a, int b, int c, bool expectedResult)
{
Assert.AreEqual(expectedResult, Math.IsPythagoreanTriple(a, b, c));
}
}
A generic "Type"-test.
Note: The [TypeFixture] attribute is not needed anymore and has been eliminated.
[TestFixture]
[Row(typeof(List<int>))]
[Row(typeof(LinkedList<int>))]
public class Fixture<T> where T : IList<int>, new()
{
[Test]
public void Test()
{
T list = new T();
list.Add(123);
Assert.AreEqual(1, list.Count);
}
}
A combinatorial test.
Note: The [CombinatorialTest] attribute is not needed anymore and has been eliminated.
[Test]
public void Test([Column(0, 1)] int a, [Column("a", "b")] string b, [Column(false, true)] bool c)
{
// Test something with the specified combination.
// Will be executed 8 times total.
}
A pairwise test.
Only tests all combinations of pairs of values. Can yield a significant reduction in test cases compared to ordinary combinatorial tests while still obtaining great coverage. See also: www.pairwise.org.
[Test, PairwiseJoin]
public void Test([Column(0, 1)] int a, [Column("a", "b")] string b, [Column(false, true)] bool c)
{
// Test something with the specified combination.
// Will be executed only 4 times instead of 8.
}
A CSV data source.
Just one of many external data sources to be implemented.
[Test]
[CsvData(ResourcePath="CsvDataTest.csv", HasHeader=true)]
public void ImplicitlyScopedResourceWithHeader(decimal price, string item)
{
Log.WriteLine("{0}: {1}", item, price);
}
Automatic data type conversions.
Can automatically convert strings into DateTimes and doubles into decimals for example.
Architecture Improvements
There were a number of internal architecture improvements in Gallio Alpha 3. Here's a quick summary.
1. Unified Hosting Apparatus
In Gallio Alpha 2 there was some duplication between the responsibilities of the IHost and ITestDomain components. These abstractions have been refactored. As a result, ITestDomain was replaced by ITestDriver to handle the specific needs of setting up a testing environment.
The IHost abstraction provides a mechanism for Gallio to run code remotely such as in an isolated AppDomain or another process.
2. Pattern Test Framework Refactoring
There was an important piece missing from the data binding story in Alpha 2 that prevented certain kinds of parameterized tests from being expressed. This has been resolved by refactoring of the Pattern Test Framework (from which MbUnit v3 is derived) and introducing the concept of a test data context. IPattern is significantly simpler now.
This work also prompted the introduction of the Annotations mechanism when it became clear that we needed a much more direct mechanism for reporting semantic errors in a test to the user.
3. Reflection Policy Enhancements
Introduced the concept of a typed ConstantValue for performing reflection over unevaluated attribute values. For example, when populating tests for the ReSharper add-in we may encounter attribute values that are of type Type. It's quite possible for those types to not be loaded in the runtime (since we're performing reflection against an in-memory code model). So instead we can obtain a ConstantValue that represents the Type without actually loading it.
Gallio Alpha 3 also includes a Cecil-based reflection policy for scanning disk-based assemblies without loading them into the runtime. The Visual Studio Team System add-in uses this policy to perform reflection without locking assemblies or introducing memory leaks.
4. Test Runner Redesign
The ITestRunner interface has been completely redesigned to make it easier to use and more versatile. For one, the reporting functionality has now been integrated somewhat more tightly into the runner itself. An ITestRunnerExtension can register new behaviors on a test runner using the ITestRunnerEvents interface.
5. Aggregate Test Drivers
Supporting test assembly configuration files is not as easy as it sounds. For one, it requires us to create a separate AppDomain for each test assembly. Then we have to somehow merge the test model and results.
Gallio Alpha 3 introduces an aggregate test driver to help with this. However, the current design is still somewhat in flux. I'd appreciate some feedback on how we can improve and simplify the test framework integration story overall.
Bugs Fixed
Here are a few of the bugs from Alpha 2 that have been fixed:
- Icarus now unloads assemblies promptly to avoid locking them. You can also enable shadow-copy mode for improved efficiency.
- MbUnit v2 version compatibility problems fixed. Should work with any version of MbUnit v2.4+.
- Hung Gallio.Host.exe processes are no more.
- xUnit.Net adapter has been upgraded to v1.0 RTM.
- Installer PATH variable corruption.
- Castle dependencies have been internalized to avoid version mismatches.
- Default thread apartment state is STA and can be overridden with the [ApartmentState] attribute.
- [Explicit] tests are now silent and do not display any output in reports unless explicitly selected, not even a message to say they are being skipped. If you want noisy output, use [Ignore], [Pending] or [Annotation] instead.
