Tuesday, May 13, 2008

Announcing Gallio v3.0 Alpha 3

Today we have released Gallio v3.0 Alpha 3, including a preview of MbUnit v3.  Gallio Leaf

The installer is available here: GallioBundle-3.0.0.247-Setup.exe (temporary link)

It will also be available from the Gallio website shortly: www.gallio.org.

There has been quite a lot of progress in this release including new features, bug fixes, and various internal changes to make it easier for runner and framework authors to build new cool stuff on top of Gallio.

Here is a quick summary of these changes.

New and Notable

New Features

1. Annotations

Annotations are a means for test frameworks to communicate errors, warnings and informational messages to the user in association with particular code elements.

MbUnit v3 uses annotations to inform the user about improperly used attributes and to highlight points of interest such as ignored or pending tests.

Annotations are displayed as messages in test reports.

image

They manifest as error/warning stripes in R#.

image

Graham Hay has also added an Annotation view in the Icarus GUI.  Double-clicking on the annotation displays the source code.

image

2. Report Enhancements

HTML reports now include navigation stripes to provide a quick overview of the contents of the report and to navigate to failed, inconclusive and skipped tests.

When JavaScript is enabled, test results are automatically collapsed for ease of reading.  They are automatically expanded during navigation.

image

HTML reports can grow to become quite large when there are many tests.  Accordingly, we have added a "Condensed" version of this report that omits passing tests.  This is particularly useful in combination with CruiseControl.Net.

image 

3. Icarus Test Execution Log

Graham Hay has added a new Execution Log view to Icarus.  You can view test results here while tests execute.   If you periodically refresh the view then you can watch test execution in near real-time including output from long-running tests that are still in flight.

 image

4. Test Assembly Configuration Files

Gallio now provides all of the machinery required to support test assembly configuration files.  All frameworks and adapters can benefit from this support, including MbUnit v3.  Just create a configuration file named: "MyTestAssembly.dll.config" and put it next to your test assembly (assuming it is called MyTestAssembly).

5. MSTest Adapter

Julian Hidalgo implemented an adapter for MSTest.  This feature enables all Gallio runners to run MSTest tests.  One of the interesting consequences of this design is that R# v3.1 is now capable of viewing and running MSTest tests via Gallio.

image

6. Visual Studio Team System Integration (Experimental)

Gallio now includes an experimental ITip provider for integration with Visual Studio Team System 2008.

To use it, be sure to select the feature in the Gallio Installer.  It is currently excluded from installation by default because of its experimental nature.

Then just load up your favorite test project in Visual Studio 2008 and run a Build.  The tests should now appear in the Visual Studio Test View.  You can then run them from there.  It should work with all supported test frameworks.

image

As far as I know, we are the first Open Source testing project to provide this kind of integration.

If would like to contribute to the development of this feature please let me know!

7. Test Runner Extensions

Many new features can be added to Gallio by implementing plug-ins.  However, some users have requested the ability to extend test runners to change how test results are reported.

Gallio Alpha 2 included an ITestMonitor interface that roughly served this purposes by making it possible to inject additional behavior into the test runner's event handlers.  In Gallio Alpha 3, we have introduced the ITestRunnerExtension interface and provided a mechanism for loading extensions into the runners by means of an additional argument.

Implementing a test runner extension is straightforward:

  1. Create a new assembly for your extension.
  2. Add a reference to Gallio.dll.
  3. Create a subclass of TestRunnerExtension.
  4. In the Initialize method, register event handlers on the Events object provided to the extension.
  5. Build your assembly.
  6. Copy the file wherever you like.

Using a test runner extension is similar to registering an MSBuild logger.

  • If you are using Gallio.Echo.exe (the command-line test runner), add an argument like: "/extension:MyFancyExtension,MyFancyExtension.dll".
  • If you are using the Gallio NAnt or MSBuild tasks, the process is similar.  Refer to the documentation of these tasks for details.
  • There is a built-in extension for debugging.  Add "/extension:DebugExtension,Gallio" or an equivalent argument to enable it.  Be sure to also set the message verbosity to Debug.

The plan is to open up more of the platform to lightweight extensions of this sort.

8. Parameterized Test Enhancements

MbUnit v3 now fully supports generic test fixtures and test methods and various other parameterizations.

A test can now be parameterized in any of the following ways:

  • Adding a method parameter to a test method.
  • Adding a generic method parameter to a test method.
  • Adding a constructor parameter to a test fixture.
  • Adding a field to a test fixture.
  • Adding a writable property to a test fixture.
  • Adding a generic type parameter to a test fixture.

Correspondingly data-source attributes can be specified on any of the following code elements:

  • Test methods.
  • Test method parameters.
  • Test method generic parameters.
  • Test fixture types.
  • Test fixture constructors.
  • Test fixture constructor parameters.
  • Test fixture fields.
  • Test fixture properties.
  • Test fixture generic parameters.

So there are many choices, but each choice is quite simple.

Here are a few interesting combinations.

A basic "Row"-test.

Note: The [RowTest] attribute is not needed anymore and has been eliminated.

[TestFixture]
public class Fixture
{
    [Test]
    [Row(3, 4, 5, true)]
    [Row(1, 2, 3, false)]
    public void Test(int a, int b, int c, bool expectedResult)
    {
        Assert.AreEqual(expectedResult, Math.IsPythagoreanTriple(a, b, c));
    }
}

A generic "Type"-test.

Note: The [TypeFixture] attribute is not needed anymore and has been eliminated.

[TestFixture]
[Row(typeof(List<int>))]
[Row(typeof(LinkedList<int>))]
public class Fixture<T> where T : IList<int>, new()
{
    [Test]
    public void Test()
    {
        T list = new T();
        list.Add(123);
        Assert.AreEqual(1, list.Count);
    }
}

A combinatorial test.

Note: The [CombinatorialTest] attribute is not needed anymore and has been eliminated.

[Test]
public void Test([Column(0, 1)] int a, [Column("a", "b")] string b, [Column(false, true)] bool c)
{
    // Test something with the specified combination.
    // Will be executed 8 times total.
}

A pairwise test.

Only tests all combinations of pairs of values.  Can yield a significant reduction in test cases compared to ordinary combinatorial tests while still obtaining great coverage.  See also: www.pairwise.org.

[Test, PairwiseJoin]
public void Test([Column(0, 1)] int a, [Column("a", "b")] string b, [Column(false, true)] bool c)
{
    // Test something with the specified combination.
    // Will be executed only 4 times instead of 8.
}

A CSV data source.

Just one of many external data sources to be implemented.

[Test]
[CsvData(ResourcePath="CsvDataTest.csv", HasHeader=true)]
public void ImplicitlyScopedResourceWithHeader(decimal price, string item)
{
    Log.WriteLine("{0}: {1}", item, price);
}

The data itself should be stored in an embedded resource called "CsvDataTest.csv".  We just create a file with that name in the same folder as the test class and then set its Build Action property to Embedded Resource in Visual Studio.

Item, Price
Apples, 1.00
Bananas, 1.50
Cookies, 2.00

Automatic data type conversions.

Can automatically convert strings into DateTimes and doubles into decimals for example.

Architecture Improvements

There were a number of internal architecture improvements in Gallio Alpha 3.  Here's a quick summary.

1. Unified Hosting Apparatus

In Gallio Alpha 2 there was some duplication between the responsibilities of the IHost and ITestDomain components.  These abstractions have been refactored.  As a result, ITestDomain was replaced by ITestDriver to handle the specific needs of setting up a testing environment.

The IHost abstraction provides a mechanism for Gallio to run code remotely such as in an isolated AppDomain or another process.

2. Pattern Test Framework Refactoring

There was an important piece missing from the data binding story in Alpha 2 that prevented certain kinds of parameterized tests from being expressed.  This has been resolved by refactoring of the Pattern Test Framework (from which MbUnit v3 is derived) and introducing the concept of a test data context.  IPattern is significantly simpler now.

This work also prompted the introduction of the Annotations mechanism  when it became clear that we needed a much more direct mechanism for reporting semantic errors in a test to the user.

3. Reflection Policy Enhancements

Introduced the concept of a typed ConstantValue for performing reflection over unevaluated attribute values.  For example, when populating tests for the ReSharper add-in we may encounter attribute values that are of type Type.  It's quite possible for those types to not be loaded in the runtime (since we're performing reflection against an in-memory code model).  So instead we can obtain a ConstantValue that represents the Type without actually loading it.

Gallio Alpha 3 also includes a Cecil-based reflection policy for scanning disk-based assemblies without loading them into the runtime.  The Visual Studio Team System add-in uses this policy to perform reflection without locking assemblies or introducing memory leaks.

4. Test Runner Redesign

The ITestRunner interface has been completely redesigned to make it easier to use and more versatile.  For one, the reporting functionality has now been integrated somewhat more tightly into the runner itself.  An ITestRunnerExtension can register new behaviors on a test runner using the ITestRunnerEvents interface.

5. Aggregate Test Drivers

Supporting test assembly configuration files is not as easy as it sounds.  For one, it requires us to create a separate AppDomain for each test assembly.  Then we have to somehow merge the test model and results.

Gallio Alpha 3 introduces an aggregate test driver to help with this.  However, the current design is still somewhat in flux.  I'd appreciate some feedback on how we can improve and simplify the test framework integration story overall.

Bugs Fixed

Here are a few of the bugs from Alpha 2 that have been fixed:

  • Icarus now unloads assemblies promptly to avoid locking them.  You can also enable shadow-copy mode for improved efficiency.
  • MbUnit v2 version compatibility problems fixed.  Should work with any version of MbUnit v2.4+.
  • Hung Gallio.Host.exe processes are no more.
  • xUnit.Net adapter has been upgraded to v1.0 RTM.
  • Installer PATH variable corruption.
  • Castle dependencies have been internalized to avoid version mismatches.
  • Default thread apartment state is STA and can be overridden with the [ApartmentState] attribute.
  • [Explicit] tests are now silent and do not display any output in reports unless explicitly selected, not even a message to say they are being skipped.  If you want noisy output, use [Ignore], [Pending] or [Annotation] instead.
Technorati Tags: ,

7 comments:

Stlan said...

Great job! I like the way it integrates into VSTS.

Jeremy Gray said...

I'm loving the direction that Gallio is headed in and the progress that is being made but if I may make one suggestion:

Those reports need some serious reformatting into a tabular form that can be scanned more easily than the samples shown in this blog post, with their coloured bars and stats jumping left and right with every line.

Jeff Brown said...

@Jeremy
I agree we can do better here. I think a tabular layout is a little more difficult because test runs are hierarchically structured. Not sure. It might be enough to right-align the color bars and omit the stats.

Anyways, please let us know what you think. If you've got some specific layout ideas try editing the report XSLT or sending us a mockup.

Jim Burger said...

Im also very impressed, although F# support would be nice. xUnit.NET provides for some very clean syntax, yet is undiscoverable by the gallio runners. E.G
#light
open Xunit

[Fact>]
let this_is_a_test() =
Assert.True(true)

Jeff Brown said...

@Jim
Have you tried using xUnit.Net 1.0 RTM? Unfortunately the current xUnit adapter for Gallio is version sensitive.

Jeremy Gray said...

@Jeff - You're quite right in that even a simple right alignment of the fixed width information (the bar and the counts) would do the trick quite nicely.

Keep up the awesome work. I can't quite bring the alpha version into my current engagement but as soon as you guys hit beta I'll be all over it. Gallio is bringing many exciting things to the table and I'm really looking forward to it.

Kent said...

Are there any plans to integrate with the code coverage tool in VSTS as well?