Announcing Gallio and MbUnit v3.0.5

Today we are releasing Gallio and MbUnit v3.0.5.  This is primarily a bug fix release but we are introducing a few new features.

Assert.AreApproximatelyEqual, test factories, structural object formatting, AutoCAD integration, x86 tests on x64, Visual Studio CTP 2010.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Release notes: All release notes from all versions.

MbUnit

Assert.AreApproximatelyEqual

We have added Assert.AreApproximatelyEqual to express an assertion about equality within some delta (aka. tolerance).  Likewise there is Assert.AreNotApproximatelyEqual.

Examples:

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.3);
-> Success: 2.2 is in the range 2.0 +/- 0.3.

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Failure: 2.2 is not in the range 2.0 +/- 0.1.

Assert.AreNotApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Success: 2.2 is not in the range 2.0 +/- 0.1.

The implementation supports any comparable type that defines a subtraction operator, and for which the result of subtraction is also comparable.

Examples:

Assert.AreApproximatelyEqual(-5, 2, 10);
-> Success: 2 is in the range -5 +/- 10.

Assert.AreApproximatelyEqual(new DateTime(2008, 11, 17), new DateTime(2008, 11, 18), TimeSpan.FromDays(2));
-> Success: 2008/11/18 is in the range 2008/11/17 +/- 1 day.

Static and Dynamic Test Factories (aka. Test Suites)

Some people were wondering where the MbUnit v2 [TestSuite] mechanism went.  Now it's back with a few interesting twists.

A test factory in MbUnit v3 is a method that generates a test suite programmatically.  They are typically used when the structure of a test is derived from custom metadata.  For example, a test factory might produce a test suite generated from some information stored in the database.

A test suite produced by a test factory differs from a standard data-driven test (such as a Row-test) in that the tester has more control over the generated structure of the test suite.  Instead of plugging in values to a parameterized test, whole new tests can be created arbitrarily.

There are two kinds of test factories.

• A static test factory adds new tests to the static test tree.  This makes the generated test suite part of the static test model that UI-based test runners such as Icarus use to present tests to the user.
  
  Pros: The structure of the test suite is determined before tests actually run.
  
  Cons: The structure of the test suite is fixed at test exploration time.  Moreover, any data that was needed to produce the tests will be loaded all at once up front regardless of whether the test suite will actually be executed.  Also, for this mechanism to work at all, the code that defines the factory must be executable at test exploration time.  This is not the case in ReSharper or Visual Studio Test Tools so static test suites will not appear there, alas.  (At least not as things are written today...)
  
• A dynamic test factory adds new tests to the dynamic test tree, aka. the test step tree.  The tests are not included in the static test model which makes it impossible for the user to pick a subset of the dynamic tests to run (basically they are all-or-nothing).
  
  Pros: The structure of the test suite is determined when the fixture that contains it runs so it can vary based on input data determined at runtime.   The test factory can be parameterized so it can produce multiple test suites with different input data.  Likewise, a dynamic test factory can emit an unbounded number of tests.  It can keep on producing tests to run for as long as it likes.
  
  Cons: The dynamic tests are not visible to the user until test execution time.

It might help to think of a static test factory as working something like a [TestFixture] or [Test] where the test suite is constructed ahead of time using an algorithm of your choice.  On the other hand, a dynamic test factory is more like TestStep.Run which is used to create dynamic test steps within a running test or like parameterized tests which pull contents from a data source at runtime.

Clear as mud huh?  How about some examples.

Static Test Factory Example

In this example, we create a few silly tests statically in the CreateTests static method.  Notice that it returns an enumeration of Test objects.  It is also static because no instance of the fixture exists at the time this method is called.  It follows that we cannot define a static test factory as a member of a generic test fixture because we won't know what values are intended to be plugged into the fixture's type parameters.

There are currently 3 kinds of built-in Test subclasses (you can make your own too):

• A TestCase is a kind of Test that contains executable code to run.  It can have a name, description and other custom metadata.
• A TestSuite is another kind of Test that composes a list of other Tests (including other suites) to run.  It can also have a name, description, and other custom metadata.  In addition to its children, it can also include SetUp, TearDown, SuiteSetUp, and SuiteTearDown functions.
• A TestFixtureReference is yet another kind of Test.  We do not show it here but basically it allows you to create test suites that reference .

 

When we run the test fixture, it will include a test called "Sample Test" as well as a suite called "Sample Suite" with two silly test cases within.

Dynamic Test Factory Example

A dynamic test factory is quite similar to a static test factory.  It just runs at runtime instead.

The first thing you will notice is that this dynamic test factory is not declared static.  It can be if you like, but that's optional.  The second thing is that this test factory can actually be parameterized!

Here we create 3 groups of dynamically generated test cases using different input data each time.

When we run the test fixture, it will produce 1 test case when n = 1, 4 test cases when n = 2 and 9 test cases when n = 3.

Of course this example is silly.  I suppose next time I could write up something like a miniature file-based DSL integrated using test factories.

Structural Object Formatting

How often has this happened to you?

So you write this test...

And then it fails...

Ok, now actually it's pretty cool that it tells you that both values look the same when printed but are actually distinct instances, but that's not really good enough.

The problem is that the Data object does not provide a custom ToString() override.  We could add one to that class, but maybe we can't or we don't want to.

Gallio provides a pluggable object formatter that MbUnit and other frameworks can use.  It provides built-in support for primitives, arrays, lists, and many other common types.

Now we also provide a built-in structural formatter for objects.  In the absence of a better formatting rule or a custom ToString() override, it uses reflection over the public properties and fields of an object to try to display its component parts.  So we don't need to define useless ToString() methods for test diagnostics anymore.

Here is the result... can you tell what the problem is?

AutoCAD Integration

Mike Sandberg has added support for testing AutoCAD plugins.

It turns out that AutoCAD has a managed extensibility model so you can create your own plugins using .Net and the ObjectARX toolkit.  Unfortunately it is somewhat difficult to write unit tests for plugins becuase they must run within the main UI thread of AutoCAD.

The AutoCAD integration for Gallio works by loading a shim into the AutoCAD application from which it can launch tests.  To enable this integration, specify the "AutoCAD" runner type to the Echo, Icarus, MSBuild, NAnt or PowerShell runners.

For example:

    Gallio.Echo.exe MyTestAssembly.dll /r:AutoCAD [other options...]

AutoCAD integration is not yet available from within the IDE.  We will be working to improve this use case in the future.

Running x86 tests as 32-bit on x64

We now support running x86 tests within a 32-bit process on x64.  This is necessary when the subject under test is linked to native 32-bit components since the 32-bit and 64-bit Application Binary Interfaces are incompatible.

To ensure that tests run in an x86 process, simply compile them for the x86 platform.

The only caveat is that this support might not work in embedded test runners such as TestDriven.Net, ReSharper or Visual Studio Test Tools because Gallio does not control process creation there.

However, it will work from the Icarus GUI, the build tasks and other tools as long as the tests are configured to run out of process (which is the default).

Visual Studio 2010 CTP Support

As mentioned previously, we are also shipping a preview of Gallio and MbUnit with support for .Net Framework 4.0 and Visual Studio 2010 CTP.

This is currently distributed as a separate download.  It has only been lightly tested so please provide feedback.

External Tools Compatibility

We've updated integration with a variety of external tools.

• CCNet v1.0.4
• CSUnit v2.0.5
• NUnit v2.4.8
• Pex v0.8
• ReSharper v3.1, v4.0, v4.1
• TeamCity v4.0 EAP
• xUnit.Net v1.0.3
• Visual Studio 2005, 2008, 2010 (separate installer req'd for 2010)

Other Improvements and Bug Fixes

• Improved the robustness of ReSharper, TestDriven.Net and Visual Studio Test Tools integration by loading Gallio in its own isolated AppDomain to prevent test assemblies from interfering with Gallio's runtime.
• Added a "resident" test runner for TestDriven.Net to improve start-up performance.  Use TestDriven.Net v2.17 or newer for this feature.
• Fixed some issues relates to the setting of the application base directory and working directory.  We were inadvertently attempting to create files within the Gallio installation folder at runtime which should not have been happening.
• Cleaned up the documentation of MbUnit assertions and attributes.  Repaired a few minor API inconsistencies along the way.
• Added support for the ReSharper shadow copy and application base directory options.
• Fixed a minor re-entrance issue when the Gallio MSBuild task was launched from within Visual Studio while other Gallio plug-ins for Visual Studio were already active.
• Icarus tweaks for performance and robustness.
• Added code to the isolated process hosting to interpret stack overflow related exit codes and log them.
• Fixed a couple of bugs related logging the result of an asynchronous ThreadTask or ProcessTask spawned by a test using the Tasks abstraction.
• Enhanced [ThreadedRepeat] and [Repeat] attributes to work on fixtures and to ensure that the setup/teardown runs.
• Improved the reporting of xUnit.Net theories.
• Fixed a bug that would sometimes prevent MbUnit v3 generic tests from running.
• Fixed TeamCity reporting to ensure that it includes data-driven tests.
• Fixed a bug in Assert.Throws that caused a StackOverflowException.
• Fixed a bug printing the inner exceptions of MbUnit v2 tests.

Technorati Tags: gallio,mbunit