Tuesday, November 18, 2008

Announcing Gallio and MbUnit v3.0.5

Today we are releasing Gallio and MbUnit v3.0.5.  This is primarily a bug fix release but we are introducing a few new features.

Assert.AreApproximatelyEqual, test factories, structural object formatting, AutoCAD integration, x86 tests on x64, Visual Studio CTP 2010.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Release notes: All release notes from all versions.

MbUnit

Assert.AreApproximatelyEqual

We have added Assert.AreApproximatelyEqual to express an assertion about equality within some delta (aka. tolerance).  Likewise there is Assert.AreNotApproximatelyEqual.

Examples:

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.3);
-> Success: 2.2 is in the range 2.0 +/- 0.3.

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Failure: 2.2 is not in the range 2.0 +/- 0.1.

Assert.AreNotApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Success: 2.2 is not in the range 2.0 +/- 0.1.

The implementation supports any comparable type that defines a subtraction operator, and for which the result of subtraction is also comparable.

Examples:

Assert.AreApproximatelyEqual(-5, 2, 10);
-> Success: 2 is in the range -5 +/- 10.

Assert.AreApproximatelyEqual(new DateTime(2008, 11, 17), new DateTime(2008, 11, 18), TimeSpan.FromDays(2));
-> Success: 2008/11/18 is in the range 2008/11/17 +/- 1 day.

Static and Dynamic Test Factories (aka. Test Suites)

Some people were wondering where the MbUnit v2 [TestSuite] mechanism went.  Now it's back with a few interesting twists.

A test factory in MbUnit v3 is a method that generates a test suite programmatically.  They are typically used when the structure of a test is derived from custom metadata.  For example, a test factory might produce a test suite generated from some information stored in the database.

A test suite produced by a test factory differs from a standard data-driven test (such as a Row-test) in that the tester has more control over the generated structure of the test suite.  Instead of plugging in values to a parameterized test, whole new tests can be created arbitrarily.

There are two kinds of test factories.

  • A static test factory adds new tests to the static test tree.  This makes the generated test suite part of the static test model that UI-based test runners such as Icarus use to present tests to the user.

    Pros: The structure of the test suite is determined before tests actually run.

    Cons: The structure of the test suite is fixed at test exploration time.  Moreover, any data that was needed to produce the tests will be loaded all at once up front regardless of whether the test suite will actually be executed.  Also, for this mechanism to work at all, the code that defines the factory must be executable at test exploration time.  This is not the case in ReSharper or Visual Studio Test Tools so static test suites will not appear there, alas.  (At least not as things are written today...)
  • A dynamic test factory adds new tests to the dynamic test tree, aka. the test step tree.  The tests are not included in the static test model which makes it impossible for the user to pick a subset of the dynamic tests to run (basically they are all-or-nothing).

    Pros: The structure of the test suite is determined when the fixture that contains it runs so it can vary based on input data determined at runtime.   The test factory can be parameterized so it can produce multiple test suites with different input data.  Likewise, a dynamic test factory can emit an unbounded number of tests.  It can keep on producing tests to run for as long as it likes.

    Cons: The dynamic tests are not visible to the user until test execution time.

It might help to think of a static test factory as working something like a [TestFixture] or [Test] where the test suite is constructed ahead of time using an algorithm of your choice.  On the other hand, a dynamic test factory is more like TestStep.Run which is used to create dynamic test steps within a running test or like parameterized tests which pull contents from a data source at runtime.

Clear as mud huh?  How about some examples.

Static Test Factory Example

In this example, we create a few silly tests statically in the CreateTests static method.  Notice that it returns an enumeration of Test objects.  It is also static because no instance of the fixture exists at the time this method is called.  It follows that we cannot define a static test factory as a member of a generic test fixture because we won't know what values are intended to be plugged into the fixture's type parameters.

There are currently 3 kinds of built-in Test subclasses (you can make your own too):

  • A TestCase is a kind of Test that contains executable code to run.  It can have a name, description and other custom metadata.
  • A TestSuite is another kind of Test that composes a list of other Tests (including other suites) to run.  It can also have a name, description, and other custom metadata.  In addition to its children, it can also include SetUp, TearDown, SuiteSetUp, and SuiteTearDown functions.
  • A TestFixtureReference is yet another kind of Test.  We do not show it here but basically it allows you to create test suites that reference .

image 

When we run the test fixture, it will include a test called "Sample Test" as well as a suite called "Sample Suite" with two silly test cases within.

Dynamic Test Factory Example

A dynamic test factory is quite similar to a static test factory.  It just runs at runtime instead.

The first thing you will notice is that this dynamic test factory is not declared static.  It can be if you like, but that's optional.  The second thing is that this test factory can actually be parameterized!

Here we create 3 groups of dynamically generated test cases using different input data each time.

image

When we run the test fixture, it will produce 1 test case when n = 1, 4 test cases when n = 2 and 9 test cases when n = 3.

Of course this example is silly.  I suppose next time I could write up something like a miniature file-based DSL integrated using test factories.

Structural Object Formatting

How often has this happened to you?

So you write this test...

image

And then it fails...

image

Ok, now actually it's pretty cool that it tells you that both values look the same when printed but are actually distinct instances, but that's not really good enough.

The problem is that the Data object does not provide a custom ToString() override.  We could add one to that class, but maybe we can't or we don't want to.

Gallio provides a pluggable object formatter that MbUnit and other frameworks can use.  It provides built-in support for primitives, arrays, lists, and many other common types.

Now we also provide a built-in structural formatter for objects.  In the absence of a better formatting rule or a custom ToString() override, it uses reflection over the public properties and fields of an object to try to display its component parts.  So we don't need to define useless ToString() methods for test diagnostics anymore.

Here is the result... can you tell what the problem is?

image

AutoCAD Integration

Mike Sandberg has added support for testing AutoCAD plugins.

It turns out that AutoCAD has a managed extensibility model so you can create your own plugins using .Net and the ObjectARX toolkit.  Unfortunately it is somewhat difficult to write unit tests for plugins becuase they must run within the main UI thread of AutoCAD.

The AutoCAD integration for Gallio works by loading a shim into the AutoCAD application from which it can launch tests.  To enable this integration, specify the "AutoCAD" runner type to the Echo, Icarus, MSBuild, NAnt or PowerShell runners.

For example:

    Gallio.Echo.exe MyTestAssembly.dll /r:AutoCAD [other options...]

AutoCAD integration is not yet available from within the IDE.  We will be working to improve this use case in the future.

Running x86 tests as 32-bit on x64

We now support running x86 tests within a 32-bit process on x64.  This is necessary when the subject under test is linked to native 32-bit components since the 32-bit and 64-bit Application Binary Interfaces are incompatible.

To ensure that tests run in an x86 process, simply compile them for the x86 platform.

image

The only caveat is that this support might not work in embedded test runners such as TestDriven.Net, ReSharper or Visual Studio Test Tools because Gallio does not control process creation there.

However, it will work from the Icarus GUI, the build tasks and other tools as long as the tests are configured to run out of process (which is the default).

Visual Studio 2010 CTP Support

As mentioned previously, we are also shipping a preview of Gallio and MbUnit with support for .Net Framework 4.0 and Visual Studio 2010 CTP.

This is currently distributed as a separate download.  It has only been lightly tested so please provide feedback.

External Tools Compatibility

We've updated integration with a variety of external tools.

  • CCNet v1.0.4
  • CSUnit v2.0.5
  • NUnit v2.4.8
  • Pex v0.8
  • ReSharper v3.1, v4.0, v4.1
  • TeamCity v4.0 EAP
  • xUnit.Net v1.0.3
  • Visual Studio 2005, 2008, 2010 (separate installer req'd for 2010)

Other Improvements and Bug Fixes

  • Improved the robustness of ReSharper, TestDriven.Net and Visual Studio Test Tools integration by loading Gallio in its own isolated AppDomain to prevent test assemblies from interfering with Gallio's runtime.
  • Added a "resident" test runner for TestDriven.Net to improve start-up performance.  Use TestDriven.Net v2.17 or newer for this feature.
  • Fixed some issues relates to the setting of the application base directory and working directory.  We were inadvertently attempting to create files within the Gallio installation folder at runtime which should not have been happening.
  • Cleaned up the documentation of MbUnit assertions and attributes.  Repaired a few minor API inconsistencies along the way.
  • Added support for the ReSharper shadow copy and application base directory options.
  • Fixed a minor re-entrance issue when the Gallio MSBuild task was launched from within Visual Studio while other Gallio plug-ins for Visual Studio were already active.
  • Icarus tweaks for performance and robustness.
  • Added code to the isolated process hosting to interpret stack overflow related exit codes and log them.
  • Fixed a couple of bugs related logging the result of an asynchronous ThreadTask or ProcessTask spawned by a test using the Tasks abstraction.
  • Enhanced [ThreadedRepeat] and [Repeat] attributes to work on fixtures and to ensure that the setup/teardown runs.
  • Improved the reporting of xUnit.Net theories.
  • Fixed a bug that would sometimes prevent MbUnit v3 generic tests from running.
  • Fixed TeamCity reporting to ensure that it includes data-driven tests.
  • Fixed a bug in Assert.Throws that caused a StackOverflowException.
  • Fixed a bug printing the inner exceptions of MbUnit v2 tests.
Technorati Tags: ,

Don't change semantics of well-known types!

I was writing a little quick and dirty PowerShell script to compare items in Visual Studio 2008 and Visual Studio 2010 project files side-by-side to ensure they are kept in sync.

Now I will admit that I am a total PowerShell newbie but I was not prepared to deal with this oddity:

$xml = New-Object System.Xml.XmlDocument
$xml.Load($path)
$nsmgr = New-Object System.Xml.XmlNamespaceManager $xml.psbase.NameTable

(...crash... cannot resolve the constructor for XmlNamespaceManager...)

See that little psbase?  Well, because without it, PowerShell will apply some kind of built-in adapter semantics.  Instead of giving back the NameTable, it will actually try to look for an element or attribute called "NameTable" and return that.  Of course, since it doesn't find one, it returns Null instead!

Now I'm sure PowerShell provides some cool Xml slinging features I don't know about right now.  However it's also a .Net language bound to the .Net framework.  It just diverged pretty dramatically from my mental model of some pretty fundamental .Net framework types.  It takes time to figure that out and recover...

This isn't like adding a fairly innocuous Ruby-ish to_s() method to types as an alias for ToString().  This is instead rather intrusive behavior that changes what member selection means for all XmlDocument objects.  I think it would be better if PowerShell introduced its own Xml type instead.  Then it could play with the semantics all it wanted while still preserving the expected behavior of the member selection operator.

Are there any other landmines I should beware of?

Monday, November 17, 2008

Speaking at QCon on Thursday

This week I'll be at QCon speaking about Gallio and MbUnit in the ALT.Net track.

I've been looking forward to this for months!  ;-)

Blogging_SF

Thursday, November 6, 2008

C# Signatures

On a thread in the Visual Studio 2010 C# and VB Feedback forum, DMeglio posted a question about whether C# 4.0 could be extended to support extended generic constraints.

I'm just curious, is the feature set for C# 4.0 locked? There is one thing that I feel is a glaring omission, and really hurts C#. The fact that generics do not support operator constraints. This is a huge issue for many of us. Why can't we do:

public class List<K> where operator+

I know this would make my life easier, I guess I'm just wondering if it's definitely been tabled or if it's still a possibility for 4.0

So I thought about this for a bit and proposed something similar to the C++ signature abstraction or a Haskell Type Class.

Nominative vs. Structural Typing

Consider the following types.

public class Cat
{
    void Speak() { Console.WriteLine("Meow!"); }
}

public class Dog
{
    void Speak() { Console.WriteLine("Woof!"); }
}

Well, they both have a Speak() method but they do not implement any common interfaces or inherit from a common type (besides Object).  If we want to call Speak() polymorphically on Cat and Dog then we could modify them to implement an ISpeakable interface or Animal abstract base class.

That's because the CLR uses Nominative Typing.  Two types are related only in the ways they explicitly declare as part of their definition.  We take this for granted every day, but it's not the only way to do things.  There's also Structural Typing.

The Cat and Dog classes obviously share structural similarities even if they don't declare them.  They both have Speak methods...

C# Signature Proposal

Consider the following pseudo-code.

public signature Numeric<T>
{
    static T operator+(T left, T right);
    static T Zero { get; }
}

Any type can be said to satisfy this signature if it has the specified members.  It need not actually declare that is satisfies the signature.

Assuming this notation, we can define methods like the following:

public static class Math
{
    public static T Sum(IEnumerable<T> values)
        where T : Numeric<T> // signifying conformance to a signature, NOT inheritance
    {
        T total = T.Zero; // the type must have a Zero property, per the signature
        foreach (T value in values)
            total += value; // valid because the type has an operator+, per the signature
       
return total;
    }
}

Intuitively, the Sum method can be used on any type that supports all of the Numeric<T> operations.

Ok, but Int32 does not actually have operator+, or Zero defined on it.  So those methods must come from somewhere else!  Some of them could be defined as a default implementation provided by the signature but that's not enough.

C# Extension Properties and Static Methods

In C# 3.0, we got some fancy compiler syntax for extension methods.  They are defined like this:

public static class EnumerableExtensions
{
    public static int Count<T>(this IEnumerable<T> values)
    {
        int count = 0;
        foreach (T value in values)
            count += 1;
        return count;
    }
}

This is all well and good but there is no syntax for extension properties and extension events, or for static extension methods, et. al.

It turns out that the current syntax for extension methods is very awkward to extend.  Where will you insert this in those extension property and event declarations?

C# Signature Extension Proposal

Ah!  But what if we allow signatures to extend one another or to extend existing types?

Then we can define new behaviors on those signatures that will apply to any types that satisfy the signature.  If the signature extends another signature then it incorporates all parts of that other signature.  Likewise if it extends a type then it takes on all aspects of that type.

But the compiler could also suppose that if a type conforms to a signature that is in scope, then all members of that signature can be used implicitly as if they were members of that type.  Just like extension methods.

public signature Int32Extensions : Int32, Numeric<Int32>
{
    public static Int32 Zero
    {
        get { return 0; }
    }

    public static Int32 operator+ (Int32 a, Int32 b)
    {
        return a + b;
    }
}

Voila!  Static extension methods and properties.

How about non-static extension methods and properties?

public signature Int32Extensions : Int32
{
    // I'm an extension property of Int32!
    public bool IsZero
    {
        return this == 0;
    }
}

Simplifications

If all we want is support for things like extension properties then it's clear that we can simplify the proposal greatly.  Instead of defining something new like a signature we could define something like an extension class.  However I feel that the structural typing ideas are quite useful on their own.

Wednesday, November 5, 2008

Gallio Store on Zazzle!

Thanks to Mark Haley, our fantastic graphics designer, we are now offering a selection of Gallio branded products on Zazzle.  Expect MbUnit branded products to arrive soon.

Proceeds from the store will be used to offset our hosting costs.

Full Text of the Announcement

You can now show your support for Gallio with merchandise bearing the Gallio Logo from a store we have set up on Zazzle.

At present there is a selection of T-Shirts, Polos, Hooded Sweatshirts, Hats, Mugs and Bumper Stickers available.  This first run has one particular style, but other styles may follow.  We intend to add MbUnit branded items in the future as well, so stay tuned!

A new menu item has been added to the gallio.org website that will direct you to the Zazzle site, or the direct link is here:
http://www.zazzle.com/gallio

Please note:  most items can be customised extensively on the product page.  For example, shirt colors or even style may be adjusted from the default style shown, so don't be afraid to experiment to get it just how you like it.

Technorati Tags: ,

Gallio and MbUnit v3 for Visual Studio 2010 CTP

I suppose by now you have all downloaded the Visual Studio 2010 CTP(Well, maybe not.)

But if you have and you are looking for a great unit testing framework for your first experiments with .Net Framework 4.0, I have published an early preview of Gallio and MbUnit v3 designed to work with the CTP.

Out of the box, it supports the VSTS test runner so you can run your tests using the Visual Studio Test View.  Just be sure to create a new test project using the MbUnit v3 Test Project template.

Try it out!

Download: Gallio and MbUnit v3 for Visual Studio 2010 CTP

Screenshot

image

Technorati Tags: ,