Tuesday, November 18, 2008

Announcing Gallio and MbUnit v3.0.5

Today we are releasing Gallio and MbUnit v3.0.5.  This is primarily a bug fix release but we are introducing a few new features.

Assert.AreApproximatelyEqual, test factories, structural object formatting, AutoCAD integration, x86 tests on x64, Visual Studio CTP 2010.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Release notes: All release notes from all versions.

MbUnit

Assert.AreApproximatelyEqual

We have added Assert.AreApproximatelyEqual to express an assertion about equality within some delta (aka. tolerance).  Likewise there is Assert.AreNotApproximatelyEqual.

Examples:

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.3);
-> Success: 2.2 is in the range 2.0 +/- 0.3.

Assert.AreApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Failure: 2.2 is not in the range 2.0 +/- 0.1.

Assert.AreNotApproximatelyEqual(/*expectedValue*/ 2.0, /*actualValue*/ 2.2, /*delta*/ 0.1);
-> Success: 2.2 is not in the range 2.0 +/- 0.1.

The implementation supports any comparable type that defines a subtraction operator, and for which the result of subtraction is also comparable.

Examples:

Assert.AreApproximatelyEqual(-5, 2, 10);
-> Success: 2 is in the range -5 +/- 10.

Assert.AreApproximatelyEqual(new DateTime(2008, 11, 17), new DateTime(2008, 11, 18), TimeSpan.FromDays(2));
-> Success: 2008/11/18 is in the range 2008/11/17 +/- 1 day.

Static and Dynamic Test Factories (aka. Test Suites)

Some people were wondering where the MbUnit v2 [TestSuite] mechanism went.  Now it's back with a few interesting twists.

A test factory in MbUnit v3 is a method that generates a test suite programmatically.  They are typically used when the structure of a test is derived from custom metadata.  For example, a test factory might produce a test suite generated from some information stored in the database.

A test suite produced by a test factory differs from a standard data-driven test (such as a Row-test) in that the tester has more control over the generated structure of the test suite.  Instead of plugging in values to a parameterized test, whole new tests can be created arbitrarily.

There are two kinds of test factories.

  • A static test factory adds new tests to the static test tree.  This makes the generated test suite part of the static test model that UI-based test runners such as Icarus use to present tests to the user.

    Pros: The structure of the test suite is determined before tests actually run.

    Cons: The structure of the test suite is fixed at test exploration time.  Moreover, any data that was needed to produce the tests will be loaded all at once up front regardless of whether the test suite will actually be executed.  Also, for this mechanism to work at all, the code that defines the factory must be executable at test exploration time.  This is not the case in ReSharper or Visual Studio Test Tools so static test suites will not appear there, alas.  (At least not as things are written today...)
  • A dynamic test factory adds new tests to the dynamic test tree, aka. the test step tree.  The tests are not included in the static test model which makes it impossible for the user to pick a subset of the dynamic tests to run (basically they are all-or-nothing).

    Pros: The structure of the test suite is determined when the fixture that contains it runs so it can vary based on input data determined at runtime.   The test factory can be parameterized so it can produce multiple test suites with different input data.  Likewise, a dynamic test factory can emit an unbounded number of tests.  It can keep on producing tests to run for as long as it likes.

    Cons: The dynamic tests are not visible to the user until test execution time.

It might help to think of a static test factory as working something like a [TestFixture] or [Test] where the test suite is constructed ahead of time using an algorithm of your choice.  On the other hand, a dynamic test factory is more like TestStep.Run which is used to create dynamic test steps within a running test or like parameterized tests which pull contents from a data source at runtime.

Clear as mud huh?  How about some examples.

Static Test Factory Example

In this example, we create a few silly tests statically in the CreateTests static method.  Notice that it returns an enumeration of Test objects.  It is also static because no instance of the fixture exists at the time this method is called.  It follows that we cannot define a static test factory as a member of a generic test fixture because we won't know what values are intended to be plugged into the fixture's type parameters.

There are currently 3 kinds of built-in Test subclasses (you can make your own too):

  • A TestCase is a kind of Test that contains executable code to run.  It can have a name, description and other custom metadata.
  • A TestSuite is another kind of Test that composes a list of other Tests (including other suites) to run.  It can also have a name, description, and other custom metadata.  In addition to its children, it can also include SetUp, TearDown, SuiteSetUp, and SuiteTearDown functions.
  • A TestFixtureReference is yet another kind of Test.  We do not show it here but basically it allows you to create test suites that reference .

image 

When we run the test fixture, it will include a test called "Sample Test" as well as a suite called "Sample Suite" with two silly test cases within.

Dynamic Test Factory Example

A dynamic test factory is quite similar to a static test factory.  It just runs at runtime instead.

The first thing you will notice is that this dynamic test factory is not declared static.  It can be if you like, but that's optional.  The second thing is that this test factory can actually be parameterized!

Here we create 3 groups of dynamically generated test cases using different input data each time.

image

When we run the test fixture, it will produce 1 test case when n = 1, 4 test cases when n = 2 and 9 test cases when n = 3.

Of course this example is silly.  I suppose next time I could write up something like a miniature file-based DSL integrated using test factories.

Structural Object Formatting

How often has this happened to you?

So you write this test...

image

And then it fails...

image

Ok, now actually it's pretty cool that it tells you that both values look the same when printed but are actually distinct instances, but that's not really good enough.

The problem is that the Data object does not provide a custom ToString() override.  We could add one to that class, but maybe we can't or we don't want to.

Gallio provides a pluggable object formatter that MbUnit and other frameworks can use.  It provides built-in support for primitives, arrays, lists, and many other common types.

Now we also provide a built-in structural formatter for objects.  In the absence of a better formatting rule or a custom ToString() override, it uses reflection over the public properties and fields of an object to try to display its component parts.  So we don't need to define useless ToString() methods for test diagnostics anymore.

Here is the result... can you tell what the problem is?

image

AutoCAD Integration

Mike Sandberg has added support for testing AutoCAD plugins.

It turns out that AutoCAD has a managed extensibility model so you can create your own plugins using .Net and the ObjectARX toolkit.  Unfortunately it is somewhat difficult to write unit tests for plugins becuase they must run within the main UI thread of AutoCAD.

The AutoCAD integration for Gallio works by loading a shim into the AutoCAD application from which it can launch tests.  To enable this integration, specify the "AutoCAD" runner type to the Echo, Icarus, MSBuild, NAnt or PowerShell runners.

For example:

    Gallio.Echo.exe MyTestAssembly.dll /r:AutoCAD [other options...]

AutoCAD integration is not yet available from within the IDE.  We will be working to improve this use case in the future.

Running x86 tests as 32-bit on x64

We now support running x86 tests within a 32-bit process on x64.  This is necessary when the subject under test is linked to native 32-bit components since the 32-bit and 64-bit Application Binary Interfaces are incompatible.

To ensure that tests run in an x86 process, simply compile them for the x86 platform.

image

The only caveat is that this support might not work in embedded test runners such as TestDriven.Net, ReSharper or Visual Studio Test Tools because Gallio does not control process creation there.

However, it will work from the Icarus GUI, the build tasks and other tools as long as the tests are configured to run out of process (which is the default).

Visual Studio 2010 CTP Support

As mentioned previously, we are also shipping a preview of Gallio and MbUnit with support for .Net Framework 4.0 and Visual Studio 2010 CTP.

This is currently distributed as a separate download.  It has only been lightly tested so please provide feedback.

External Tools Compatibility

We've updated integration with a variety of external tools.

  • CCNet v1.0.4
  • CSUnit v2.0.5
  • NUnit v2.4.8
  • Pex v0.8
  • ReSharper v3.1, v4.0, v4.1
  • TeamCity v4.0 EAP
  • xUnit.Net v1.0.3
  • Visual Studio 2005, 2008, 2010 (separate installer req'd for 2010)

Other Improvements and Bug Fixes

  • Improved the robustness of ReSharper, TestDriven.Net and Visual Studio Test Tools integration by loading Gallio in its own isolated AppDomain to prevent test assemblies from interfering with Gallio's runtime.
  • Added a "resident" test runner for TestDriven.Net to improve start-up performance.  Use TestDriven.Net v2.17 or newer for this feature.
  • Fixed some issues relates to the setting of the application base directory and working directory.  We were inadvertently attempting to create files within the Gallio installation folder at runtime which should not have been happening.
  • Cleaned up the documentation of MbUnit assertions and attributes.  Repaired a few minor API inconsistencies along the way.
  • Added support for the ReSharper shadow copy and application base directory options.
  • Fixed a minor re-entrance issue when the Gallio MSBuild task was launched from within Visual Studio while other Gallio plug-ins for Visual Studio were already active.
  • Icarus tweaks for performance and robustness.
  • Added code to the isolated process hosting to interpret stack overflow related exit codes and log them.
  • Fixed a couple of bugs related logging the result of an asynchronous ThreadTask or ProcessTask spawned by a test using the Tasks abstraction.
  • Enhanced [ThreadedRepeat] and [Repeat] attributes to work on fixtures and to ensure that the setup/teardown runs.
  • Improved the reporting of xUnit.Net theories.
  • Fixed a bug that would sometimes prevent MbUnit v3 generic tests from running.
  • Fixed TeamCity reporting to ensure that it includes data-driven tests.
  • Fixed a bug in Assert.Throws that caused a StackOverflowException.
  • Fixed a bug printing the inner exceptions of MbUnit v2 tests.
Technorati Tags: ,

Don't change semantics of well-known types!

I was writing a little quick and dirty PowerShell script to compare items in Visual Studio 2008 and Visual Studio 2010 project files side-by-side to ensure they are kept in sync.

Now I will admit that I am a total PowerShell newbie but I was not prepared to deal with this oddity:

$xml = New-Object System.Xml.XmlDocument
$xml.Load($path)
$nsmgr = New-Object System.Xml.XmlNamespaceManager $xml.psbase.NameTable

(...crash... cannot resolve the constructor for XmlNamespaceManager...)

See that little psbase?  Well, because without it, PowerShell will apply some kind of built-in adapter semantics.  Instead of giving back the NameTable, it will actually try to look for an element or attribute called "NameTable" and return that.  Of course, since it doesn't find one, it returns Null instead!

Now I'm sure PowerShell provides some cool Xml slinging features I don't know about right now.  However it's also a .Net language bound to the .Net framework.  It just diverged pretty dramatically from my mental model of some pretty fundamental .Net framework types.  It takes time to figure that out and recover...

This isn't like adding a fairly innocuous Ruby-ish to_s() method to types as an alias for ToString().  This is instead rather intrusive behavior that changes what member selection means for all XmlDocument objects.  I think it would be better if PowerShell introduced its own Xml type instead.  Then it could play with the semantics all it wanted while still preserving the expected behavior of the member selection operator.

Are there any other landmines I should beware of?

Monday, November 17, 2008

Speaking at QCon on Thursday

This week I'll be at QCon speaking about Gallio and MbUnit in the ALT.Net track.

I've been looking forward to this for months!  ;-)

Blogging_SF

Thursday, November 6, 2008

C# Signatures

On a thread in the Visual Studio 2010 C# and VB Feedback forum, DMeglio posted a question about whether C# 4.0 could be extended to support extended generic constraints.

I'm just curious, is the feature set for C# 4.0 locked? There is one thing that I feel is a glaring omission, and really hurts C#. The fact that generics do not support operator constraints. This is a huge issue for many of us. Why can't we do:

public class List<K> where operator+

I know this would make my life easier, I guess I'm just wondering if it's definitely been tabled or if it's still a possibility for 4.0

So I thought about this for a bit and proposed something similar to the C++ signature abstraction or a Haskell Type Class.

Nominative vs. Structural Typing

Consider the following types.

public class Cat
{
    void Speak() { Console.WriteLine("Meow!"); }
}

public class Dog
{
    void Speak() { Console.WriteLine("Woof!"); }
}

Well, they both have a Speak() method but they do not implement any common interfaces or inherit from a common type (besides Object).  If we want to call Speak() polymorphically on Cat and Dog then we could modify them to implement an ISpeakable interface or Animal abstract base class.

That's because the CLR uses Nominative Typing.  Two types are related only in the ways they explicitly declare as part of their definition.  We take this for granted every day, but it's not the only way to do things.  There's also Structural Typing.

The Cat and Dog classes obviously share structural similarities even if they don't declare them.  They both have Speak methods...

C# Signature Proposal

Consider the following pseudo-code.

public signature Numeric<T>
{
    static T operator+(T left, T right);
    static T Zero { get; }
}

Any type can be said to satisfy this signature if it has the specified members.  It need not actually declare that is satisfies the signature.

Assuming this notation, we can define methods like the following:

public static class Math
{
    public static T Sum(IEnumerable<T> values)
        where T : Numeric<T> // signifying conformance to a signature, NOT inheritance
    {
        T total = T.Zero; // the type must have a Zero property, per the signature
        foreach (T value in values)
            total += value; // valid because the type has an operator+, per the signature
       
return total;
    }
}

Intuitively, the Sum method can be used on any type that supports all of the Numeric<T> operations.

Ok, but Int32 does not actually have operator+, or Zero defined on it.  So those methods must come from somewhere else!  Some of them could be defined as a default implementation provided by the signature but that's not enough.

C# Extension Properties and Static Methods

In C# 3.0, we got some fancy compiler syntax for extension methods.  They are defined like this:

public static class EnumerableExtensions
{
    public static int Count<T>(this IEnumerable<T> values)
    {
        int count = 0;
        foreach (T value in values)
            count += 1;
        return count;
    }
}

This is all well and good but there is no syntax for extension properties and extension events, or for static extension methods, et. al.

It turns out that the current syntax for extension methods is very awkward to extend.  Where will you insert this in those extension property and event declarations?

C# Signature Extension Proposal

Ah!  But what if we allow signatures to extend one another or to extend existing types?

Then we can define new behaviors on those signatures that will apply to any types that satisfy the signature.  If the signature extends another signature then it incorporates all parts of that other signature.  Likewise if it extends a type then it takes on all aspects of that type.

But the compiler could also suppose that if a type conforms to a signature that is in scope, then all members of that signature can be used implicitly as if they were members of that type.  Just like extension methods.

public signature Int32Extensions : Int32, Numeric<Int32>
{
    public static Int32 Zero
    {
        get { return 0; }
    }

    public static Int32 operator+ (Int32 a, Int32 b)
    {
        return a + b;
    }
}

Voila!  Static extension methods and properties.

How about non-static extension methods and properties?

public signature Int32Extensions : Int32
{
    // I'm an extension property of Int32!
    public bool IsZero
    {
        return this == 0;
    }
}

Simplifications

If all we want is support for things like extension properties then it's clear that we can simplify the proposal greatly.  Instead of defining something new like a signature we could define something like an extension class.  However I feel that the structural typing ideas are quite useful on their own.

Wednesday, November 5, 2008

Gallio Store on Zazzle!

Thanks to Mark Haley, our fantastic graphics designer, we are now offering a selection of Gallio branded products on Zazzle.  Expect MbUnit branded products to arrive soon.

Proceeds from the store will be used to offset our hosting costs.

Full Text of the Announcement

You can now show your support for Gallio with merchandise bearing the Gallio Logo from a store we have set up on Zazzle.

At present there is a selection of T-Shirts, Polos, Hooded Sweatshirts, Hats, Mugs and Bumper Stickers available.  This first run has one particular style, but other styles may follow.  We intend to add MbUnit branded items in the future as well, so stay tuned!

A new menu item has been added to the gallio.org website that will direct you to the Zazzle site, or the direct link is here:
http://www.zazzle.com/gallio

Please note:  most items can be customised extensively on the product page.  For example, shirt colors or even style may be adjusted from the default style shown, so don't be afraid to experiment to get it just how you like it.

Technorati Tags: ,

Gallio and MbUnit v3 for Visual Studio 2010 CTP

I suppose by now you have all downloaded the Visual Studio 2010 CTP(Well, maybe not.)

But if you have and you are looking for a great unit testing framework for your first experiments with .Net Framework 4.0, I have published an early preview of Gallio and MbUnit v3 designed to work with the CTP.

Out of the box, it supports the VSTS test runner so you can run your tests using the Visual Studio Test View.  Just be sure to create a new test project using the MbUnit v3 Test Project template.

Try it out!

Download: Gallio and MbUnit v3 for Visual Studio 2010 CTP

Screenshot

image

Technorati Tags: ,

Saturday, October 25, 2008

See you at Microsoft PDC

I will be at the Microsoft PDC.

If you're there then find me walking around wearing a kilt, as usual.  I will also be a member of a lunchtime panel on the Future of Unit Testing, probably on Wednesday.

See you there!

Monday, October 20, 2008

Leaky Exceptions

Take a look at this exception:

System.Runtime.Serialization.SerializationException: Unable to find assembly 'Castle.MicroKernel, Version=1.0.3.0, Culture=neutral, PublicKeyToken=407dd0808d44fbdc'.

<snip>

at System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage.FixupForNewAppDomain()
at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(IMessage reqMsg)

Exception rethrown at [0]: 
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Gallio.TDNetRunner.Core.IProxyTestRunner.Run(IFacadeTestListener testListener, String assemblyPath, String cref)

What Happened

  1. We made a call into a remote AppDomain.
  2. The call was processed and now we are trying to produce a return message.
  3. The call threw an exception of some type.
  4. The exception could not be deserialized by the caller because the exception is defined in the Castle.MicroKernel assembly.
  5. As a result, the remoting library "helpfully" replaced the real exception with a SerializationException instead.

Diagnosis

The server is leaking an exception of a type that the client does not understand.

The remote call interface is a Leaky Abstraction.  Technically, the implementation of that interface is leaky.

Fixes

Case 1: Wrap the inner exception.

There are several things we can do to fix leaky exceptions.  In most cases, it's simple enough to wrap the exception.

public void Wibble(string wibbleUser, string wibbleParam)
{
    try
    {
        wibbleAuditor.RecordWibble(wibbleUser, wibbleParam);
    }
    catch (AuditException ex)
    {
        throw new WibbleException("Could not write audit log message before wibbling.  Wibble aborted.");
    }
    try
    {
        wibbleHandler.Execute(wibbleParam);
    }
    catch (WibbleProcessorException ex)
    {
        throw new WibbleException("An error occurred while wibbling.", ex);
    }
}

Case 2: Capture a description of the inner exception as a string.

As we saw in the case of our SerializationException above, in some cases we cannot actually wrap the exception since the "InnerException" itself might not be serializable or deserializable.

In just those cases, we can do something like the following instead:

throw new WibbleException(String.Format("An error occurred while wibbling.  {0}", ex));

Capturing the exception as a string lets us smuggle it over into places where the exception type might not be accessible.  However, we do lose some flexibility in how the client can handle the exception.

Case 3: Discard the inner exception.

There is such a thing as too much information.

Sometimes revealing the contents of the inner exception to the client at all may constitute a security risk (leaking sensitive data in the stack trace or the message) or it may just provide way more details than the client can understand.

In these cases, it's best to provide a clear description of the exception in-situ without relying on the information from the internal exception at all.  Such an exception might instead refer to another source for obtaining the nitty-gritty details for low-level diagnosis.

throw new WibbleException("An error occurred while wibbling.  Consult the wibble processor log for details.");

Similar Problems

A method that throws an exception of an internal type.  This effectively prevents the client from filtering by exception type.

A method that throws an exception of an undocumented type.  While the exception may be caught, the client does not know to expect it.

A method that throws an exception of a vague type or with a vague message.  While the exception may be filtered and may be expected, the client will not be able to make sense of its contents.

An improperly defined custom exception type that lacks the special deserialization constructor and [Serializable].  Such an exception cannot cross remoting boundaries even if it was expected to.

An exception that reveals sensitive information as part of the exception message.

Examples:

  • Allowing a generic platform exception such as Win32Exception or COMException to escape through a high-level supposedly platform-independent interface.
  • Allowing an exception thrown by a some lower-level component or implementation detail that the client does not know about to escape.
  • Allowing an exception of a type not specified in the published documentation to escape.
  • Allowing a FileNotFoundException from an internal (non-obvious) I/O operation to be returned to the client without wrapping it with a higher-level exception type to explain what was going on.
  • Throwing an exception that contains a database connection string or user password in the message.

Full Circle

In this case I modified my code to return a SafeException containing the inner exception message as a string (applied case #2 above).  Here's what was hiding behind that SerializationException.

Gallio.Loader.SafeException: Gallio.Runtime.RuntimeException: Could not resolve service of type Gallio.Runner.ITestRunnerFactory. ---> Castle.MicroKernel.Resolvers.DependencyResolverException: Could not resolve non-optional dependency for 'NCoverIntegration2.NCoverTestRunnerFactory' (Gallio.Runner.DefaultTestRunnerFactory). Parameter 'name' type 'System.String'
   at Castle.MicroKernel.Resolvers.DefaultDependencyResolver.Resolve(CreationContext context, ISubDependencyResolver parentResolver, ComponentModel model, DependencyModel dependency)

<snip>

Ahh!  Now I know what's going on so I can fix it...

Saturday, October 18, 2008

Building Stuff

At yellowpages.com, I lead a Tools & Automation team.  In case you've ever wondered where MbUnit and Gallio fit into my work life, one of my responsibilities is managing a team of Test Automation Engineers.  We also use these tools for unit testing of our core systems.

Another one of my responsibilities is Release Engineering which is sort of an umbrella term governing processes related to Build, Deploy, Continuous Integration, System Monitoring, Defect Tracking, Source Control and the like.  In short, stuff that enables us to ship high-quality code when we want to.

I have built lots of great stuff as part of my job here:

  • For Unit Testing and Integration Testing, I built numerous extensions for MbUnit then eventually took up the lead on MbUnit v3 and Gallio.
  • For System Monitoring, I built a tool called Odin that has yet to be open-sourced itself but which did produce 3 OSS libraries, Castle.FlexBridge, Castle.Components.Cache and Castle.Components.Scheduler.
  • For FxCop integration in Visual Studio 2005, I built FxCopAddIn (now defunct).
  • For Build, I built MetaBuild, a lightweight modular build system.
  • For Deploy, I'm going to be building or borrowing something to replace what we've got...  I don't know what yet but I know I will want more help.
  • I've contributed little bits and pieces here and there to several other great OSS projects like Castle, Rhino.Mocks, WatiN, CruiseControl.Net and more.

I have also built a strong team that I am very proud of.  We build great things together!

Here are some other things my team has built:

  • A test scheduling and reporting system.
  • A diagnostic "inspector" tool for troubleshooting applications.
  • An automated test spider tool to search for broken links.
  • A comprehensive model and system test suite.
  • A phone-testing harness.

And there's tons more stuff to build!  Which reminds me...

Yet another one of my responsibilities is to recruit great talent.

Want to build cool stuff?  Call me!

Friday, October 17, 2008

Gallio and MbUnit v3.0.4 build 485 Released

We have released a minor update to Gallio and MbUnit v3.0.4 to resolve a problem that occurs when a user creates a partial copy of Gallio which is missing certain required plug-in assemblies.  Gallio will now silently disable those plug-ins.

If you plan to copy Gallio and MbUnit into your source tree then you may benefit from this update.  Without it you may observe a "ConfigurationErrorsException" due to the Visual Studio plug-in's dependency on the GAC.  (Of course, you could just delete the Visual Studio plug-in from your source tree since it will not do anything useful in there anyway.)

Download Link: Gallio Downloads

Technorati Tags: ,

Thursday, October 16, 2008

Announcing Gallio and MbUnit v3.0.4

Today we are releasing v3.0.4 of Gallio and MbUnit.  This release incorporates many new features as well improvements for robustness.

New asserts, assertion diffs, 0rdered tests, Rollback, Repeat, ThreadedRepeat, R# 4.1, VSTS, CSUnit, TeamCity, Gallio.Ambience, clickable report links (IE/Firefox integration), and more...

Try it out!  Please be aware that there are now separate x86 and x64 installers.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

MbUnit

Assertions Refresh

We have consolidated and significantly improved the set of built-in MbUnit assertions.  Refer to the migration guide at the end of this post for more information.  Vadim Kreynin was a huge help with this effort.

Assertion Failures

All new assertions are implemented with a common pattern designed to make assertion failures easier to understand.  Notice how the parameters of the assertion are clearly labeled.  Also, the fact that the values are quoted tells you that the objects involved are actually strings.

Yes... and you can click on the stack trace in the report to go to source.

image

And putting to rest that age-old question "are these two values the same or do they just look that way"?

image

Assertions with Diffs

When comparing two values for equality, we now display detailed character by character differences between their formatted representations.  These differences appear as colored and styled regions in the test report.

In the future, we will be providing a new Visual Studio integrated test runner that will be able to manifest differences as styled output in the test console as well.

image

Assertions Over Collections

MbUnit assertions can be composed to form more complex assertions out of simpler ones.  The Assert.Over syntax provides some useful compositions for performing assertions over collections.

image

image

AssertEx.That...

For C# 3.0 / .Net 3.5 users, MbUnit now offers some new syntax for expressing assertions using lambda expressions using AssertEx.That.  If the expression evaluates to false, MbUnit will try to explain why by printing the values of referenced variables and intermediate sub-expressions that were computed during the assertion.

Suppose we have the following test:

image

When we run this test, it will fail because the last two values of the sequence should have been '13' and '21'.  The assertion failure pretty clearly spells this out for us by telling us all about the variables involved.

image 

Multiple Asserts

Sometimes a test performs multiple related assertions that are logically part of a single verification.  In these scenarios, we would like to be able to continue running the test even if one of the assertions fails.  When the test is finished, we can the report all of the failures together.

There are two ways to do this in MbUnit.

Tag the method with [MultipleAsserts]:

image

Or use the Assert.Multiple function and pass in a delegate:

image

In either case, MbUnit will produce a failure message like this:

image

Stack Trace Filtering

We now filter out framework code from stack traces.  You may have noticed that the assertion failure examples shown above were particularly uncluttered...

Contract Verifiers

Tired of writing tests for exceptions, equality operators, and comparers?  Yann Trévin has made it easier!

Believe it or not, this is a complete test fixture for a custom exception type.

image

And here's what it takes to test the Equals, GetHashCode, operator==, and operator!= methods of our Range class.  We just need to provide a few examples of ranges that are in different equivalence classes.

image

And this is a test for CompareTo, and all of the standard comparison operators.

image

We plan to write more of these soon...

Dependent / Ordered Tests

When writing integration tests, it is sometimes useful to chain several tests together that capture a logical sequence of operations.

MbUnit can capture this chaining either with dependencies:

image

Or with explicit ordering:

image

Icarus

Graham Hay has been hard at work on Icarus.

image 

ReSharper Test Runner

ReSharper v4.1

We now fully support ReSharper v4.1.  (Actually, we have for quite some time now, this is just the first official release with support for it.)

Clickable Stack Traces

One popular request has been to provide clickable stack traces in ReSharper.  We are somewhat limited in our ability to do this well (due to ReSharper API limitations) but we tried our best!

image

Reports

It is now possible to view Condensed and Full test reports in ReSharper.  The test report may provide more details under certain situations.  For example, it will include those assertion diff highlights (which we are unable to display in the ReSharper test result panel itself).

image

Visual Studio Team System Test Runner

Yann Trévin also put a lot of work into the Visual Studio Team System test runner.

You can run tests using the Test View.

image

View results in the Test Results view.

image

And if you double click on the test result, you can view the test report for more information.

image

Notes:

  • You will need Visual Studio 2008 SP1 for this feature.
  • The tests must be located in a project that has the appropriate ProjectTypeGuids specified.

Installer

New Installer

Francois Retief rewrote our old installer using WiX.  It is so much better now!

Highlights:

  • Works on Vista.
  • Supports 640bit architectures.
  • We now provide separate x86 and x64 versions.
  • More configuration options.

CSUnit Adapter

Francois Retief also implemented an adapter for CSUnit.  So now you can use Gallio to run your CSUnit tests.

Please let us know what you think.

TeamCity Integration

Gallio now knows how to emit TeamCity service messages to notice the build server about test progress and test results.

To use it, set the RunnerExtensions argument of the Gallio test runner you are using to "TeamCityExtension,Gallio.TeamCityIntegration".

Examples:

If you are using the Gallio MSBuild task:
    <Gallio RunnerExtensions="TeamCityExtension,Gallio.TeamCityIntegration"
        ... other arguments... />

If you are using the Gallio NAnt task:
    <gallio runner-extension="TeamCityExtension,Gallio.TeamCityIntegration"
        ... other arguments... />

If you are using the Gallio Echo task:
    Gallio.Echo /e:TeamCityExtension,Gallio.TeamCityIntegration ... other arguments...

Note: We plan to automatically detect TeamCity in future releases.

Gallio.Ambience

This release includes an initial preview of Gallio.Ambience.  Ambience manages what we are tentatively calling the Ambient object store.

The Ambient object store is a shared lightweight repository for intermediate test data.  It is like a persistent whiteboard that is used to pass information from one test to another or to store it for subsequent analysis.  In effect, you can use it to create stateful tests.

A stateful test is one which may load or store data generated by previous test runs and use it for some purpose.  This practice is strongly discouraged in unit testing best practice because it introduces a dependency on previous runs.  However, it can be useful for integration testing of stateful systems that cannot easily be torn down and built up again.

Testers will commonly resort to flat files for storing test data between runs.  This system works well on a local machine but it is problematic for distributed testing clusters unless the files are stored on a network drive (where they risk corruption).

Instead of using flat files, Gallio Ambience is client/server based.  The server in this case is a Db4o Object Database instance.  Clients may connect to it remotely to store and retrieve data.  LINQ query syntax is supported.

Example

First we write a test that stores some data in the Ambient object store for later.

image

image

Then we write another test which consumes the data in the Ambient object store.

image

In this way we can test stateful processes using one or more tests that communicate via the Ambient object store.

Note:

  • If you are able to refactor your system to avoid the need for stateful tests then we recommend doing so.  Stateful tests are inherently more brittle than properly isolated unit tests.  That said, we think the can be useful for solving certain otherwise messy problems.
  • This feature is new and experimental.  We welcome your feedback!

Documentation

API Documentation

We have published the Gallio and MbUnit v3 API documentation on the website here.

Gallio Book

The Gallio Book is very much a work in progress but we have a fairly complete outline and some content produced so far.  It is also published up on the website here.

External Tools Compatibility

We've updated integration with a variety of external tools.

  • CCNet v1.0.4
  • CSUnit v2.0.5
  • NUnit v2.4.8 (now supports NUnit add-ins correctly)
  • Pex v0.7
  • ReSharper v3.1, v4.0, v4.1
  • TeamCity v3.1, v4.0 EAP
  • xUnit.Net v1.0.3

Misc. Improvements

  • Improved backwards compatibility with older versions.
  • More careful handling of the runtime environment to avoid certain exotic assembly version conflict issues in TestDriven.Net, ReSharper and Visual Studio.
  • Performance improvements in upcoming versions of TestDriven.Net.
  • Lots of bug fixes.

Upcoming

A few features did not quite make it into v3.0.4 but will appear in upcoming releases:

  • New plug-in model (will significantly improve startup performance).
  • TeamCity auto-detection.
  • NCover 2 support.  It's already in there but there remain some unresolved issues to address.
  • A new Visual Studio integrated test runner.
  • Hosting Gallio inside exotic test environments such as ASP.Net and AutoCAD.
  • Launching x86 testing processes on x64.

Mono

Gallio almost runs on Mono.   We have made cross-platform support a priority for Gallio and MbUnit but there is still some work remaining.  The build server now regularly runs tests using Mono.

We are seeking volunteers to help make full Mono support a reality in upcoming releases.

Migration Guide

If you are migrating code from MbUnit v2 or from previous versions of MbUnit v3, the following tables will help you.

What To Do If XYZ Is Not In This List

If one or more of your favorite methods or attributes from MbUnit v2 is not in this list here's what to do:

  1. Consult the tables for a replacement.
  2. Old assertions are available in the MbUnit.Compatibility.dll.  They have been renamed with the prefix Old* to clearly distinguish them.
  3. If you still cannot find what you need, all is not lost.  Remember that Gallio can run tests from MbUnit v2 and v3 at the same time.  Several old versioning issues have been resolved so this process should be smoother too.
  4. We are planning to replace / upgrade some (but not all) of the deprecated methods and attributes.  If there are any in particular you think we should tackle first, please open an issue in our issue tracker.
  5. Please feel free to make other suggestions for improvement.

Assertion Migration Guide

This table describes changes to assertions:

MbUnit v3 (NEW) MbUnit v2 (OLD)
Assert.AreApproximatelyEqual (v3.0.5) Assert.AreEqual (with delta)
Assert.AreElementsEqual CollectionAssert.AreElementsEqual
Assert.AreElementsNotEqual <n/a>
Assert.AreEqual Assert.AreEqual
ArrayAssert.AreEqual
CollectionAssert.AreEqual
CollectionAssert.AreEquivalent
GenericAssert.AreEqual
StringAssert.AreEqualIgnoreCase
Assert.AreElementsEqualIgnoringOrder <n/a>
Assert.AreNotEqual Assert.AreNotEqual
CollectionAssert.AreNotEqual
CollectionAssert.AreNotEquivalent
Assert.AreNotApproximatelyEqual (v3.0.5) Assert.AreNotEqual (with delta)
Assert.AreNotSame Assert.AreNotSame
Assert.AreSame Assert.AreSame
<none> Assert.AreValueEqual
Assert.Between Assert.Between
Assert.Contains Assert.Contains
Assert.In
CollectionAssert.Contains
StringAssert.Contains
Assert.ContainsKey Assert.In
Assert.DoesNotContain Assert.NotIn
CollectionAssert.DoesNotContain
StringAssert.DoesNotContain
Assert.DoesNotContainKey Assert.NotIn
Assert.DoesNotThrow <n/a>
Assert.EndsWith StringAssert.EndsWith
Assert.Fail Assert.Fail
Assert.FullMatch StringAssert.FullMatch
Assert.GreaterThan Assert.Greater
Assert.GreaterThan
Assert.GreaterThanOrEqualTo Assert.GreaterEqualThan
Assert.Inconclusive <n/a>
Assert.IsAssignableFrom Assert.IsAssignableFrom
ReflectionAssert.IsAssignableFrom
Assert.IsEmpty Assert.IsEmpty
GenericAssert.IsEmpty
StringAssert.IsEmpty
Assert.IsFalse Assert.IsFalse
Assert.IsInstanceOfType Assert.IsInstanceOfType
ReflectionAssert.IsInstanceOf
<none> Assert.IsNaN
Assert.IsNotAssignableFrom Assert.IsNotAssignableFrom
Assert.IsNotEmpty Assert.IsNotEmpty
GenericAssert.IsNotEmpty
StringAssert.IsNonEmpty
Assert.IsNotInstanceOfType Assert.IsNotInstanceOfType
Assert.IsNotNull Assert.IsNotNull
Assert.IsNull Assert.IsNull
Assert.IsTrue Assert.IsTrue
Assert.LessThan Assert.Less
Assert.LowerThan
Assert.LessThanOrEqualTo Assert.LowerEqualThan
Assert.Like StringAssert.Like
Assert.Multiple <n/a>
Assert.NotBetween Assert.NotBetween
Assert.NotLike StringAssert.NotLike
Assert.StartsWith StringAssert.StartsWith
Assert.Throws <n/a>
TestLog.Warning.WriteLine Assert.Warning
<none> CollectionAssert.AllItemsAreInstancesOfType
<none> CollectionAssert.AllItemsAreNotNull
<none> CollectionAssert.AllItemsAreUnique
<none> CollectionAssert.AreCountEqual
<none> CollectionAssert.AreIsSynchronizedEqual
<none> CollectionAssert.AreSyncRootEqual
<none> CollectionAssert.IsCountCorrect
<none> CollectionAssert.IsNotSubsetOf
<none> CollectionAssert.IsNotSynchronized
<none> CollectionAssert.IsSubsetOf
<none> CollectionAssert.IsSynchronized
<none> FileAssert.AreEqual
<none> FileAssert.AreStreamContentEqual
<none> ReflectionAssert.HasConstructor
<none> ReflectionAssert.HasDefaultConstructor
<none> ReflectionAssert.HasField
<none> ReflectionAssert.HasMethod
<none> ReflectionAssert.IsSealed
<none> ReflectionAssert.NotCreatable
<none> ReflectionAssert.ReadOnlyProperty
<none> ReflectionAssert.WriteOnlyProperty
<none> CompilerAssert.*
<none> ControlAssert.*
<none> DataAssert.*
<none> PerfAssert.*
<none> SecurityAssert.*
<none> SerialAssert.*
<none> WebAssert.*
<none> XmlAssert.*

 

Attribute Migration Guide

This table describes changes to attributes:

MbUnit v3 (NEW) MbUnit v2 (OLD)
[Annotation] <n/a>
[ApartmentState] Apartment property of [TestFixture]
[AssemblyFixture] [AssemblyCleanup]
[AssemblyResolver] [AssemblyResolver]
[Author] [Author]
[Bind] <n/a>
[Category] [TestCategory]
[FixtureCategory]
[Column] <n/a>
[CombinatorialJoin] (optional) CombinationType = Cartesion property of [CombinatorialTest]
[CsvData] <n/a>
[DependsOnAssembly] [AssemblyDependsOn]
[DependsOn] [DependsOn]
[Description] Description property of [TestFixture] and [Test]
[ExpectedArgumentException] [ExpectedArgumentException]
[ExpectedArgumentNullException] [ExpectedArgumentNullException]
[ExpectedArgumentOutOfRangeException] <n/a>
[ExpectedException] [ExpectedException]
[Explicit] <n/a>
[Factory] [Factory] (note: very different semantics)
[FixtureInitializer] <n/a>
[FixtureSetUp] [TestFixtureSetUp]
[FixtureTearDown] [TestFixtureTearDown]
[Header] <n/a>
[Ignore] [Ignore]
[Importance] [Importance]
[Metadata] <n/a>
[MultipleAsserts] <n/a>
[Name] <n/a>
[PairwiseJoin] CombinationType = AllPairs property of [CombinatorialTest]
[Parameter] <n/a>
[Pending] <n/a>
[Repeat] [Repeat]
[RepeatTest]
[Rollback] [RollBack]
[RollBack2]
[Row] [Row]
[SequentialJoin] <n/a>
[SetUp] [SetUp]
[TearDown] [TearDown]
[Test] [Test]
[RowTest]
[CombinatorialTest]
[TestFixture] (optional!) [TestFixture]
[TestsOn] [TestsOn]
[ThreadedRepeat] [ThreadedRepeat]
[Timeout] TimeOut property of [TestFixture]
[VerifyComparisonContract] <n/a>
[VerifyEqualityContract] <n/a>
[VerifyExceptionContract] <n/a>
<n/a> [CollectionIndexingFixture]
<n/a> [CollectionOrderFixture]
<n/a> [CompositeFixture]
<n/a> [Copy]
<n/a> [CopyToProvider]
<n/a> [CurrentFixture]
<n/a> [DataFixture]
<n/a> [DataProvider]
<n/a> [Duration]
<n/a> [EnumerationFixture]
<n/a> [ExtractResource]
<n/a> [ForEachTest]
<n/a> [Grammar]
<n/a> [GrammarFixture]
<n/a> [IndexerProvider]
<n/a> [Information]
<n/a> [IntIndexerProvider]
<n/a> [MultipleCulture]
<n/a> [NamespaceProvider]
<n/a> [NotInheritedExpectedException]
<n/a> [NullableAttribute]
<n/a> [Pelikhan]
<n/a> [PerfCounter]
<n/a> [PostIt]
Unncessary. [ProcessTestFixture]
<n/a> [Provider]
<n/a> [ProviderFactory]
<n/a> [Read]
<n/a> [ResourceXmlDataProvider]
<n/a> [RestoreDatabaseFirst]
<n/a> [Seed]
<n/a> [SqlRestoreInfo]
<n/a> [SuiteProvider]
Order property of [Test] or [TestFixture] [TestSequence]
<n/a> [TestSuite]
<n/a> [TestSuiteFixture]
<n/a> [TestSuiteSetUp]
Unncessary. [TypeFixture]
<n/a> [UsingEnum]
Similar to [Factory] [UsingFactories]
<n/a> [UsingImplementations]
<n/a> [UsingLinear]
Similar to [Column] [UsingLiterals]
<n/a> [Write]
<n/a> [XmlDataProvider]

 

Edit: Fixed Vadim's name  :-)

Technorati Tags: ,

Friday, September 19, 2008

Quickie

Sorry for the long silence.  I've been super busy lately.

Anyways... I'll tantalize you with this: Gallio v3.0.4 is almost ready to ship.  We're just cleaning up some loose ends and putting on the finishing touches.

I am really looking forward to making the final announcement.  There's lots of cool (and useful) new stuff on the way!

Friday, August 22, 2008

How to Find Me

I live in the East Bay and work in San Francisco.  I'm always keen on meeting smart and interesting people.  If you're local and would like to hang out, shoot me an email.

Upcoming Conferences

November 17th-21st: QCon

On Thursday I'll be giving a talk about Gallio and MbUnit at QCon in the ALT.Net track.  I'm planning to talk about the architecture of Gallio and demonstrate a few ways that it can be extended to build new test tools. I will also discuss how MbUnit integrates with Gallio and how other test frameworks can do the same.

November 8th-9th: Silicon Valley Code Camp

I'm planning to attend Silicon Valley Code Camp, a free conference in the Bay Area.  I've also signed up to give a talk similar to the one at QCon.  Looks like a great opportunity to meet more local talent.

Community Events

I sing Tenor with San Francisco City Chorus, a local amateur choir.  It's a lot of fun and we're always looking for new members.

My wife and I also sing with the Mount Diablo Unitarian Universalist Church.  I'm still not sure about this whole "church" thing, but it's a nice community of hippies genuine, empathetic, forward-thinking, individualists.

I used to do English Country dancing with the Merrie Pryanksters at the California Renaissance Faires and do Irish dancing at the Starry Plough Pub in Berkeley.  Lately my participation has been superceded by other activities but I still hang out.

Occasionally you'll find me hanging out with other Canadian members of the Digital Moose Lounge at a hockey game or Canada Day picnic.

Thursday, August 21, 2008

Diffs in Assertion Failures

Just one of many features I've been working on lately.

More to come... now it's wayyy past bedtime.

Spot the Case-Change

image

Something More Complex

image

I'm considering using strike-through to give added emphasis to changed or deleted text from the expected value.  What do you think?

Technorati Tags: ,

Tuesday, July 15, 2008

Getting Stuff Done

Liz and I have to move next month.  The landlady simply wants to move back into her house so she will not renew the lease.

Thus begins the process of dismantling The Cave and going somewhere else.  But where?  We have searched all over for homes to buy or to rent over the past two months and have not found anything acceptable yet.

A small part of me would like nothing more than for someone else (perhaps my employer) to take care of these details so I can move on with the process of getting stuff done.  Foolishness!  Alas.

Wednesday, July 9, 2008

Attribution and the Golden Rule

Gallio uses the Apache License, Version 2.0.

Today I discovered a breach of that license wherein an author of another Open Source project copied code from Gallio and removed the copyright notices and attribution.

I have contacted the author and hope to resolve the matter amicably.

Giving Credit Where it is Due

You may have noticed that I take pains to acknowledge the work of others when I borrow their ideas.  For example, I credit xUnit.Net for their idea of abstracting the reflection layer (among other things).  That simple idea has shaped a fair bit of the Gallio test object model so I'm grateful to have encountered it.

On an even grander scale, there are many others whom I credit for the development of unit testing and the idiomatic form it has taken in .Net.  MbUnit borrows heavily from NUnit, JUnit and other frameworks, not to mention the work of Kent Beck and other founders.

Professional Courtesy

Recently I have noticed others borrowing ideas from Gallio and MbUnit.  I think it's great!  Imitation is the sincerest form of flattery.  But I do feel something is missing.

Open Source software does not get built in a vacuum.  There are many people involved in the conception, development, promotion and distribution of software.  We depend crucially upon the goodwill of others and so it behooves us to act with humility and respect for one another and our respective projects.

The more widely we disseminate the good ideas of others, the more positive impact they will have.

Thanks.

Technorati Tags: ,,

Tuesday, June 17, 2008

Announcing Gallio Alpha 3 Update 3

Here comes the next update to Gallio and MbUnit v3.

This is an interim release with a few new features and updates.

Download here: (link)

New and Noteworthy

ReSharper v4.0 Final Integration

This update now supports the final release of ReSharper v4.0.

Enjoy!

MbUnit [Factory] Attribute

Suppose we've got a function like this:

image

To test it, we'll need some different kinds of objects.

So let's use a factory:

image

That's quite simple, assuming we only want to use single objects for each run.

But we can do better.  We can also return arrays of objects:

image

However, this doesn't test the case where the value was null.  In that case, the formatter function is supposed to throw an ArgumentNullException.

We can simulate the behavior of the [ExpectedException] attribute by returning additional metadata along with the data item.  Of course, we can include other metadata too, like test descriptions, and other data.

image

 

MbUnit [Repeat] and [ThreadedRepeat] Attributes

Sometimes when testing features with non-deterministic behavior, it's useful to be able to run a test repeatedly.

Use the [Repeat] and [ThreadedRepeat] attributes in MbUnit to run tests sequentially or concurrently.

Since the attribute is a test decorator, it may be used with most any kind of test, including data-driven tests.  It can also be used on fixtures, if you like.

image

Each run appears as a separate item in the report.  They are considered dynamic test steps that are nested within the test.

image 

CSV Files with Metadata

Several people have requested the ability to include additional metadata in rows that are specified by CSV files using the [CsvData] attribute in MbUnit.

Now you can.

Just enclose the name of the column that contains metadata with '[' and ']' and that column will be exposed as a metadata item whose key is the name of the column.

So given this test:

image

And this data file:

Item, Price, [ConsumedBy]
Apples, 1.00, Worm
Bananas, 1.50, Monkey
Cookies, 2.00, "Cookie Monster"

We get this output in the report:

image

Notice that the metadata appears in the report as well as the name of the resource and the line number where the data came from!

Technorati Tags: ,