See you at Microsoft PDC

I will be at the Microsoft PDC.

If you're there then find me walking around wearing a kilt, as usual.  I will also be a member of a lunchtime panel on the Future of Unit Testing, probably on Wednesday.

See you there!

Leaky Exceptions

Take a look at this exception:

System.Runtime.Serialization.SerializationException: Unable to find assembly 'Castle.MicroKernel, Version=1.0.3.0, Culture=neutral, PublicKeyToken=407dd0808d44fbdc'.

<snip>

at System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage.FixupForNewAppDomain()
at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(IMessage reqMsg)

Exception rethrown at [0]: 
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Gallio.TDNetRunner.Core.IProxyTestRunner.Run(IFacadeTestListener testListener, String assemblyPath, String cref)

What Happened

1. We made a call into a remote AppDomain.
2. The call was processed and now we are trying to produce a return message.
3. The call threw an exception of some type.
4. The exception could not be deserialized by the caller because the exception is defined in the Castle.MicroKernel assembly.
5. As a result, the remoting library "helpfully" replaced the real exception with a SerializationException instead.

Diagnosis

The server is leaking an exception of a type that the client does not understand.

The remote call interface is a Leaky Abstraction.  Technically, the implementation of that interface is leaky.

Fixes

Case 1: Wrap the inner exception.

There are several things we can do to fix leaky exceptions.  In most cases, it's simple enough to wrap the exception.

public void Wibble(string wibbleUser, string wibbleParam)
{
    try
    {
        wibbleAuditor.RecordWibble(wibbleUser, wibbleParam);
    }
    catch (AuditException ex)
    {
        throw new WibbleException("Could not write audit log message before wibbling.  Wibble aborted.");
    }
    try
    {
        wibbleHandler.Execute(wibbleParam);
    }
    catch (WibbleProcessorException ex)
    {
        throw new WibbleException("An error occurred while wibbling.", ex);
    }
}

Case 2: Capture a description of the inner exception as a string.

As we saw in the case of our SerializationException above, in some cases we cannot actually wrap the exception since the "InnerException" itself might not be serializable or deserializable.

In just those cases, we can do something like the following instead:

throw new WibbleException(String.Format("An error occurred while wibbling.  {0}", ex));

Capturing the exception as a string lets us smuggle it over into places where the exception type might not be accessible.  However, we do lose some flexibility in how the client can handle the exception.

Case 3: Discard the inner exception.

There is such a thing as too much information.

Sometimes revealing the contents of the inner exception to the client at all may constitute a security risk (leaking sensitive data in the stack trace or the message) or it may just provide way more details than the client can understand.

In these cases, it's best to provide a clear description of the exception in-situ without relying on the information from the internal exception at all.  Such an exception might instead refer to another source for obtaining the nitty-gritty details for low-level diagnosis.

throw new WibbleException("An error occurred while wibbling.  Consult the wibble processor log for details.");

Similar Problems

A method that throws an exception of an internal type.  This effectively prevents the client from filtering by exception type.

A method that throws an exception of an undocumented type.  While the exception may be caught, the client does not know to expect it.

A method that throws an exception of a vague type or with a vague message.  While the exception may be filtered and may be expected, the client will not be able to make sense of its contents.

An improperly defined custom exception type that lacks the special deserialization constructor and [Serializable].  Such an exception cannot cross remoting boundaries even if it was expected to.

An exception that reveals sensitive information as part of the exception message.

Examples:

• Allowing a generic platform exception such as Win32Exception or COMException to escape through a high-level supposedly platform-independent interface.
• Allowing an exception thrown by a some lower-level component or implementation detail that the client does not know about to escape.
• Allowing an exception of a type not specified in the published documentation to escape.
• Allowing a FileNotFoundException from an internal (non-obvious) I/O operation to be returned to the client without wrapping it with a higher-level exception type to explain what was going on.
• Throwing an exception that contains a database connection string or user password in the message.

Full Circle

In this case I modified my code to return a SafeException containing the inner exception message as a string (applied case #2 above).  Here's what was hiding behind that SerializationException.

Gallio.Loader.SafeException: Gallio.Runtime.RuntimeException: Could not resolve service of type Gallio.Runner.ITestRunnerFactory. ---> Castle.MicroKernel.Resolvers.DependencyResolverException: Could not resolve non-optional dependency for 'NCoverIntegration2.NCoverTestRunnerFactory' (Gallio.Runner.DefaultTestRunnerFactory). Parameter 'name' type 'System.String'
   at Castle.MicroKernel.Resolvers.DefaultDependencyResolver.Resolve(CreationContext context, ISubDependencyResolver parentResolver, ComponentModel model, DependencyModel dependency)

<snip>

Ahh!  Now I know what's going on so I can fix it...

Building Stuff

At yellowpages.com, I lead a Tools & Automation team.  In case you've ever wondered where MbUnit and Gallio fit into my work life, one of my responsibilities is managing a team of Test Automation Engineers.  We also use these tools for unit testing of our core systems.

Another one of my responsibilities is Release Engineering which is sort of an umbrella term governing processes related to Build, Deploy, Continuous Integration, System Monitoring, Defect Tracking, Source Control and the like.  In short, stuff that enables us to ship high-quality code when we want to.

I have built lots of great stuff as part of my job here:

• For Unit Testing and Integration Testing, I built numerous extensions for MbUnit then eventually took up the lead on MbUnit v3 and Gallio.
• For System Monitoring, I built a tool called Odin that has yet to be open-sourced itself but which did produce 3 OSS libraries, Castle.FlexBridge, Castle.Components.Cache and Castle.Components.Scheduler.
• For FxCop integration in Visual Studio 2005, I built FxCopAddIn (now defunct).
• For Build, I built MetaBuild, a lightweight modular build system.
• For Deploy, I'm going to be building or borrowing something to replace what we've got...  I don't know what yet but I know I will want more help.
• I've contributed little bits and pieces here and there to several other great OSS projects like Castle, Rhino.Mocks, WatiN, CruiseControl.Net and more.

I have also built a strong team that I am very proud of.  We build great things together!

Here are some other things my team has built:

• A test scheduling and reporting system.
• A diagnostic "inspector" tool for troubleshooting applications.
• An automated test spider tool to search for broken links.
• A comprehensive model and system test suite.
• A phone-testing harness.

And there's tons more stuff to build!  Which reminds me...

Yet another one of my responsibilities is to recruit great talent.

Want to build cool stuff?  Call me!

Gallio and MbUnit v3.0.4 build 485 Released

We have released a minor update to Gallio and MbUnit v3.0.4 to resolve a problem that occurs when a user creates a partial copy of Gallio which is missing certain required plug-in assemblies.  Gallio will now silently disable those plug-ins.

If you plan to copy Gallio and MbUnit into your source tree then you may benefit from this update.  Without it you may observe a "ConfigurationErrorsException" due to the Visual Studio plug-in's dependency on the GAC.  (Of course, you could just delete the Visual Studio plug-in from your source tree since it will not do anything useful in there anyway.)

Download Link: Gallio Downloads

Technorati Tags: mbunit,gallio

Announcing Gallio and MbUnit v3.0.4

Today we are releasing v3.0.4 of Gallio and MbUnit.  This release incorporates many new features as well improvements for robustness.

New asserts, assertion diffs, 0rdered tests, Rollback, Repeat, ThreadedRepeat, R# 4.1, VSTS, CSUnit, TeamCity, Gallio.Ambience, clickable report links (IE/Firefox integration), and more...

Try it out!  Please be aware that there are now separate x86 and x64 installers.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

MbUnit

Assertions Refresh

We have consolidated and significantly improved the set of built-in MbUnit assertions.  Refer to the migration guide at the end of this post for more information.  Vadim Kreynin was a huge help with this effort.

Assertion Failures

All new assertions are implemented with a common pattern designed to make assertion failures easier to understand.  Notice how the parameters of the assertion are clearly labeled.  Also, the fact that the values are quoted tells you that the objects involved are actually strings.

Yes... and you can click on the stack trace in the report to go to source.

And putting to rest that age-old question "are these two values the same or do they just look that way"?

Assertions with Diffs

When comparing two values for equality, we now display detailed character by character differences between their formatted representations.  These differences appear as colored and styled regions in the test report.

In the future, we will be providing a new Visual Studio integrated test runner that will be able to manifest differences as styled output in the test console as well.

Assertions Over Collections

MbUnit assertions can be composed to form more complex assertions out of simpler ones.  The Assert.Over syntax provides some useful compositions for performing assertions over collections.

AssertEx.That...

For C# 3.0 / .Net 3.5 users, MbUnit now offers some new syntax for expressing assertions using lambda expressions using AssertEx.That.  If the expression evaluates to false, MbUnit will try to explain why by printing the values of referenced variables and intermediate sub-expressions that were computed during the assertion.

Suppose we have the following test:

When we run this test, it will fail because the last two values of the sequence should have been '13' and '21'.  The assertion failure pretty clearly spells this out for us by telling us all about the variables involved.

 

Multiple Asserts

Sometimes a test performs multiple related assertions that are logically part of a single verification.  In these scenarios, we would like to be able to continue running the test even if one of the assertions fails.  When the test is finished, we can the report all of the failures together.

There are two ways to do this in MbUnit.

Tag the method with [MultipleAsserts]:

Or use the Assert.Multiple function and pass in a delegate:

In either case, MbUnit will produce a failure message like this:

Stack Trace Filtering

We now filter out framework code from stack traces.  You may have noticed that the assertion failure examples shown above were particularly uncluttered...

Contract Verifiers

Tired of writing tests for exceptions, equality operators, and comparers?  Yann Trévin has made it easier!

Believe it or not, this is a complete test fixture for a custom exception type.

And here's what it takes to test the Equals, GetHashCode, operator==, and operator!= methods of our Range class.  We just need to provide a few examples of ranges that are in different equivalence classes.

And this is a test for CompareTo, and all of the standard comparison operators.

We plan to write more of these soon...

Dependent / Ordered Tests

When writing integration tests, it is sometimes useful to chain several tests together that capture a logical sequence of operations.

MbUnit can capture this chaining either with dependencies:

Or with explicit ordering:

Icarus

Graham Hay has been hard at work on Icarus.

 

ReSharper Test Runner

ReSharper v4.1

We now fully support ReSharper v4.1.  (Actually, we have for quite some time now, this is just the first official release with support for it.)

Clickable Stack Traces

One popular request has been to provide clickable stack traces in ReSharper.  We are somewhat limited in our ability to do this well (due to ReSharper API limitations) but we tried our best!

Reports

It is now possible to view Condensed and Full test reports in ReSharper.  The test report may provide more details under certain situations.  For example, it will include those assertion diff highlights (which we are unable to display in the ReSharper test result panel itself).

Visual Studio Team System Test Runner

Yann Trévin also put a lot of work into the Visual Studio Team System test runner.

You can run tests using the Test View.

View results in the Test Results view.

And if you double click on the test result, you can view the test report for more information.

Notes:

• You will need Visual Studio 2008 SP1 for this feature.
• The tests must be located in a project that has the appropriate ProjectTypeGuids specified.

Installer

New Installer

Francois Retief rewrote our old installer using WiX.  It is so much better now!

Highlights:

• Works on Vista.
• Supports 640bit architectures.
• We now provide separate x86 and x64 versions.
• More configuration options.

CSUnit Adapter

Francois Retief also implemented an adapter for CSUnit.  So now you can use Gallio to run your CSUnit tests.

Please let us know what you think.

TeamCity Integration

Gallio now knows how to emit TeamCity service messages to notice the build server about test progress and test results.

To use it, set the RunnerExtensions argument of the Gallio test runner you are using to "TeamCityExtension,Gallio.TeamCityIntegration".

Examples:

If you are using the Gallio MSBuild task:
    <Gallio RunnerExtensions="TeamCityExtension,Gallio.TeamCityIntegration"
        ... other arguments... />

If you are using the Gallio NAnt task:
    <gallio runner-extension="TeamCityExtension,Gallio.TeamCityIntegration"
        ... other arguments... />

If you are using the Gallio Echo task:
    Gallio.Echo /e:TeamCityExtension,Gallio.TeamCityIntegration ... other arguments...

Note: We plan to automatically detect TeamCity in future releases.

Gallio.Ambience

This release includes an initial preview of Gallio.Ambience.  Ambience manages what we are tentatively calling the Ambient object store.

The Ambient object store is a shared lightweight repository for intermediate test data.  It is like a persistent whiteboard that is used to pass information from one test to another or to store it for subsequent analysis.  In effect, you can use it to create stateful tests.

A stateful test is one which may load or store data generated by previous test runs and use it for some purpose.  This practice is strongly discouraged in unit testing best practice because it introduces a dependency on previous runs.  However, it can be useful for integration testing of stateful systems that cannot easily be torn down and built up again.

Testers will commonly resort to flat files for storing test data between runs.  This system works well on a local machine but it is problematic for distributed testing clusters unless the files are stored on a network drive (where they risk corruption).

Instead of using flat files, Gallio Ambience is client/server based.  The server in this case is a Db4o Object Database instance.  Clients may connect to it remotely to store and retrieve data.  LINQ query syntax is supported.

Example

First we write a test that stores some data in the Ambient object store for later.

Then we write another test which consumes the data in the Ambient object store.

In this way we can test stateful processes using one or more tests that communicate via the Ambient object store.

Note:

• If you are able to refactor your system to avoid the need for stateful tests then we recommend doing so.  Stateful tests are inherently more brittle than properly isolated unit tests.  That said, we think the can be useful for solving certain otherwise messy problems.
• This feature is new and experimental.  We welcome your feedback!

Documentation

API Documentation

We have published the Gallio and MbUnit v3 API documentation on the website here.

Gallio Book

The Gallio Book is very much a work in progress but we have a fairly complete outline and some content produced so far.  It is also published up on the website here.

External Tools Compatibility

We've updated integration with a variety of external tools.

• CCNet v1.0.4
• CSUnit v2.0.5
• NUnit v2.4.8 (now supports NUnit add-ins correctly)
• Pex v0.7
• ReSharper v3.1, v4.0, v4.1
• TeamCity v3.1, v4.0 EAP
• xUnit.Net v1.0.3

Misc. Improvements

• Improved backwards compatibility with older versions.
• More careful handling of the runtime environment to avoid certain exotic assembly version conflict issues in TestDriven.Net, ReSharper and Visual Studio.
• Performance improvements in upcoming versions of TestDriven.Net.
• Lots of bug fixes.

Upcoming

A few features did not quite make it into v3.0.4 but will appear in upcoming releases:

• New plug-in model (will significantly improve startup performance).
• TeamCity auto-detection.
• NCover 2 support.  It's already in there but there remain some unresolved issues to address.
• A new Visual Studio integrated test runner.
• Hosting Gallio inside exotic test environments such as ASP.Net and AutoCAD.
• Launching x86 testing processes on x64.

Mono

Gallio almost runs on Mono.   We have made cross-platform support a priority for Gallio and MbUnit but there is still some work remaining.  The build server now regularly runs tests using Mono.

We are seeking volunteers to help make full Mono support a reality in upcoming releases.

Migration Guide

If you are migrating code from MbUnit v2 or from previous versions of MbUnit v3, the following tables will help you.

What To Do If XYZ Is Not In This List

If one or more of your favorite methods or attributes from MbUnit v2 is not in this list here's what to do:

1. Consult the tables for a replacement.
2. Old assertions are available in the MbUnit.Compatibility.dll.  They have been renamed with the prefix Old* to clearly distinguish them.
3. If you still cannot find what you need, all is not lost.  Remember that Gallio can run tests from MbUnit v2 and v3 at the same time.  Several old versioning issues have been resolved so this process should be smoother too.
4. We are planning to replace / upgrade some (but not all) of the deprecated methods and attributes.  If there are any in particular you think we should tackle first, please open an issue in our issue tracker.
5. Please feel free to make other suggestions for improvement.

Assertion Migration Guide

This table describes changes to assertions:

MbUnit v3 (NEW)	MbUnit v2 (OLD)
Assert.AreApproximatelyEqual (v3.0.5)	Assert.AreEqual (with delta)
Assert.AreElementsEqual	CollectionAssert.AreElementsEqual
Assert.AreElementsNotEqual	<n/a>
Assert.AreEqual	Assert.AreEqual ArrayAssert.AreEqual CollectionAssert.AreEqual CollectionAssert.AreEquivalent GenericAssert.AreEqual StringAssert.AreEqualIgnoreCase
Assert.AreElementsEqualIgnoringOrder	<n/a>
Assert.AreNotEqual	Assert.AreNotEqual CollectionAssert.AreNotEqual CollectionAssert.AreNotEquivalent
Assert.AreNotApproximatelyEqual (v3.0.5)	Assert.AreNotEqual (with delta)
Assert.AreNotSame	Assert.AreNotSame
Assert.AreSame	Assert.AreSame
<none>	Assert.AreValueEqual
Assert.Between	Assert.Between
Assert.Contains	Assert.Contains Assert.In CollectionAssert.Contains StringAssert.Contains
Assert.ContainsKey	Assert.In
Assert.DoesNotContain	Assert.NotIn CollectionAssert.DoesNotContain StringAssert.DoesNotContain
Assert.DoesNotContainKey	Assert.NotIn
Assert.DoesNotThrow	<n/a>
Assert.EndsWith	StringAssert.EndsWith
Assert.Fail	Assert.Fail
Assert.FullMatch	StringAssert.FullMatch
Assert.GreaterThan	Assert.Greater Assert.GreaterThan
Assert.GreaterThanOrEqualTo	Assert.GreaterEqualThan
Assert.Inconclusive	<n/a>
Assert.IsAssignableFrom	Assert.IsAssignableFrom ReflectionAssert.IsAssignableFrom
Assert.IsEmpty	Assert.IsEmpty GenericAssert.IsEmpty StringAssert.IsEmpty
Assert.IsFalse	Assert.IsFalse
Assert.IsInstanceOfType	Assert.IsInstanceOfType ReflectionAssert.IsInstanceOf
<none>	Assert.IsNaN
Assert.IsNotAssignableFrom	Assert.IsNotAssignableFrom
Assert.IsNotEmpty	Assert.IsNotEmpty GenericAssert.IsNotEmpty StringAssert.IsNonEmpty
Assert.IsNotInstanceOfType	Assert.IsNotInstanceOfType
Assert.IsNotNull	Assert.IsNotNull
Assert.IsNull	Assert.IsNull
Assert.IsTrue	Assert.IsTrue
Assert.LessThan	Assert.Less Assert.LowerThan
Assert.LessThanOrEqualTo	Assert.LowerEqualThan
Assert.Like	StringAssert.Like
Assert.Multiple	<n/a>
Assert.NotBetween	Assert.NotBetween
Assert.NotLike	StringAssert.NotLike
Assert.StartsWith	StringAssert.StartsWith
Assert.Throws	<n/a>
TestLog.Warning.WriteLine	Assert.Warning
<none>	CollectionAssert.AllItemsAreInstancesOfType
<none>	CollectionAssert.AllItemsAreNotNull
<none>	CollectionAssert.AllItemsAreUnique
<none>	CollectionAssert.AreCountEqual
<none>	CollectionAssert.AreIsSynchronizedEqual
<none>	CollectionAssert.AreSyncRootEqual
<none>	CollectionAssert.IsCountCorrect
<none>	CollectionAssert.IsNotSubsetOf
<none>	CollectionAssert.IsNotSynchronized
<none>	CollectionAssert.IsSubsetOf
<none>	CollectionAssert.IsSynchronized
<none>	FileAssert.AreEqual
<none>	FileAssert.AreStreamContentEqual
<none>	ReflectionAssert.HasConstructor
<none>	ReflectionAssert.HasDefaultConstructor
<none>	ReflectionAssert.HasField
<none>	ReflectionAssert.HasMethod
<none>	ReflectionAssert.IsSealed
<none>	ReflectionAssert.NotCreatable
<none>	ReflectionAssert.ReadOnlyProperty
<none>	ReflectionAssert.WriteOnlyProperty
<none>	CompilerAssert.*
<none>	ControlAssert.*
<none>	DataAssert.*
<none>	PerfAssert.*
<none>	SecurityAssert.*
<none>	SerialAssert.*
<none>	WebAssert.*
<none>	XmlAssert.*

 

Attribute Migration Guide

This table describes changes to attributes:

MbUnit v3 (NEW)	MbUnit v2 (OLD)
[Annotation]	<n/a>
[ApartmentState]	Apartment property of [TestFixture]
[AssemblyFixture]	[AssemblyCleanup]
[AssemblyResolver]	[AssemblyResolver]
[Author]	[Author]
[Bind]	<n/a>
[Category]	[TestCategory] [FixtureCategory]
[Column]	<n/a>
[CombinatorialJoin] (optional)	CombinationType = Cartesion property of [CombinatorialTest]
[CsvData]	<n/a>
[DependsOnAssembly]	[AssemblyDependsOn]
[DependsOn]	[DependsOn]
[Description]	Description property of [TestFixture] and [Test]
[ExpectedArgumentException]	[ExpectedArgumentException]
[ExpectedArgumentNullException]	[ExpectedArgumentNullException]
[ExpectedArgumentOutOfRangeException]	<n/a>
[ExpectedException]	[ExpectedException]
[Explicit]	<n/a>
[Factory]	[Factory] (note: very different semantics)
[FixtureInitializer]	<n/a>
[FixtureSetUp]	[TestFixtureSetUp]
[FixtureTearDown]	[TestFixtureTearDown]
[Header]	<n/a>
[Ignore]	[Ignore]
[Importance]	[Importance]
[Metadata]	<n/a>
[MultipleAsserts]	<n/a>
[Name]	<n/a>
[PairwiseJoin]	CombinationType = AllPairs property of [CombinatorialTest]
[Parameter]	<n/a>
[Pending]	<n/a>
[Repeat]	[Repeat] [RepeatTest]
[Rollback]	[RollBack] [RollBack2]
[Row]	[Row]
[SequentialJoin]	<n/a>
[SetUp]	[SetUp]
[TearDown]	[TearDown]
[Test]	[Test] [RowTest] [CombinatorialTest]
[TestFixture] (optional!)	[TestFixture]
[TestsOn]	[TestsOn]
[ThreadedRepeat]	[ThreadedRepeat]
[Timeout]	TimeOut property of [TestFixture]
[VerifyComparisonContract]	<n/a>
[VerifyEqualityContract]	<n/a>
[VerifyExceptionContract]	<n/a>
<n/a>	[CollectionIndexingFixture]
<n/a>	[CollectionOrderFixture]
<n/a>	[CompositeFixture]
<n/a>	[Copy]
<n/a>	[CopyToProvider]
<n/a>	[CurrentFixture]
<n/a>	[DataFixture]
<n/a>	[DataProvider]
<n/a>	[Duration]
<n/a>	[EnumerationFixture]
<n/a>	[ExtractResource]
<n/a>	[ForEachTest]
<n/a>	[Grammar]
<n/a>	[GrammarFixture]
<n/a>	[IndexerProvider]
<n/a>	[Information]
<n/a>	[IntIndexerProvider]
<n/a>	[MultipleCulture]
<n/a>	[NamespaceProvider]
<n/a>	[NotInheritedExpectedException]
<n/a>	[NullableAttribute]
<n/a>	[Pelikhan]
<n/a>	[PerfCounter]
<n/a>	[PostIt]
Unncessary.	[ProcessTestFixture]
<n/a>	[Provider]
<n/a>	[ProviderFactory]
<n/a>	[Read]
<n/a>	[ResourceXmlDataProvider]
<n/a>	[RestoreDatabaseFirst]
<n/a>	[Seed]
<n/a>	[SqlRestoreInfo]
<n/a>	[SuiteProvider]
Order property of [Test] or [TestFixture]	[TestSequence]
<n/a>	[TestSuite]
<n/a>	[TestSuiteFixture]
<n/a>	[TestSuiteSetUp]
Unncessary.	[TypeFixture]
<n/a>	[UsingEnum]
Similar to [Factory]	[UsingFactories]
<n/a>	[UsingImplementations]
<n/a>	[UsingLinear]
Similar to [Column]	[UsingLiterals]
<n/a>	[Write]
<n/a>	[XmlDataProvider]

 

Edit: Fixed Vadim's name  :-)

Technorati Tags: gallio,mbunit