Announcing Gallio and MbUnit v3.0.6

Today we are releasing Gallio and MbUnit v3.0.6.  Lots of new stuff, including new features and performance enhancements.

Highlights:

• Parallelizable tests.
• One-click test debugging from Icarus.
• Auto-reload and auto-run tests from Icarus after each build.
• ReSharper 4.5 beta support with improved performance.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Release notes: All release notes from all versions.

 

Remark (added 3/27): We strongly recommend upgrading to ReSharper v4.5 beta and TestDriven.Net v2.18+ before installing this release since there have been significant performance improvements.  Also it appears there may be a regression in the Gallio support for older ReSharper versions that can result in deadlock during test exploration.  (If for some reason you cannot upgrade at this time and you are experiencing this problem, you may try a recent nightly build instead.)

Acknowledgements

In this release I would like to especially acknowledge Graham Hay for his work enhancing Icarus with new features such as running tests with the debugger and also Yann Trévin for his work designing and building the MbUnit v3 contract verifiers.  Eddy Young also helped add support for running MSTest 2005 tests (previously we only supported MSTest 2008).

I would also like to thank all of the other members of the community who contributed valuable feedback, patches and support during this release cycle.

Thanks!

Edit: Fixed Eddy Young's name in the acknowledgements.  Thanks Eddy!

MbUnit

Parallelizable

With v3.0.6, MbUnit helps you get the most out of your multi-core CPU.  Mark any test [Parallelizable] and it will be permitted to run in parallel with other parallelizable tests in the same fixture.

Fixtures can also be marked parallelizable to enable them to be run in parallel with other parallelizable fixtures.

Please note that if you want all tests within a fixture to be considered parallelizable then you still need to add [Parallelizable] to each of them.  (We might add a feature to set this at the fixture or assembly level later based on user feedback.)

Also note that just because a test or fixture is marked parallelizable does not mean it will run in parallel with other tests in particular.  For the sake of efficiency, we limit the number of active tests threads based on the configured degree of parallelism.  If you want a specific number of instances of a test to run in parallel with each other, consider using [ThreadedRepeat].

DegreeOfParallelism

The degree of parallelism setting controls the maximum number of tests that MbUnit will attempt to run in parallel with one another.  By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.

If you don't like the default then you can override the degree of parallelism at the assembly-level like this:

DefaultTestCaseTimeout (Breaking Change!)

In MbUnit v2 and earlier versions of MbUnit v3, there was a non-configurable default timeout of 10 minutes per test fixture.  Now we have a configurable default timeout of 10 minutes per test case.

Old behaviour:

• 10 minute timeout per fixture
• non-configurable
• override by setting [Timeout] attribute on the fixture

New behaviour:

• 10 minute timeout per test case
• configurable with [DefaultTestCaseTimeout] on the assembly
• override by setting [Timeout] attribute on the test

We have a default timeout in place because it's useful for the sake of robustness.  Otherwise tests could just run forever if left unattended.  This is different from NUnit which has no timeout by default (even if you use Gallio to run your NUnit tests).

If you absolutely hate timeouts... here's what you need to do:

Contact Verifiers Redux (Breaking Change!)

Contract verifiers simplify the process of testing that an implementation of a common contract is correct.  For example, they can help you test your custom Exception class, verify that your datatype's Equals & GetHashCode are consistent, and check IList<T> invariants.

We originally introduced this feature in MbUnit v3.0.4, updated it in v3.0.5 and Yann Trévin has been busy making it better for v3.0.6!

Instead of using custom attributes to use a contract verifier, we now use a read-only field in the test fixture.  This change makes the syntax much more expressive, but you will have to update any tests written in the old style.

More documentation here: Contract Verifiers Chapter in Gallio Book (url subject to change)

In this example, we are testing whether the Parity class implements the Equatable contract correctly.  The contract verifier ensures that all members of the equivalence classes are considered equal to each other (and have the same hashcode); that all members of different equivalence classes are considered unequal; ensures that null values are handled correctly and that both Equals(object) and Equals(Parity) behave the same way.  It could also check operator overloads, if we implemented them.

The subject under test...

The test...

Xml Data Source

The CSV data source has been very popular.  Did you know you can associate metadata with CSV data rows, like expected exceptions?  Just add a column with a header like "[ExpectedException]", "[Description]", or some other metadata.

In v3.0.6, we now have an XML data source too.

Some data, including a little custom metadata...

The test...

The output in the report, notice the metadata...

Next up, does anyone want a SQL data source?

Serialization Asserts

This release includes a couple of new Assertions to help with testing serialization.

.Net serialization protocol asserts:

• Assert.IsSerializableType: Asserts that a type supports .Net serialization by way of the [Serializable] attribute.  Also ensures that type has a deserialization constructor if it also implements ISerializable.
• Assert.Serialize: Serializes a value using a .Net serialization formatter.  Asserts that serialization succeeded and returns resulting serialized value as a stream.
• Assert.Deserialize: Deserializes a value using a .Net serialization formatter.  Asserts that deserialization succeeded and returns the resulting deserialized value.
• Assert.SerializeThenDeserialize: Serializes then deserializes a value using a .Net serialization formatter.  Asserts that both serialization and deserialization succeeded and returns the resulting deserialized value after the round-trip.
• Assert.BinarySerialize: Shortcut for Assert.Serialize using a BinaryFormatter.
• Assert.BinaryDeserialize: Shortcut for Assert.Deserialize using a BinaryFormatter.
• Assert.BinarySerializeThenDeserialize: Shortcut for Assert.SerializeThenDeserialize using a BinaryFormatter.

Xml serialization protocol asserts:

• Assert.IsXmlSerializableType: Asserts that an XmlSerializer can be constructed for a given type without error.
• Assert.XmlSerialize: Serializes a value using an XmlSerializer.  Asserts that serialization succeeded and returns the resulting Xml as a string.
• Assert.XmlDeserialize: Deserializes a value using an XmlSerializer.  Asserts that deserialization succeeded and returns the resulting deserialized value.
• Assert.XmlSerializeThenDeserialize: Serializes then deserialized a value using an XmlSerializer.  Asserts that both serialization and deserialization succeeded and returns the resulting deserialized value after the round-trip.

Typical usage...

Terminating Tests with Specific Outcomes

MbUnit v3 includes several methods for controlling the outcome of a test, including a couple of new ones introduced in v3.0.6.

• Assert.Fail: Raises an assertion failure by throwing an AssertionFailedException, resulting in a failed outcome.  If executed within an Assert.Multiple block or in a test with [MultipleAsserts] then the failure is logged but the exception is deferred until the end of the block or of the test.
• Assert.Inconclusive: Terminates a test by throwing a TestInconclusiveException, resulting in an inconclusive outcome.
• [new!]  Assert.Terminate: Terminates a test by throwing a TestTerminatedException, resulting in a user-specified outcome.
• [new!]  Assert.TerminateSilently: Terminates a test by throwing a SilentTestException, resulting in a user-specified outcome.  Unlike Assert.Terminate, does not log a stack trace at the point of termination.
• [new!]  Assert.IsFailurePending: Indicates whether an assertion failure has been raised but the AssertionFailedException has not yet been thrown.  This is used in conjunction with Assert.Multiple to determine whether any failures have already been observed within the block.

Reminder:  There are two mechanisms in MbUnit to enable a test with multiple assertions to keep running even if some of them fail.

• Assert.Multiple: Runs a block of code that may contain multiple assertions.  If any assertions fail then the test will be terminated  with a failed outcome when the block finishes.
• [MultipleAsserts]: You may add the MultipleAsserts attribute to a test to effectively run the whole test within an Assert.Multiple block.

Diagnostic Log

A few people have asked how to print progress messages from tests.  They try TestLog, Debug.WriteLine and Console.WriteLine, and sure enough, nothing gets printed to the console, by design.  We capture all of this information and write it to the test report.

Now you can use DiagnosticLog.WriteLine to  write progress messages to the TestDriven.Net output window, the Icarus Runtime Log view or the console when running Echo, NAnt, MSBuild or PowerShell.

Icarus

One-Click Debugging

Icarus can now run tests under the debugger with just one click of the "Debug" button.  It will attach (or launch) the Visual Studio debugger to the test process and start the tests.

There is no longer any need to change the test runner type or to manually attach to the Icarus process.

Auto-Reload and Auto-Run

Icarus now supports automatically reloading and running all loaded tests whenever they are recompiled.

In v3.0.6, these features are controlled by a Preference.  In future releases we will probably make them available from the toolbar so that it is easier to switch modes on demand.

Additional Test Runner Enhancements

Passing Additional Arguments to NCover

Gallio includes support for launching tests with NCover already attached.  By default, the coverage report is written out to "Coverage.xml" in the current directory of the test run.

In v3.0.6 we have added the ability to change the location of the coverage file and to pass additional arguments to NCover.  Typically you will want to pass additional arguments to inform NCover to exclude certain assemblies from code coverage.

Here is an example using Gallio.Echo to run the tests from the command-line.  If you are using NAnt, MSBuild or Powershell then the process is similar.  Refer to the Documentation for details of how to pass the NCoverCoverageFile and NCoverArguments "runner properties" in each case.

Gallio.Echo.exe Widget.Tests.dll /runner:NCover /runner-property:NCoverCoverageFile='C:\Temp\WidgetCoverage.xml' /runner-property:NCoverArguments='//eas .*.Tests;Gallio;MbUnit;OtherIgnoredAssembly'

Echo Support for Test Projects

Echo now supports running test projects such as are created by Icarus.  Just pass it in on the command-line.

Gallio.Echo.exe WidgetTests.gallio

Launch Tests With Debugger Attached

It is now possible to launch tests with the debugger attached from Echo, NAnt, MSBuild or PowerShell by adding the "debug" argument.

Here is an example for Echo.  The other runners are similar.  Refer to the Documentation for details of how to pass the Debug option in each case.

Gallio.Echo.exe Widget.Tests.dll /debug

Total Run-Time Limit

To limit the total amount of  time that the tests are allowed to run, specify the total run-time limit.  This is a good idea when incorporating tests into a Continuous Integration build to ensure that runaway tests don't get to run for too long.

Here is an example for Echo.  The other runners are similar.  Refer to the Documentation for details of how to pass the RunTimeLimit option in each case.  The time span is specified in seconds.

Gallio.Echo.exe Widget.Tests.dll /run-time-limit:600

External Tools Compatibility

We've updated integration with a variety of external tools.

• CCNet v1.0.4
• CSUnit v2.0.5
• NUnit v2.4.8
• Pex v0.9 Academic
• TypeMock v5.1
• TestDriven.Net v2.19
• NCover v1.5.8, v2, v3.
• ReSharper v3.1, v4.0, v4.1 , v4.5 Beta
• TeamCity v4.0
• xUnit.Net v1.0.3
• Visual Studio 2005, 2008 (for VS 2010 support, use Gallio v3.0.5 for now) 

Change Summary

Gallio Core

• Redesigned test lifecycle to significantly improve performance when running tests in multiple test assemblies as a batch.
• Added a debugging API, use to provide new debugging features in test runners.
• Fixed an issue with significant whitespace being unintentionally dropped from the report due to XML whitespace handling.
• Added an XSD file for Gallio reports.
• Dropped dependency on ISymWrapper.  There should be no more Exception Assistant dialogs about COMExceptions popping up in Visual Studio while launching the tests.
• Removed a stray UTF8 Byte Order Mark that appeared in HTML formatted reports.
• Gallio registers its TestDriven.Net extensions even if TestDriven.Net is not yet installed.
• Added support for passing custom properties to test hosts and report formatters.
• Improved diagnostics when running tests out of process.
• Various API changes and refactoring.
• Added support for jumping to the source code of a test from the report.
• Added support for capturing diagnostic log messages during test execution.
• Improved the Navigator feature (jump to source from a report) to automatically launch Visual Studio if needed.

MbUnit v3

• Added [Parallelizable], [DegreeOfParallelism], [DefaultTestCaseTimeout] and [XmlData].
• Added assertions and helpers: Terminate, TerminateSilently, IsFailurePending, IsBinarySerializable, IsXmlSerializable.
• Added DiagnosticLog class for writing to the log.
• Fixed [Factory] data sources defined on generic types.
• Fixed dynamic test outcome reporting.
• Fixed project templates for VB.Net.
• Fixed Assert.EndsWith failure message.
• Added generic variations of Assert.Is[Not]InstanceOfType and Assert.Is[Not]AssignableFrom.
• Fixed a faulty assertion in the Exception contract verifier regarding the setting of the Exception's Message property.
• Fixed an issue enumerating tests in nested types that inherit from one of their own containing types.
• Modified [Rollback] to enter a COM+ transaction context like MbUnit v2's original rollback attribute.  This should fix compatibility issues others have noted.
• Fixed inheritance of [Row] attribute (and others) on test methods in derived classes.
• Added TestStep.RunStepAndVerifyOutcome.
• Fixed an issue with [MultipleAssert] affecting nested test steps.
• Renamed MetadataKeys.CategoryName to MetadataKeys.Category.
• Modified [StaticTestFactory] to run tests in the exact order specified.
• Now tolerates missing metadata columns in CSV files.
• Applied neoeinstein's patch providing IComparer<T> based overloads for relational asserts such as Assert.IsLessThan.
• Added DiagnosticLog class to enable users to write messages to the console during test execution.
• Eliminated a deep recursion issue in BaseTest.IsLocalIdUniqueAmongSiblings that caused a StackOverflowException.
• Changed timeout policy to apply per-test case not per fixture.

MbUnit v2 Adapter

• Fixed MbUnit v2 adapter top-level assembly outcome reporting.
• Ported old MbUnit v2 XmlAsserts over to MbUnit.Compatibility.dll.
• Fixed a problem where the CCNet integration documentation was not being installed.
• Fixed a problem displaying attachments in CCNet.

MSTest Adapter

• Auto-detect the version of the MSTest framework referenced by tests.  MSTest 2005 not currently supported.  (volunteer needed)

Icarus

• Added "Debug" command to run tests with the debugger!
• Fixed Icarus filter inconclusive.
• Improved Icarus view source from the test tree.
• Check to ensure assemblies exist before reloading.
• Icarus now remembers its position and size between executions.

Echo

• Added support for running Icarus projects.
• Added /debug switch to run tests with the debugger.
• Added /runtimelimit switch to set a time limit on test execution.
• Added /runner-property option to pass custom options to the test runner.
• Added /report-formatter-property option to pass custom options to the report formatter.

NAnt

• Added debug attribute to run tests with the debugger.
• Added run-time-limit attribute to set a time limit on test execution.
• Added runner-property element to pass custom options to the test runner.
• Added report-formatter-property element to pass custom options to the report formatter.

MSBuild

• Added Debug attribute to run tests with the debugger.
• Added RunTimeLimit attribute to set a time limit on test execution.
• Added RunnerProperties element to pass custom options to the test runner.
• Added ReportFormatterProperties element to pass custom options to the report formatter.

PowerShell

• Added -DebugTests switch to run tests with the debugger.
• Added -RunTimeLimit switch to set a time limit on test execution.
• Added -RunnerProperties option to pass custom options to the test runner.
• Added -ReportFormatterProperties option to pass custom options to the report formatter.

ReSharper

• Added support for ReSharper v4.5 beta.
• Added memoization to the reflection API to improve performance.

Visual Studio Team Test

• Now runs tests in a background thread to provide better feedback during execution.
• Added memoization to the reflection API to improve performance.
• The MSTest adapter now support MSTest 2005 in addition to 2008.

NCover

• Added support for NCover v3.
• Added "NCoverCoverageFile" and "NCoverArguments" properties to tweak the command-line.
• Fixed several issues that caused long-running coverage runs to fail.

TypeMock

• Updated to TypeMock v5.1 integration assembly.

Pex

• Upgraded Pex support for v0.9 Academic.

AutoCAD

• Read Gallio's process id more robustly.

Technorati Tags: gallio,mbunit