Thursday, October 16, 2008

Announcing Gallio and MbUnit v3.0.4

Today we are releasing v3.0.4 of Gallio and MbUnit.  This release incorporates many new features as well improvements for robustness.

New asserts, assertion diffs, 0rdered tests, Rollback, Repeat, ThreadedRepeat, R# 4.1, VSTS, CSUnit, TeamCity, Gallio.Ambience, clickable report links (IE/Firefox integration), and more...

Try it out!  Please be aware that there are now separate x86 and x64 installers.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

MbUnit

Assertions Refresh

We have consolidated and significantly improved the set of built-in MbUnit assertions.  Refer to the migration guide at the end of this post for more information.  Vadim Kreynin was a huge help with this effort.

Assertion Failures

All new assertions are implemented with a common pattern designed to make assertion failures easier to understand.  Notice how the parameters of the assertion are clearly labeled.  Also, the fact that the values are quoted tells you that the objects involved are actually strings.

Yes... and you can click on the stack trace in the report to go to source.

image

And putting to rest that age-old question "are these two values the same or do they just look that way"?

image

Assertions with Diffs

When comparing two values for equality, we now display detailed character by character differences between their formatted representations.  These differences appear as colored and styled regions in the test report.

In the future, we will be providing a new Visual Studio integrated test runner that will be able to manifest differences as styled output in the test console as well.

image

Assertions Over Collections

MbUnit assertions can be composed to form more complex assertions out of simpler ones.  The Assert.Over syntax provides some useful compositions for performing assertions over collections.

image

image

AssertEx.That...

For C# 3.0 / .Net 3.5 users, MbUnit now offers some new syntax for expressing assertions using lambda expressions using AssertEx.That.  If the expression evaluates to false, MbUnit will try to explain why by printing the values of referenced variables and intermediate sub-expressions that were computed during the assertion.

Suppose we have the following test:

image

When we run this test, it will fail because the last two values of the sequence should have been '13' and '21'.  The assertion failure pretty clearly spells this out for us by telling us all about the variables involved.

image 

Multiple Asserts

Sometimes a test performs multiple related assertions that are logically part of a single verification.  In these scenarios, we would like to be able to continue running the test even if one of the assertions fails.  When the test is finished, we can the report all of the failures together.

There are two ways to do this in MbUnit.

Tag the method with [MultipleAsserts]:

image

Or use the Assert.Multiple function and pass in a delegate:

image

In either case, MbUnit will produce a failure message like this:

image

Stack Trace Filtering

We now filter out framework code from stack traces.  You may have noticed that the assertion failure examples shown above were particularly uncluttered...

Contract Verifiers

Tired of writing tests for exceptions, equality operators, and comparers?  Yann Trévin has made it easier!

Believe it or not, this is a complete test fixture for a custom exception type.

image

And here's what it takes to test the Equals, GetHashCode, operator==, and operator!= methods of our Range class.  We just need to provide a few examples of ranges that are in different equivalence classes.

image

And this is a test for CompareTo, and all of the standard comparison operators.

image

We plan to write more of these soon...

Dependent / Ordered Tests

When writing integration tests, it is sometimes useful to chain several tests together that capture a logical sequence of operations.

MbUnit can capture this chaining either with dependencies:

image

Or with explicit ordering:

image

Icarus

Graham Hay has been hard at work on Icarus.

image 

ReSharper Test Runner

ReSharper v4.1

We now fully support ReSharper v4.1.  (Actually, we have for quite some time now, this is just the first official release with support for it.)

Clickable Stack Traces

One popular request has been to provide clickable stack traces in ReSharper.  We are somewhat limited in our ability to do this well (due to ReSharper API limitations) but we tried our best!

image

Reports

It is now possible to view Condensed and Full test reports in ReSharper.  The test report may provide more details under certain situations.  For example, it will include those assertion diff highlights (which we are unable to display in the ReSharper test result panel itself).

image

Visual Studio Team System Test Runner

Yann Trévin also put a lot of work into the Visual Studio Team System test runner.

You can run tests using the Test View.

image

View results in the Test Results view.

image

And if you double click on the test result, you can view the test report for more information.

image

Notes:

  • You will need Visual Studio 2008 SP1 for this feature.
  • The tests must be located in a project that has the appropriate ProjectTypeGuids specified.

Installer

New Installer

Francois Retief rewrote our old installer using WiX.  It is so much better now!

Highlights:

  • Works on Vista.
  • Supports 640bit architectures.
  • We now provide separate x86 and x64 versions.
  • More configuration options.

CSUnit Adapter

Francois Retief also implemented an adapter for CSUnit.  So now you can use Gallio to run your CSUnit tests.

Please let us know what you think.

TeamCity Integration

Gallio now knows how to emit TeamCity service messages to notice the build server about test progress and test results.

To use it, set the RunnerExtensions argument of the Gallio test runner you are using to "TeamCityExtension,Gallio.TeamCityIntegration".

Examples:

If you are using the Gallio MSBuild task:
    <Gallio RunnerExtensions="TeamCityExtension,Gallio.TeamCityIntegration"
        ... other arguments... />

If you are using the Gallio NAnt task:
    <gallio runner-extension="TeamCityExtension,Gallio.TeamCityIntegration"
        ... other arguments... />

If you are using the Gallio Echo task:
    Gallio.Echo /e:TeamCityExtension,Gallio.TeamCityIntegration ... other arguments...

Note: We plan to automatically detect TeamCity in future releases.

Gallio.Ambience

This release includes an initial preview of Gallio.Ambience.  Ambience manages what we are tentatively calling the Ambient object store.

The Ambient object store is a shared lightweight repository for intermediate test data.  It is like a persistent whiteboard that is used to pass information from one test to another or to store it for subsequent analysis.  In effect, you can use it to create stateful tests.

A stateful test is one which may load or store data generated by previous test runs and use it for some purpose.  This practice is strongly discouraged in unit testing best practice because it introduces a dependency on previous runs.  However, it can be useful for integration testing of stateful systems that cannot easily be torn down and built up again.

Testers will commonly resort to flat files for storing test data between runs.  This system works well on a local machine but it is problematic for distributed testing clusters unless the files are stored on a network drive (where they risk corruption).

Instead of using flat files, Gallio Ambience is client/server based.  The server in this case is a Db4o Object Database instance.  Clients may connect to it remotely to store and retrieve data.  LINQ query syntax is supported.

Example

First we write a test that stores some data in the Ambient object store for later.

image

image

Then we write another test which consumes the data in the Ambient object store.

image

In this way we can test stateful processes using one or more tests that communicate via the Ambient object store.

Note:

  • If you are able to refactor your system to avoid the need for stateful tests then we recommend doing so.  Stateful tests are inherently more brittle than properly isolated unit tests.  That said, we think the can be useful for solving certain otherwise messy problems.
  • This feature is new and experimental.  We welcome your feedback!

Documentation

API Documentation

We have published the Gallio and MbUnit v3 API documentation on the website here.

Gallio Book

The Gallio Book is very much a work in progress but we have a fairly complete outline and some content produced so far.  It is also published up on the website here.

External Tools Compatibility

We've updated integration with a variety of external tools.

  • CCNet v1.0.4
  • CSUnit v2.0.5
  • NUnit v2.4.8 (now supports NUnit add-ins correctly)
  • Pex v0.7
  • ReSharper v3.1, v4.0, v4.1
  • TeamCity v3.1, v4.0 EAP
  • xUnit.Net v1.0.3

Misc. Improvements

  • Improved backwards compatibility with older versions.
  • More careful handling of the runtime environment to avoid certain exotic assembly version conflict issues in TestDriven.Net, ReSharper and Visual Studio.
  • Performance improvements in upcoming versions of TestDriven.Net.
  • Lots of bug fixes.

Upcoming

A few features did not quite make it into v3.0.4 but will appear in upcoming releases:

  • New plug-in model (will significantly improve startup performance).
  • TeamCity auto-detection.
  • NCover 2 support.  It's already in there but there remain some unresolved issues to address.
  • A new Visual Studio integrated test runner.
  • Hosting Gallio inside exotic test environments such as ASP.Net and AutoCAD.
  • Launching x86 testing processes on x64.

Mono

Gallio almost runs on Mono.   We have made cross-platform support a priority for Gallio and MbUnit but there is still some work remaining.  The build server now regularly runs tests using Mono.

We are seeking volunteers to help make full Mono support a reality in upcoming releases.

Migration Guide

If you are migrating code from MbUnit v2 or from previous versions of MbUnit v3, the following tables will help you.

What To Do If XYZ Is Not In This List

If one or more of your favorite methods or attributes from MbUnit v2 is not in this list here's what to do:

  1. Consult the tables for a replacement.
  2. Old assertions are available in the MbUnit.Compatibility.dll.  They have been renamed with the prefix Old* to clearly distinguish them.
  3. If you still cannot find what you need, all is not lost.  Remember that Gallio can run tests from MbUnit v2 and v3 at the same time.  Several old versioning issues have been resolved so this process should be smoother too.
  4. We are planning to replace / upgrade some (but not all) of the deprecated methods and attributes.  If there are any in particular you think we should tackle first, please open an issue in our issue tracker.
  5. Please feel free to make other suggestions for improvement.

Assertion Migration Guide

This table describes changes to assertions:

MbUnit v3 (NEW) MbUnit v2 (OLD)
Assert.AreApproximatelyEqual (v3.0.5) Assert.AreEqual (with delta)
Assert.AreElementsEqual CollectionAssert.AreElementsEqual
Assert.AreElementsNotEqual <n/a>
Assert.AreEqual Assert.AreEqual
ArrayAssert.AreEqual
CollectionAssert.AreEqual
CollectionAssert.AreEquivalent
GenericAssert.AreEqual
StringAssert.AreEqualIgnoreCase
Assert.AreElementsEqualIgnoringOrder <n/a>
Assert.AreNotEqual Assert.AreNotEqual
CollectionAssert.AreNotEqual
CollectionAssert.AreNotEquivalent
Assert.AreNotApproximatelyEqual (v3.0.5) Assert.AreNotEqual (with delta)
Assert.AreNotSame Assert.AreNotSame
Assert.AreSame Assert.AreSame
<none> Assert.AreValueEqual
Assert.Between Assert.Between
Assert.Contains Assert.Contains
Assert.In
CollectionAssert.Contains
StringAssert.Contains
Assert.ContainsKey Assert.In
Assert.DoesNotContain Assert.NotIn
CollectionAssert.DoesNotContain
StringAssert.DoesNotContain
Assert.DoesNotContainKey Assert.NotIn
Assert.DoesNotThrow <n/a>
Assert.EndsWith StringAssert.EndsWith
Assert.Fail Assert.Fail
Assert.FullMatch StringAssert.FullMatch
Assert.GreaterThan Assert.Greater
Assert.GreaterThan
Assert.GreaterThanOrEqualTo Assert.GreaterEqualThan
Assert.Inconclusive <n/a>
Assert.IsAssignableFrom Assert.IsAssignableFrom
ReflectionAssert.IsAssignableFrom
Assert.IsEmpty Assert.IsEmpty
GenericAssert.IsEmpty
StringAssert.IsEmpty
Assert.IsFalse Assert.IsFalse
Assert.IsInstanceOfType Assert.IsInstanceOfType
ReflectionAssert.IsInstanceOf
<none> Assert.IsNaN
Assert.IsNotAssignableFrom Assert.IsNotAssignableFrom
Assert.IsNotEmpty Assert.IsNotEmpty
GenericAssert.IsNotEmpty
StringAssert.IsNonEmpty
Assert.IsNotInstanceOfType Assert.IsNotInstanceOfType
Assert.IsNotNull Assert.IsNotNull
Assert.IsNull Assert.IsNull
Assert.IsTrue Assert.IsTrue
Assert.LessThan Assert.Less
Assert.LowerThan
Assert.LessThanOrEqualTo Assert.LowerEqualThan
Assert.Like StringAssert.Like
Assert.Multiple <n/a>
Assert.NotBetween Assert.NotBetween
Assert.NotLike StringAssert.NotLike
Assert.StartsWith StringAssert.StartsWith
Assert.Throws <n/a>
TestLog.Warning.WriteLine Assert.Warning
<none> CollectionAssert.AllItemsAreInstancesOfType
<none> CollectionAssert.AllItemsAreNotNull
<none> CollectionAssert.AllItemsAreUnique
<none> CollectionAssert.AreCountEqual
<none> CollectionAssert.AreIsSynchronizedEqual
<none> CollectionAssert.AreSyncRootEqual
<none> CollectionAssert.IsCountCorrect
<none> CollectionAssert.IsNotSubsetOf
<none> CollectionAssert.IsNotSynchronized
<none> CollectionAssert.IsSubsetOf
<none> CollectionAssert.IsSynchronized
<none> FileAssert.AreEqual
<none> FileAssert.AreStreamContentEqual
<none> ReflectionAssert.HasConstructor
<none> ReflectionAssert.HasDefaultConstructor
<none> ReflectionAssert.HasField
<none> ReflectionAssert.HasMethod
<none> ReflectionAssert.IsSealed
<none> ReflectionAssert.NotCreatable
<none> ReflectionAssert.ReadOnlyProperty
<none> ReflectionAssert.WriteOnlyProperty
<none> CompilerAssert.*
<none> ControlAssert.*
<none> DataAssert.*
<none> PerfAssert.*
<none> SecurityAssert.*
<none> SerialAssert.*
<none> WebAssert.*
<none> XmlAssert.*

 

Attribute Migration Guide

This table describes changes to attributes:

MbUnit v3 (NEW) MbUnit v2 (OLD)
[Annotation] <n/a>
[ApartmentState] Apartment property of [TestFixture]
[AssemblyFixture] [AssemblyCleanup]
[AssemblyResolver] [AssemblyResolver]
[Author] [Author]
[Bind] <n/a>
[Category] [TestCategory]
[FixtureCategory]
[Column] <n/a>
[CombinatorialJoin] (optional) CombinationType = Cartesion property of [CombinatorialTest]
[CsvData] <n/a>
[DependsOnAssembly] [AssemblyDependsOn]
[DependsOn] [DependsOn]
[Description] Description property of [TestFixture] and [Test]
[ExpectedArgumentException] [ExpectedArgumentException]
[ExpectedArgumentNullException] [ExpectedArgumentNullException]
[ExpectedArgumentOutOfRangeException] <n/a>
[ExpectedException] [ExpectedException]
[Explicit] <n/a>
[Factory] [Factory] (note: very different semantics)
[FixtureInitializer] <n/a>
[FixtureSetUp] [TestFixtureSetUp]
[FixtureTearDown] [TestFixtureTearDown]
[Header] <n/a>
[Ignore] [Ignore]
[Importance] [Importance]
[Metadata] <n/a>
[MultipleAsserts] <n/a>
[Name] <n/a>
[PairwiseJoin] CombinationType = AllPairs property of [CombinatorialTest]
[Parameter] <n/a>
[Pending] <n/a>
[Repeat] [Repeat]
[RepeatTest]
[Rollback] [RollBack]
[RollBack2]
[Row] [Row]
[SequentialJoin] <n/a>
[SetUp] [SetUp]
[TearDown] [TearDown]
[Test] [Test]
[RowTest]
[CombinatorialTest]
[TestFixture] (optional!) [TestFixture]
[TestsOn] [TestsOn]
[ThreadedRepeat] [ThreadedRepeat]
[Timeout] TimeOut property of [TestFixture]
[VerifyComparisonContract] <n/a>
[VerifyEqualityContract] <n/a>
[VerifyExceptionContract] <n/a>
<n/a> [CollectionIndexingFixture]
<n/a> [CollectionOrderFixture]
<n/a> [CompositeFixture]
<n/a> [Copy]
<n/a> [CopyToProvider]
<n/a> [CurrentFixture]
<n/a> [DataFixture]
<n/a> [DataProvider]
<n/a> [Duration]
<n/a> [EnumerationFixture]
<n/a> [ExtractResource]
<n/a> [ForEachTest]
<n/a> [Grammar]
<n/a> [GrammarFixture]
<n/a> [IndexerProvider]
<n/a> [Information]
<n/a> [IntIndexerProvider]
<n/a> [MultipleCulture]
<n/a> [NamespaceProvider]
<n/a> [NotInheritedExpectedException]
<n/a> [NullableAttribute]
<n/a> [Pelikhan]
<n/a> [PerfCounter]
<n/a> [PostIt]
Unncessary. [ProcessTestFixture]
<n/a> [Provider]
<n/a> [ProviderFactory]
<n/a> [Read]
<n/a> [ResourceXmlDataProvider]
<n/a> [RestoreDatabaseFirst]
<n/a> [Seed]
<n/a> [SqlRestoreInfo]
<n/a> [SuiteProvider]
Order property of [Test] or [TestFixture] [TestSequence]
<n/a> [TestSuite]
<n/a> [TestSuiteFixture]
<n/a> [TestSuiteSetUp]
Unncessary. [TypeFixture]
<n/a> [UsingEnum]
Similar to [Factory] [UsingFactories]
<n/a> [UsingImplementations]
<n/a> [UsingLinear]
Similar to [Column] [UsingLiterals]
<n/a> [Write]
<n/a> [XmlDataProvider]

 

Edit: Fixed Vadim's name  :-)

Technorati Tags: ,

24 comments:

Libor said...

very impressive overview, keep it up
but one thing guys, please, post more often so we know what's
going on behind the scene. don't keep us in the dark :)

Jeff Brown said...

I'll certainly try to post more often. And not just about releases... :-)

GraemeF said...

Seriously good stuff. Well done guys :)

Borek said...

Great news. So this release is considered beta?

bydlo said...

Looks great, I'm excited to try this.

Jeff Brown said...

@Borek,
The whole alpha / beta thing got too confusing in the end due to scope creep so I dropped it.

In any case, this release is pretty much feature-complete.

Peli said...

Very nice!

Devtron said...

What is the difference between: Assert.IsEmpty and Assert.IsNotEmpty.

They both have the same definition in their descriptions. Is this a documentation bug, or are they duplicate functions? Hmmmmmmm...

Jeff Brown said...

@devtron:
Are you sure? I just checked and the descriptions are different.

Assert.IsEmpty: "Verifies that an actual value contains no elements."

Assert.IsNotEmpty: "Verifies that an actual value contains at least one element."

They are basically just inverses of one another.

Steve Bohlen said...

This is tremendous work -- I can't wait to get some time to dig deep into most of this.

One Q tho about something I don't see mentioned in the feature list: we use CI as a core foundation of a lot of our practice and so NANT integration is something that we really need. Its really more about MbUnit features rather than Gallio (the runner) but is there a NANT task out-of-the-box for MbUnit 3.x all ready to go or no?

Yann Trevin said...

@libor,
You might consider reading some of the other contributors' blogs. Although Jeff commits 95% of the job ;), they sometimes write about things occurring behind the scene, such as new features and work in progress. Look at the links in the Gallio Credit page.

Jeff Brown said...

@Steve Bohlen,
Out of the box there is a NAnt task, an MSBuild task, a PowerShell cmdlet, and a command-line runner any of which can be used to integrate with a CI build process.

Pick your poison. ;-)

vkreynin said...

Jeff, congratulation on releasing v3.0.4 and thanks for the credit. You almost spelled my first name correctly.:)

Devtron said...

Hello,

I re-checked the online documentation, which is all I have to go with. One would assume that they are "basically just inverse of one another" but we all know what assumptions do.

In that respect, can you double check if I am crazy or not?

http://docs.mbunit.com/help/html/M_MbUnit_Framework_Assert_IsEmpty_2_9496d360.htm

Assert..::IsEmpty Method (ICollection, String):
Assert that an array, list or other collection is empty


http://docs.mbunit.com/help/html/M_MbUnit_Framework_Assert_IsNotEmpty_3_8d3d3ccf.htm

Assert..::IsNotEmpty Method (ICollection, String, array xObject>[]()[])
Assert that an array, list or other collection is empty



If you look at the links above, they both have the same exact description, which is: "Assert that an array, list or other collection is empty ".

I was having some problems with some DB Unit Tests someone here wrote, and while debugging the problem ran across this issue with the documentation.

It is confusing for me that something would even be named "IsNotEmpty". One would *assume* IsEmpty would handle the scenario, and that IsNotEmpty is redundant. I may very well be wrong but we could not have a final say over the matter.

I think I ended up using something other than Assert for my solution (I cant even remember), but I had this debate with a few guys I work with here in Australia and I just wanted to point it out to someone on the MbUnit development team.

Am I crazy or do they have the same definitions? Is one wrong, or is there redundant functionality in the MbUnit framework?

Jeff Brown said...

@DevTron,
You're looking at the old MbUnit v2.4 documentation which apparently contains some inconsistencies.

Check out the v3.0.4 documentation instead:

http://www.gallio.org/Docs.aspx.

Anonymous said...

How do I get templates in Visual Studio for MbUnit?

Jeff Brown said...

Templates are installed by default. You should find templates like "MbUnit v3 Fixture" in the C# and VB sections of the Add New Item dialog.

Mikalai Kardash said...

@Jeff.
Greate release! I am more than satisfied with what is done in Gallio so far.

On the way of moving from MbUnit2.4 to MbUnit3 I found a road block that took me about 30 minutes to workaround. There is an issue/problem with integration of Gallio and VSTS2008, basically it is required to "turn a project into test project". It is described by Michael Herndon here.

Can we expect some improvements in this area in the near future? It would be great if Gallio could find MbUnit test fixtures without having to edit project files.

Dave said...

You might mention that .NET 3.5 is required to run the MSBuild tasks --
error MSB4018: The "Gallio" task failed unexpectedly.
error MSB4018: System.MissingMethodException: Method not found: 'Void Microsoft.Build.Utilities.TaskLoggingHelper.LogErrorFromException(System.Exception, Boolean, Boolean, System.String)'.
error MSB4018: at Gallio.MSBuildTasks.Gallio.Execute()
error MSB4018: at Microsoft.Build.BuildEngine.TaskEngine.ExecuteTask(ExecutionMode howToExecuteTask, Hashtable projectItemsAvailableToTask, BuildPropertyGroup projectPropertiesAvailableToTask, Boolean& taskClassWasFound)

Michael Stum said...

"Supports 640bit architectures."

Wow, THAT'S what I call Future-Proof :P

Philip Japikse, MCSD, MCDBA, CSM said...

When I run this test in R# via Gallio:
[Test]
public void Should_Fail_This_Test()
{
Assert.Fail();
}

it passes. when I run it in the gallio test runner, it fails as expected. Are there still issues with R#?

Jeff Brown said...

@Dave,
I find that MSBuild error very puzzling since TaskLoggingHelper in MSBuild v2 does indeed have that method.

In fact, we deliberately build against MSBuild v2 libraries.

That said, I haven't checked whether the method was actually added in v2.0 SP1. Could be.

Jeff Brown said...

@Philip,
That test should definitely fail, even in R#.

Since v3.0.4 was released, we have made a number of improvements to the robustness of the R# runner.

Please try one of the recent nightly builds from http://ccnet.gallio.org/Distributables or wait for v3.0.5 to be released within the next few days.

Philip Japikse, MCSD, MCDBA, CSM said...

The most recent daily build (3.0.4.523) resolved the R# issue. Nicely done!