Thursday, September 17, 2009

Gallio Next Steps

With Gallio v3.1 released, I have begun planning for what's ahead.

My first priority is to fix most of the issues that have been reported and to release a minor update to Gallio v3.1 in a week or two.

After that, the hard work begins.

Testing as a Service

As it stands, the Gallio platform mainly consists of one core library and a whole bunch of plug-ins.  The majority of the code is written in C# and runs on .Net Framework 2.0.

I want to go further.

The main goal of Gallio v3.2 will be to adapt the Gallio platform to operate as a collection of services that communicate via a message bus.  We will publish a specification of the messaging protocol so that other tools -- not necessarily written in .Net -- will be able to exchange messages with Gallio tools and participate in test execution, monitoring and reporting.

Why not pool our talents to develop great testing tools that work together across platform boundaries?

Gallio v3.2 and Beyond

Here's what you can expect from upcoming releases.

  1. Open messaging protocol.
  2. Test data warehouse.
  3. Web-based test reporting tool and test case manager.  (Archimedes)
  4. Test rigs for remote test execution.
  5. Data collection API for monitoring performance, gathering metrics and computing statistics.
  6. Improved IDE integration.
  7. Improved cross-platform support.
  8. More openness about the development process in general.
  9. Performance optimization, particularly of UI responsiveness and program start time.

How to Participate

I want to work with other members of the community to grow a practical lingua franca for test data interchange.  While at first we will be discussing how to design a suitable messaging protocol for Gallio, I hope that it will eventually evolve into an independent open standard that anyone can implement.

 

Please join us on the gallio-dev mailing list and tell us what you think!  :-)

Monday, September 14, 2009

The Gallio Plug-in Model

One of the biggest changes in Gallio v3.1 was is a new plug-in model.

The new plug-in model features comprehensive support for late-binding of components, provides a rich declarative metadata model, and provides enough information to support the installation and verification of plug-in assemblies and resources.

Concepts

Late-Binding and Declarative Metadata

Gallio includes many plug-ins out of the box and more can be installed later but not all of the plug-ins need to be loaded at all times.  To improve startup time, Gallio defers plug-in assembly loading until the last possible moment.  Consequently Gallio's plug-in model relies heavily on late-binding and declarative metadata that is external to the plug-in assemblies.

We don't use .Net custom attributes to provide most of this metadata because then we would have to load assemblies in order to read it out.  (Note: We could read metadata from custom attributes once and cache it like MEF does but it is not practical for some purposes such as plug-in installation.)

So the basic idea is to pack a bunch of useful information in XML.

Drop-In Plug-In Installation and Caching

Gallio strives to support XCopy-style deployment as much as possible.  This feature makes it possible to check-out a copy of Gallio from your source tree and just run it with no strings attached (except for special features that require installation like Visual Studio integration).

When designing the Gallio plug-in model it was important to ensure that Gallio plug-ins could be installed just by dropping files in the right place.  At run-time Gallio locates its plug-ins by searching plug-in directories recursively for .plugin files.  There is one major problem with this process: recursively scanning directories and loading lots of little files can be very time-consuming.  So the new plug-in model incorporates a cache.

Gallio maintains a per-user cache of installed plug-ins in any given list of plug-in directories.  This way when it starts up, it only needs to load the cache file and move on.

However there is a small problem with this approach.  By caching plug-in metadata, it is no longer sufficient to just drop in new plug-in files to install them.  The cache must be cleared somehow.

Gallio implements three strategies for refreshing its cache:

  1. When loading a cache file, it checks the timestamps on the original plug-in files to verify that they have not been changed.  This way Gallio can automatically detect an edit to a .plugin file and refresh the cache.  Unfortunately it cannot detect the presence of new .plugin files.
  2. A user can explicitly clear the plug-in cache using the  "Gallio.Utility.exe ClearUserPluginCache" command.  Unfortunately if there are multiple users of the machine then all users will need to have their plug-in cache cleared to detect the presence of a new plug-in.
  3. A admin user can explicitly reset the Gallio "installation id" of the machine by running "Gallio.Utility.exe ResetInstallationId".  This command effectively invalidates the caches of all users running on the machine at once.

It's not ideal but it works well for now.  Plug-in metadata caching has significantly improved Gallio start-up performance.

Plug-in Metadata

.plugin files

Each Gallio plug-ins has a ".plugin" file that describes the plug-in using XML.  The description includes the id of the plug-in, some traits, dependencies, associated files, referenced assemblies, services and components.

Here is an excerpt of Gallio.plugin.

<?xml version="1.0" encoding="utf-8" ?>
<plugin pluginId="Gallio" recommendedInstallationPath="" xmlns="http://www.gallio.org/">
  <traits>
    <name>Gallio</name>
    <version>0.0.0.0</version>
    <description>The heart of Gallio.</description>
    <icon>plugin://Gallio/Resources/Gallio.ico</icon>
  </traits>

  <dependencies>
    <dependency pluginId="BuiltIn" />
  </dependencies>

  <files>
    <file path="Gallio.plugin" />
    <file path="Gallio.dll" />

    <!-- SNIP -->

  </files>

  <assemblies>
    <assembly fullName="Gallio, Version=0.0.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e" codeBase="Gallio.dll" qualifyPartialName="true" />
  </assemblies>

  <services>
    <service serviceId="Gallio.TestFramework"
             serviceType="Gallio.Model.ITestFramework, Gallio" />

    <service serviceId="Gallio.TestFrameworkManager"
             serviceType="Gallio.Model.ITestFrameworkManager, Gallio" />

    <!-- SNIP -->

  </services>

  <components>
    <component componentId="Gallio.FallbackTestFramework"
               serviceId="Gallio.TestFramework"
               componentType="Gallio.Model.FallbackTestFramework, Gallio">
      <traits>
        <name>Fallback</name>
        <icon>plugin://Gallio/Resources/Gallio.ico</icon>
      </traits>
    </component>

    <component componentId="Gallio.TestFrameworkManager"
               serviceId="Gallio.TestFrameworkManager"
               componentType="Gallio.Model.DefaultTestFrameworkManager, Gallio">
      <parameters>
        <fallbackTestFrameworkHandle>${Gallio.FallbackTestFramework}</fallbackTestFrameworkHandle>
      </parameters>
    </component>

    <!-- SNIP -->

  </components>
</plugin>

Services and Components

Following the nomenclature used by several .Net Inversion of Control Containers, Gallio refers to its extension points as services and its extensions as components.  Services nominally represent interfaces.  Components provide implementations of those interfaces.

Internally Gallio refers to all service and component types by name using a TypeName object until it needs to instantiate it.  This is different from most .Net IoCs which immediately resolve most type names to runtime Type objects during initialization which causes most assemblies to be loaded up front.

Dependency Injection and Configuration

Gallio's plug-in model supports constructor and property dependency injection and configuration for components.

In the .plugin XML, the component parameters appear in a <parameters> element which contains an element for each constructor parameter or property that is to be bound.  If no value is specified for a given constructor parameter, then Gallio will attempt to resolve a service of the appropriate type to bind.  Otherwise if a value is specified, then Gallio will attempt to convert the provided string to an object of the required type using a DefaultObjectDependencyResolver.

The default object dependency resolver knows how to resolve all sorts of object types including strings, arrays, components, ComponentHandles, IComponentDescriptors, Versions, Guids, Icons, Images, Conditions, FileInfos, DirectoryInfos, Assemblies, AssemblySignatures, AssemblyNames, Types, TypeNames, and explicit references to other components (using the ${component.id} syntax seen in the excerpt above.)

Gallio resolves Icons, FileInfos, and other disk-based plug-in resources using a special Uri scheme called "plugin".   For example, a component might require a reference to a file that belongs to the plug-in so it declares a constructor parameter of type FileInfo.  In the XML, it provides the path of the file relative to the plug-in base directory by specifing a parameter with a Uri like: "plugin://MyPlugin/Path/To/File.txt".  When the component is instantiated, Gallio will resolve the Uri and provide a FileInfo with the appropriate full path to the file.

Plug-in and Component Traits

Traits are collections of typed properties that Gallio uses to describe plug-ins and components.

At runtime, plug-in traits are represented by instances of the PluginTraits class which looks a bit like this:

public class PluginTraits : Traits
{
    public PluginTraits(string name) { ... }
    public string Name { get; }
    public string Version { get; set; }
    public Icon Icon { get; set; }
    // SNIP...
}

Likewise, component traits are represented by instances of the service's associated Traits type.  To establish that association, a service type may have an optional TraitsAttribute applied.

[Traits(typeof(MyServiceTraits))]
public interface MyService
{
    void DoSomething();
}

public class MyServiceTraits : Traits
{
    public string SomeInformation { get; set; }
    public string[] MoreInformation { get; set; }
    public Icon MyIcon { get; set; }
}

Traits parameters are bound just like component parameters except that they appear in the <traits> element of the ,plugin XML.

Plug-in Dependencies

In the .plugin XML, the <dependencies> element specifies the ids of other plug-ins that are required by this plug-in.

The dependency information allows Gallio to do a few useful things:

  • If a plug-in dependency is missing or disabled then all dependent plug-ins are automatically disabled.
  • Before activating a plug-in, all of its dependencies are activated.  (Note: Not implemented in v3.1 but may appear in a later version.)
  • When verifying the plug-in installation, Gallio can check that all of the services required by a plug-in are provided by that plug-in or by one of its dependencies.  This check helps developers find errors in component registrations.
Assembly References and Probing Paths

Each plug-in describes the list of assemblies that it required in the <assemblies> element of its plug-in XML.  Before attempting to use any components defined by a plug-in, Gallio first ensures that the plug-in's assemblies can be resolved.  Assembly references can also be qualified with binding redirects and other loader options.

A plug-in can also declare custom probing paths in a <probingPaths> element.

Here are a few possibilities:

<probingPaths>
    <probingPath>customBins</probingPaths>
</probingPaths>

<assemblies>
    <assembly fullName="MyAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e" codeBase="MyAssembly.dll" qualifyPartialName="true"  applyPublisherPolicy="true">
        <bindingRedirects>
            <bindingRedirect oldVersions="0.0.0.0-1.1.0.0" newVersion="2.0.0.0" />
        </bindingRedirects>
    </assembly>

    <assembly fullName="MyAssemblyInGAC, Version=1.0.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e" />
</assemblies>

Plug-in Files and Recommended Installation Path

In order to facilitate plug-in installation, each plug-in includes two pieces of information:

  • A list of files in a <files> element.
  • A recommended installation path (relative to the Gallio bin directory) in the recommendedInstallationPath attribute of the <plugin> element.

Gallio uses this information in two ways:

  • During plug-in installation it is able to determine which files belong to which plug-ins so that it can copy or delete just the necessary files.  (Note: Automatic plug-in installation is not implemented in Gallio v3.1 and will appear in a later release.)
  • On request, it checks the information to verify that all plug-ins are installed as expected and that no files are missing.  This is part of what happens when you run the "Gallio.Utility.exe VerifyInstallation" command.
Conditional Plugin Enablement

Plug-ins can include a conditional expression that specifies when the plug-in should be enabled based on characteristics of the environment.  For example, some plug-ins related to Visual Studio integration are only enabled when Gallio is hosted by the appropriate version of Visual Studio.

The conditional expression appears in the enableCondition attribute of the <plugin> element of the .plugin XML files.

Conditions are represented in the code by Condition objects.  Each Condition consists of a simple boolean expression of terms drawn from a ConditionContext and combined using logical "or".  For plug-in enablement the runtime uses the RuntimeConditionContext object to query properties in the environment.

The following condition properties are currently supported.

  • "${env:ENVIRONMENTVARIABLE}": Satisfied when the environment contains a variable called "ENVIRONMENTVARIABLE".
  • "${minFramework:NET20}", "${minFramework:NET30}", "${minFramework:NET35}", "${minFramework:NET40}": Satisfied when the currently running .Net runtime version is at least the specified version.
  • "${process:PROC.EXE}", "${process:PROC.EXE_V1}", "${process:PROC.EXE_V1.2}", "${process:PROC.EXE_V1.2.3}", "${process:PROC.EXE_V1.2.3.4}": Satisfied when the currently running process main module is "PROC.EXE" and exactly matches the specified file version components (if any).

For example, here's part of the .plugin XML file for the Gallio35 plug-in which provides .Net 3.5 specific extensions.

<plugin pluginId="Gallio35"
        recommendedInstallationPath=""
        enableCondition="${minFramework:NET35}"
        xmlns="http://www.gallio.org/">

Runtime API

Gallio clients mainly interact with the plug-in model by way of the following types:

  • RuntimeAccessor: Accessor for the core runtime objects.
  • IRuntime: Configures and provides access to the plug-in registry, service locator, runtime logger, and other runtime parameters.
  • IRegistry: A registry of all installed plug-ins, services and components.
  • IServiceLocator: A service locator for resolving services and components provided by plug-ins.

The initialization process looks a bit like this:

  • Client code calls RuntimeBootstrap.Initialize() to initialize the runtime.
  • The runtime creates a CachingPluginLoader.
  • The caching plugin loader loads plug-in metadata into a PluginCatalog.  The metadata is represented in the catalog as PluginRegistration, ServiceRegistration, and ComponentRegistration objects.
  • The runtime augments the catalog with a few built-in services of its own.
  • The runtime creates a new Registry and populates it from the catalog.  Gallio performs many checks during this stage to ensure that plug-ins are well-formed and that all of their assemblies are accessible.  It automatically disables plug-ins that have problems.

Once the registry has been initialized, client code can use it in different ways.

  • The registry provides methods for getting IPluginDescriptor, IServiceDescriptor, and IComponentDescriptor objects.  Descriptors are used to get information about plug-ins, services and components as well as to obtain instances of components and traits objects.
  • The service locator provides methods for resolving services.  These are the same kinds of familiar operations exposed by most .Net IoCs such as Resolve, and ResolveByComponentId along with a couple novel ones like ResolveHandle and ResolveHandleByComponentId which obtain late-bound ComponentHandles.  Gallio does not use the service locator explicitly very often.  Most service location is performed implicitly by the runtime as part of dependency injection.

The ComponentHandle type is interesting in that it allows clients to obtain a typed reference to a component descriptor without resolving the component instance itself until needed.  Here is an example usage of a component handle to enumerate installed components and choose to instantiate one or more of them based on some criterion.

var frameworks = new List<ITestFramework>()
ComponentHandle<ITestFramework, TestFrameworkTraits>[]  frameworkHandles = // ...

foreach (var frameworkHandle in frameworkHandles)
{
    TestFrameworkTraits traits = frameworkHandle.GetTraits();
    if (IsSupportedFileType(traits.FileTypes, testFile))
    {
        frameworks.Add(frameworkHandle.GetComponent());
    }
}

Brief Comparison with Other Plug-In Models

Gallio's plug-in model is similar to the Eclipse plug-in model, Mono.Addins and MEF.

Eclipse and Mono.Addins both use XML to register plug-ins and to declare extensions points (services) and extensions (components).  Extensions can provide ample metadata in the form of XML extension configuration data.  Both of these frameworks also have provisions for packaging plug-ins as archives that can be installed incrementally.  (Of course, Mono.Addins was inspired by Eclipse so this is no surprise.)

MEF supports late-binding Exports (component handles) and makes available ExportMetadata (traits) for clients to use.  MEF is mainly attribute-based but can be configured to use external metadata files by providing an appropriate Catalog implementation.  It is quite powerful and will be included as a standard system component in the .Net Framework 4.0 release.

Aside: The original Gallio plug-in model used the Castle Windsor inversion of control container as its foundation.  Unfortunately Windsor does not support late-binding (it resolves all type names to Types on initialization) and it has limited support for component metadata.  It is a great tool (and I highly recommend it for many projects) but just not well suited to Gallio's needs.

Why Not Use Mono.Addins or MEF for Gallio?

This was a tough call.

I took a good long look at Mono.Addins and MEF beforehand.  Both are good options but I had a couple of problems with each of them.

I rejected Mono.Addins because I was not confident that I would be able to morph it into the kind of shape I needed for Gallio.  I found the code to be tightly coupled in several places that I knew I would have to change dramatically to get the kind of dependency injection and late-binding features that I wanted.

And MEF unfortunately required .Net 3.5 as a minimum whereas I had selected a target framework of .Net 2.0 for Gallio.  I briefly considered porting MEF to .Net 2.0 but gave up once I saw how much of System.Core it used.  Also it was clear that the attribute model would not work well for Gallio's late-binding needs so I would not benefit from those features.  Oh well.  I look forward to trying MEF out another day with a .Net 4.0 project perhaps.

So it just seemed more straightforward to start over which would give me a maximum of flexibility in the implementation.  As it happens, I have taken advantage of that flexibility many times over to make things better and I can say I am very happy I did not try to shoehorn Gallio into Mono.Addins or MEF.  For example, some of the plug-in metadata Gallio uses would be very awkward to implement and consume with either of these libraries.

YMMV.

Future Directions

Here are some things to expect from future versions of the Gallio plug-in model:

  1. A plug-in manager tool for downloading, installing, and maintaining plug-ins.
  2. A plug-in activation process to enable plug-ins to have more control over their early initialization.
  3. Extensible object dependency resolvers.
  4. More optimizations for start-up performance.
  5. Component scopes so that test frameworks can install special extensions while they run.  eg. custom object formatters for MbUnit.

 

Technorati Tags:

Friday, September 11, 2009

Announcing Gallio and MbUnit v3.1!

Gallio v3.1 is a major upgrade to the platform.  This release includes many new features, better performance, and improved robustness.

Highlights:

  • Video capture and embedding in test reports.
  • RSpec!
  • Support for .Net Framework 4.0.
  • Support for Visual Studio 2010 Beta 1.
  • Control Panel application.
  • Brand new plug-in model with improved startup performance.
  • Major internal redesign.
  • More MbUnit goodness.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.0.6, v3.0.6 update 1, v3.0.6 update 2, all versions

Installation Notes

We recommend completely uninstalling any previous installation of Gallio before installing the new one.  Be sure to shut down Visual Studio while uninstalling because some files may still be in use.

We strongly recommend installing KB963676 before installing Gallio.  This patch fixes some Visual Studio crashes that may occur while editing ASP.Net pages.

New and Noteworthy

Video Capture and Embedding

Gallio now includes support for capturing video screen recordings and embedding them in test reports.

Why?

Imagine you were writing integration tests for a Web or GUI application through the UI using a library like WatiN or White.  When the test fails, it is very handy to have a record of just what happened.

Magic

In the next couple of sections I'm going to show how you can capture and embed screen recordings like the following into Gallio test reports.

Screen Capture and Embedding

This is all it takes to capture and embed a screenshot in the test report.

image

Or if we just want to capture a screenshot when a test fails or is inconclusive, here's how we can do it.  We can also scale the image by 25% to keep the report small while still preserving enough detail to read.

image

As you can see the Capture class has some interesting methods.

Creating and Embedding Videos

A video is just an encoded sequence of frames.  So we can build up a screen recording by taking screenshots at intervals and adding them to an instance of the Video class such as a FlashScreenVideo.

image

Notice that you can create videos of anything you like.  You are not limited to screen recordings.

Screen Recording and Embedding

Gallio offers a few shortcuts to simplify screen recording in particular.

The ScreenRecorder class captures screenshots in a background thread and assembles them into a Video that can later be embedded.

image

The Capture class also provides a method for automatically embedding a screen recording based on the outcome of the test.  It starts a screen recorder in the background and when the test finishes decides whether to embed the video or discard it.

This is a very powerful one-liner...

Here we set things up to capture a screen recording at 5 frames per second and 25% zoom factor and cause it to be automatically embedded if the test fails or is inconclusive.

image

Overlays

Raw screen recordings are very useful but we can enrich them with additional information.  For example, while playing back the recording, it can be useful to see a transcript of what the system was doing when it captured the frame.

Gallio supports applying any number of Overlays to any screen recording.  An overlay is a basically an object that paints itself onto an image.

We're going to use a CaptionOverlay to add a caption to the video as we record it.  We can change the caption between frames to display a variety of information.

image

That's a lot of work so the Capture class provides a few convenience methods for setting a caption overlay for its screen captures and screen recordings.

image

Magic Recipe for WatiN

Here's the whole recipe for hooking up automatic screen recordings with WatiN 2.0.

public abstract class WatiNFixture
{
    protected Browser Browser { get; private set; }

    [SetUp]
    public void SetUp()
    {
        Logger.LogWriter = new GallioLogger();

        Capture.SetCaptionAlignment(HorizontalAlignment.Center, VerticalAlignment.Bottom);
        Capture.SetCaptionFontSize(32);
        Capture.AutoEmbedRecording(TriggerEvent.TestFailedOrInconclusive,
            "Screen Recording",
            new CaptureParameters() { Zoom = 0.25 }, 5 /*frames per second*/);

        IE ie = new IE();
        ie.ShowWindow(NativeMethods.WindowShowStyle.ShowMaximized);
        Browser = ie;
    }

    [TearDown]
    public void TearDown()
    {
        Capture.SetCaption("");

        if (Browser != null)
        {
            Browser.Close();
            Browser = null;
        }
    }

    private sealed class GallioLogger : ILogWriter
    {
        public void LogAction(string message)
        {
            Capture.SetCaption(message);
            TestLog.WriteLine(message);
        }

        public void LogInfo(string message)
        {
            TestLog.WriteLine(message);
        }

        public void LogDebug(string message)
        {
            // Ignore these messages.
        }
    }
}

public class SampleTest : WatiNFixture
{
    [Test]
    public void GoogleSearchAndVisitMaps()
    {
        Browser.GoTo("http://www.google.com");

        Browser.TextField(Find.ByName("q")).TypeText("Ottawa");
        Browser.Button(Find.ByName("btnG")).Click();

        Browser.Link(Find.ByText("Ottawa, ON Canada")).Click();

        Browser.Link(Find.ByText("Parliament Hill")).Click();

        Browser.Link(Find.ByText("Full article")).Click();

        Assert.Fail("Just for show.");
    }
}

RSpec and IronRuby

Now for something completely different: RSpec integration.

Big News

The big news here is that Gallio now supports running tests that are defined in files instead of just .Net test assemblies.  This means you can pretty easily create custom test framework adapters for any kind of file that you know how to process and then run those tests using Icarus and other tools.

Yes, that means you can implement 5 methods of ITestDriver (or one of the helper subclasses Gallio provides) and have a custom test framework of your own using whatever kinds test files you like be they spreadsheets, XML files, native executables, or programming language text.

RSpec in Action

Let's suppose we have implemented a "Bowling" class that computes the score of a bowling game.  (This example is borrowed from the RSpec site.)

class Bowling
    def hit(pins)
    end

    def score
        0
  end
end

Obviously this implementation will not get us very far.  So we write some specs to guide us along.

require 'bowling'

describe Bowling do
  it "should score 0 for gutter game" do
    bowling = Bowling.new

    20.times { bowling.hit(0) }

    bowling.score.should == 0
  end

  it "should score 300 for perfect game" do
    bowling = Bowling.new

    12.times { bowling.hit(10) }

    bowling.score.should == 300
  end

  it "should score 20 for single pin hit each ball" do
    pending("Scoring to be implemented.")
  end
end

RSpec + Icarus

Now we just add the bowling_spec.rb file to Icarus, and run...

image

RSpec + Icarus + AutoRun = Bliss

When programming in a BDD way you will probably find it useful to enable the Icarus auto-run feature.  Then whenever you edit the spec, they will be reloaded and re-executed.

It's a little slow right now but expect it to get better in future versions.

image

MbUnit Features

Yann Trévin has been busy cooking up new features for MbUnit v3.1.

Retry.Until

The Retry class makes it easy to implement polling loops in tests.  Polling loops are useful when the test must wait for an asynchronous operation to complete but does not know how long it might take.

image

More information about Retry.Until.

Assert.Sorted and Assert.Distinct

Two new assertions for checking whether the elements in a collection are sorted or are distinct.

More information about Assert.Sorted.

More information about Assert.Distinct.

Assert.ForAll and Assert.Exists

Two new assertions for checking whether a property holds true for all elements of a collection or for at least one of them.  These assertions are incredibly useful.

image

More information about Assert.ForAll and Assert.Exists.

Assert.Throws for Inner Exceptions

Assert.Throws now accepts an optional second type parameter to specify an inner exception type.

More information about Assert.Throws in general.

StructuralEqualityComparer

Did you ever have a data structure that you wanted to check for equality but it did not override Object.Equals?  Ever grumble about having to implement this check manually?

MbUnit v3.1 includes a new feature called a structural equality comparer.  The idea is to make it super easy to compare all members of a structure according to some rule.  Basically you just provide a list of lambda expressions to select structure members to compare and then you optionally provide a comparison predicate.  MbUnit will then compare the structure member by member using the provided selectors and predicates.

image

More information about StructuralEqualityComparer.

EnumData, SequentialNumbers, RandomNumbers and RandomStrings

Yann wrote a bunch more nifty data source attributes.  Here they are in one big combinatorial test.

image

[MultipleCulture]

The [MultipleCulture] attribute runs a test repeatedly with different cultures.

image

[Impersonate]

The [Impersonate] attribute runs a test using different user credentials.

image

TestContext.AutoExecute

TestContext.AutoExecute makes it easy to register actions to perform when a test passes or fails without needing to write a [TearDown] method.

This example captures and embeds a screenshot in the test report whenever the test fails.

image

Control Panel

Gallio now has a control panel application where you can set all sorts of preferences.

Feel free to suggest other things for us to add to the control panel.

Gallio Runtime Settings

The Gallio runtime settings let you tweak some global parameters.  Currently it lets you set custom plug-in paths in case you have installed other Gallio plug-ins elsewhere in your system.

Expect this list to grow...

image

Icarus Settings

All of the Icarus settings have moved to the Control Panel so you can configure them in the same place as other settings.

image

TestDriven.Net Settings

Now we have a place to configure settings for TestDriven.Net.

The Frameworks page lets you configure whether Gallio will be the default or preferred test runner for different frameworks.  For example if you have csUnit installed and you prefer to use Gallio to run csUnit tests with TestDriven.Net then you can set Gallio's priority to "Preferred".  If you set Gallio's priority to "Default" then Gallio will only be used if there is no other installed test runner available for a particular framework.

image

AutoCAD Settings

Mike Sandberg contributed a new preference panel for configuring the AutoCAD integration.

The AutoCAD integration still supports the "AcadAttachToExisting" and "AcadExePath" runner properties but having a GUI is certainly much nicer...

image

Plugin Diagnostics

Just for completeness, Gallio also displays a list of all installed plug-ins.

(Later on we will be adding a proper plug-in manager with the ability to download and install plug-ins separately.)

image

Visual Studio Test Tools Enhancements

Visual Studio 2010 Beta 1

Gallio v3.1 now includes support for Visual Studio 2010 Beta 1 out of the box.

image

Data-Driven Test Results (Fixed)

Gallio now correctly displays the results of data-driven tests like MbUnit row tests.

In the following screenshot you can clearly see both test results of a data-driven test.  Previous versions of Gallio would only show one result.

image

AutoCAD Integration

In addition to the new preference panel, we have made a couple of minor enhancements to the AutoCAD testing experience.

Debugging AutoCAD Tests

With Gallio v3.1, you can now debug your AutoCAD tests using Icarus just like any other tests.

Previously debugging did not work because the Visual Studio debugger is unable to attach to a process that uses fibers (like AutoCAD does).  So Gallio v3.1 launches AutoCAD with the debugger attached from the start and it works.

We also support zero-installation execution of AutoCAD tests.  This means you can run tests on a machine with AutoCAD installed without having to separately install Gallio as long as the files have been copied to the local disk somewhere.

TeamCity Integration

Gallio v3.1 now automatically detects when it is being run by TeamCity and enables the TeamCity test runner extension.  This means you do not need the following in your build scripts any more: "TeamCityExtension,Gallio.TeamCityIntegration".

There have also been some fixes to more correctly output test results from parallelizable tests.

Performance

We have invested some time improving the performance of Gallio and MbUnit v3.1 in several key areas.

Startup Time

Gallio v3.1 is much faster to start up than previous versions of Gallio.

Much of this gain comes from three important changes.  First, the new plug-in model in Gallio exposes more declarative metadata up front so that we can defer loading assemblies until they are actually needed.  Second, once we have identified all of the plug-ins that are installed, we cache the composite metadata for subsequent runs.  Third, we generate XmlSerializers ahead of time.

On my slow laptop Icarus startup time dropped from over 20 seconds to about 2 seconds.  These startup time improvements also carry over to the Visual Studio and ReSharper integration.

Memory Usage

Gallio v3.1 uses significantly less memory when running multiple test assemblies than it did in v3.0.6.

Older versions of Gallio used to keep all of the test assemblies loaded in memory for the duration of the test run.  When running dozens of test assemblies together at once (say, during a continuous integration build) Gallio would end up consuming quite a lot of memory in proportion to the number of test assemblies.

Now Gallio v3.1 loads and unloads each test assembly before moving on to the next one.  This change improves responsiveness and keeps memory usage in check.

Debugging and Profiling

One of the more perplexing performance problems we addressed in MbUnit v3.1 was the performance of the debugger.

In older versions of MbUnit v3 it would sometimes take half a second or more to single-step while running a test in the debugger.  The problem is that the stack was just too deep.  During a typical test run, the stack could be 160 frames deep!

Now MbUnit v3.1 is more carefully tuned to minimize its stack depth.  We have reduced the effective stack depth by a factor of 5 so single-stepping is zippy and responsive just like it should be.

One happy side-benefit of reduced stack depth is that it significantly easier to figure out what it going on when running tests with a code profiler.

.Net 4.0 Support and x86/x64 Processor Architecture Enhancements

Gallio v3.1 includes out of the box support for .Net 4.0.  We also improved our support for running tests in different processor architectures.

One key change is that Gallio runs each test assembly in its own process by default.  Gallio automatically detects the required .Net runtime version and processor architecture by examining the assembly metadata.  You can run tests designed for different .Net runtimes or processors architectures side-by-side if you like.  All you need to do is compile your test assemblies with the right options.

It just works.

Robustness

We have worked hard to improve the robustness of Gallio v3.1 while adding new features.

UAC Privilege Elevation on Vista and Windows 7

Did you notice that the Control Panel application displayed the UAC shield icon?  Here it is again:

image

The shield indicates that Gallio has detected that the Control Panel is running in a lower privilege environment and that certain settings that have changed require elevation.  Of course when you click on the button you will receive a UAC prompt after which Gallio will carry out the required operation in an elevated context.

ReSharper Integration Bug Fixes

We have spent some time improving the robustness of the ReSharper integration.  It used to throw up annoying exceptions here and there and occasionally would crash.  Hopefully this is all a thing of the past.

Here's some bad code that used to cause Visual Studio to crash but not anymore:

public class A : B
{
}

public class B : A
{
}

Gallio.Utility Power Tool

Gallio.Utility.exe is a new command-line application included with Gallio to perform various functions related to maintaining a Gallio installation.

These are the supported utility commands at this time:

  • ClearCurrentUserPluginCache:  Clears the plugin cache of the current user.
  • ResetInstallationId:  Resets the installation id.  The plugin list will be refreshed the next time a Gallio application is started.
  • Setup:  Installs or uninstalls components.
  • VerifyInstallation:  Checks for runtime installation errors.

More Goodies

MbUnit:

  • MbUnit assertions offer EqualityComparison<T> and IEqualityComparer<T> overloads.
  • Parallelizable tests run with the correct number of threads and start up immediately without requiring any prior "warm-up" period as was seen in earlier versions due to some bad logic.
  • Support the use of Assert.Inconclusive and Assert.Terminate within Assert.Multiple blocks.
  • Contract verifiers can be "static" in which case they behave like static test factories instead of like dynamic test factories.  In a "static" contract verifier, each individual verification will appear as a separate test in the test runner.

csUnit:

  • Upgraded to csUnit v2.6.

NUnit:

  • Upgraded to NUnit v2.5.2 but retained support for NUnit v2.4.8 also.

MSTest:

  • Rewrote most of the MSTest integration to be faster and more robust.

Command-line tools:

  • Gallio.Echo.exe supports wildcards.
  • The NAnt, MSBuild and PowerShell tasks support specifying the Verbosity.
  • Echo support a /no-progress switch to disable progress reporting.

Icarus:

  • Icarus now has a "Recent Projects" menu.
  • Icarus test tree is a bit faster.
  • Lots of Icarus bug fixes.
  • New error dialog.
  • Support keyboard navigation.
  • Disabled the watchdog timer during debugging runs so you can debug at leisure without worrying about the test runner killing the process prematurely.

Test Reports:

  • Support embedded HTML content and Flash videos.
  • Workaround issues when clicking links to attachments in IE due to the Local Machine Zone Lockdown.
  • Display test kind icons in the report.
  • Added a condensed text report type.
  • Test output is normalized before writing it to the log.  This is to catch problems like attempting to write a null character or BOM into the report.
  • Test framework nodes are no longer emitted.

Debugging and Code Coverage:

  • Gallio launches the debugger with the "Managed and Native" debugging engine selected so that you can debug into native code from managed code.
  • NCover v1.5.8 works on x64 but only supports running tests in x86 mode.
  • Using NCoverExplorer or NCover.Reporting to merge NCover coverage reports when running test assemblies in separate processes.

ReSharper:

  • Lots of bugfixes.

TestDriven.Net:

  • Gallio now supports zero-installation TestDriven.Net test runs with recent beta builds of TestDriven.Net.  I expect Jamie will say more about this feature soon...

Pex:

  • MbUnit.Pex no longer included.  The Pex project has taken over the maintenance of this extension.

CruiseControl.Net:

  • Added support for CruiseControl.Net v1.4.4 and more recent versions.

Documentation:

  • Yann performed a massive documentation sweep to clean up formatting.
  • Switched to Sandcastle instead of NDoc 2 Alpha.

Core:

  • Brand new test framework extension API.  Only 5 methods to implement.
  • Early Transition to a message-based protocol for test framework integration.
  • Massive refactoring of namespaces and interfaces.
Technorati Tags: ,