Death By a Thousand Cuts

It feels like I'll never get this release out.

I'm currently facing an interesting flavour of DLL hell (tastes like bananas) resulting from copy-local assemblies that get loaded into the same application (Visual Studio) in different load contexts.  Basically, ReSharper and Visual Studio Team Test both want to load Gallio.dll but they do it in different ways from different locations.  This results in some rather curious behavior because two identical copies of the assembly will be loaded but they are nevertheless considered different from one another.

It would be nice if we could just tell ReSharper and Visual Studio Team Test to load my add-in assemblies from the Gallio installation path, but neither of them actually support that behavior.  They both require their add-ins to be copied into very specific folders.

I can think of two possible fixes right now:

1. Add some hacks to install a custom assembly resolver just in time at all entry points into the add-ins so that we can control more precisely how Gallio.dll gets loaded.  That way we can avoid having multiple copies of Gallio.dll.
2. Put Gallio.dll into the GAC.

I previously used option #1 for the Gallio ReSharper add-in.  It was a real pain because I had to carefully instrument all entry points into the add-in and avoid referencing any Gallio types until the assembly resolver hook was installed.  I ripped this brittle code out a couple of weeks ago since I figured just copying Gallio.dll locally would be good enough.  (guess again!)

So option #2 really seems best.  Keep in mind that the copy of Gallio in the GAC should only be used for external tools integration.  I don't anticipate test assemblies or the basic standalone runner actually referring to anything in the GAC so Gallio should still be usable in-place without requiring any installation steps (except for these add-ins of course).

However, it's 4am now so I'm going to sleep on it for a little while.  It may be that using the GAC will cause more headaches such as problems with the NAnt or MSBuild tasks that might also involve a little non-standard assembly resolver-foo of their own.

 

Next problem: Throw out NSIS (finally!) and rewrite the installer using WiX...

Technorati Tags: gallio

The C# Null-Coalescing Operator, PEVerify and ILMerge

It seems I errantly blamed ILMerge for those PEVerify errors. At worst it may be a C# compiler optimization gone awry or it may be that PEVerify is not smart enough.

The Source

Consider this code:

public IList<ITestCommand> Children
{
    get { return children ?? (IList<ITestCommand>)EmptyArray<ITestCommand>.Instance; }
}

The left-hand side of the null-coalescing operator has type: List<ITestCommand>.

The right-hand side of the null-coalescing operator has type: ITestCommand[] before the cast.

The result has type: IList<ITestCommand>.

So we need a little silent cast in there to make the types match up on either side.  Really, it's just providing a compatible upper bound to which both sides can be converted.  At runtime, this operation is a no-op.  Technically it's something that the compiler itself could have inferred if it were smart enough.

The Unoptimized IL

Let's take a quick look at the IL that the C# compiler generates for a standard unoptimized build.

Notice that the castclass operator appears in the listing.  There is also quite a bit of jiggery-pokery involving a compiler generated temporary local variable.

.method public hidebysig specialname newslot virtual final instance class [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand> get_Children() cil managed
{
    .maxstack 2
    .locals init (
        [0] class [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand> CS$1$0000)
    L_0000: nop 
    L_0001: ldarg.0 
    L_0002: ldfld class [mscorlib]System.Collections.Generic.List`1<class Gallio.Model.Execution.ITestCommand> Gallio.Model.Execution.ManagedTestCommand::children
    L_0007: dup 
    L_0008: brtrue.s L_0015
    L_000a: pop 
    L_000b: ldsfld !0[] Gallio.Collections.EmptyArray`1<class Gallio.Model.Execution.ITestCommand>::Instance
    L_0010: castclass [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand>
    L_0015: stloc.0 
    L_0016: br.s L_0018
    L_0018: ldloc.0 
    L_0019: ret 
}

The Optimized IL

When generating optimized code, the C# compiler realizes that the castclass is unnecessary.  So it drops it.  It also drops the unnecessary load/stores to a local variable and we end up with the following code.

.method public hidebysig specialname newslot virtual final instance class [mscorlib]System.Collections.Generic.IList`1<class Gallio.Model.Execution.ITestCommand> get_Children() cil managed
{
    .maxstack 8
    L_0000: ldarg.0 
    L_0001: ldfld class [mscorlib]System.Collections.Generic.List`1<class Gallio.Model.Execution.ITestCommand> Gallio.Model.Execution.ManagedTestCommand::children
    L_0006: dup 
    L_0007: brtrue.s L_000f
    L_0009: pop 
    L_000a: ldsfld !0[] Gallio.Collections.EmptyArray`1<class Gallio.Model.Execution.ITestCommand>::Instance
    L_000f: ret 
}

PEVerify Has a Conniption

Without the castclass in the optimized code, the IL is now missing a crucial piece of information: what is the type of the value of the top of the stack when we reach the ret instruction?

In one case, we may have duplicated the value produced by the ldfld instruction so the top has type List<ITestCommand>.

In the other case, we have discarded the original value and instead loaded an array from somewhere else using ldsfld so the top has type ITestCommand[].

Uh oh!

[IL]: Error: [C:\Source\MbUnit\v3\src\Gallio\Gallio\bin\Gallio.dll : Gallio.Model.Execution.ManagedTestCommand::get_Children][offset 0x0000000F][found ref 'System.Collections.IEnumerable'][expected ref 'System.Collections.Generic.IList`1[Gallio.Model.Execution.ITestCommand]'] Unexpected type on the stack.

Without the castclass PEVerify has to do some rather more complicated work to figure out whether the types are compatible.  Apparently, it fails.  But it does get far enough to realize that one of the values can be converted to IEnumerable.

Is This a Problem?

I don't know.  Apparently we have been running this code in Gallio release builds for many months now and not seen any InvalidProgramException errors.  So even though PEVerify is confused, the CLR at least does not choke on the code.

In any case, it's definitely not ILMerge's fault.  (Sorry Mike!)

Technorati Tags: low-level

ILMerge redux

In my previous post, I mentioned a problem I was having with ILMerge and how I fixed it.

It turns out that code security attributes were not the only problem.  ILMerge is no longer complaining, but it is generating a bad output assembly just the same.  This is preventing the C# compiler from linking to the merged assembly.  Most immediately it manifested as XmlSerializers not working.

PEVerify shows a long stream of errors:

C:\Source\MbUnit\v3>"c:\Program Files\Microsoft.NET\SDK\v2.0 64bit\Bin\PEVerify.exe" build\target\bin\Gallio.dll

Microsoft (R) .NET Framework PE Verifier.  Version  2.0.50727.42
Copyright (c) Microsoft Corporation.  All rights reserved.

[MD]: Error: TypeDef is marked Nested but has no enclosing type. [token:0x020004
40]
[MD]: Warning: MemberRef has a duplicate,  token=0x0a0004f3. [token:0x0A0001AB]
[MD]: Warning: MemberRef has a duplicate, token=0x0a0004f4. [token:0x0A0001AC]

...

*sigh*

More Good Fortune?

I upgraded ILMerge from 2.8.0202 to 2.8.0422 and now it seems to be working a little better.

Now the TypeDef error is gone but we're still getting MemberRef warnings.  Makes me a bit nervous but at least the C# compiler is happy enough linking to the merged assembly and the XmlSerializer problem seems to be resolved.

However, there was nothing in the change history to suggest that any bug were fixed.

Or Maybe Not?

Turns out I missed these errors:

[IL]: Error: [C:\Source\MbUnit\v3\build\target\bin\Gallio.dll : Gallio.Model.Execution.ManagedTestCommand::get_Children][offset 0x00000012][found ref 'System.Collections.IEnumerable'][expected ref 'System.Collections.Generic.IList`1[Gallio.Model.Execution.ITestCommand]'] Unexpected type on the stack.
[IL]: Error: [C:\Source\MbUnit\v3\build\target\bin\Gallio.dll : Gallio.Model.Execution.ManagedTestCommand::get_Dependencies][offset 0x00000012][found ref 'System.Collections.IEnumerable'][expected ref 'System.Collections.Generic.IList`1[Gallio.Model.Execution.ITestCommand]'] Unexpected type on the stack.
2 Errors Verifying build\target\bin\Gallio.dll

Just great.  Well, I guess Gallio Alpha 3 will be delayed a little bit further while I cook up a workaround or something.

Technorati Tags: low-level

Don't use self-referential Code Access Security attributes.

I encountered an interesting little problem today after completing a small refactoring of Gallio.  ILMerge broke!

An exception occurred during merging:
ILMerge.Merge: The assembly 'Gallio' was not merged in correctly. It is still listed as an external reference in the target assembly.
   at ILMerging.ILMerge.Merge()
   at ILMerging.ILMerge.Main(String[] args)

True enough, the merged assembly contained a self-reference.

The proximate cause seems to be the fact that the Gallio assembly defined a code access security attribute that it itself used.  Ordinarily these attributes should be defined in other assemblies.  Indeed, the compiler will enforce the constraint for any assembly-level permission requests.  I assume ILMerge is doing a little magic here to try to enforce similar properties.

Ahh well.  I eliminated the custom attribute and all is well.

Leap of Faith

Julian suggested I blog about this issue because of the manner in which I resolved it.

Internal dialogue:

1. ILMerge is breaking because I am using it on a different assembly than I was before.  So there's something essentially wrong about this assembly or with ILMerge.
2. Google turns up this error message but only in circumstances involving the ASP.Net merge tool and user error.  I don't think that's the case here.
3. The documentation doesn't say anything about this.
4. Renaming files around and messing with the command-line arguments doesn't change anything.
5. Panic!

Time passes...

1. Hmm.  When skimming the ILMerge change history to find the entry about to the addition of this error message didn't I see something about fixing security attributes?  I wonder what that was about.
2. Gallio uses a custom security attribute.
3. Security attributes are subject to these interesting cross-assembly reference constraints when used to declare minimum permission requests...
4. Oh!

Now I have no idea why the mention of security attributes in the change history caught my attention, but it did.

You might call it lateral thinking, but I'll call it sheer luck!

Technorati Tags: low-level

Gallio integration with Visual Studio Team System

One of the questions I've often received about Gallio has been about whether it would integrate with MSTest and Visual Studio Team System.

Gallio running MSTest tests in Icarus

Julian did the work of writing a Gallio adapter plugin for MSTest tests.  Gallio explores MSTest tests using reflection and runs them by shelling out to MSTest.exe and parsing the test result XML file.

Here's a screenshot of the Gallio Icarus GUI running some MSTest tests.

Gallio running MSTest tests in ReSharper

Of course, you can use any Gallio test runner to run MSTest tests.  That means you automatically get ReSharper integration too!  (And in all of the other runners too.)

 

MSTest running Gallio tests in Visual Studio

Another common request has been to provide integration so that Visual Studio Team Test can run Gallio tests in the IDE and using the MSTest.exe command-line.

This particular request was echoed to me a couple of times at ALT.Net from people who, for good or ill, have been instructed by management to use the Team System test runner instead of TestDriven.Net or ReSharper.  So I decided to give it a shot.

Gallio Alpha 3 now includes an experimental package for Visual Studio to do just that.  (It also works on the command-line).

Technorati Tags: gallio,mbunit

Gallio Archimedes

What do you do when you have lots of automated integration tests?

Integration tests are different from unit tests.  They are developed by different people for different purposes with different priorities, different runtime characteristics, different maintenance overhead, different reporting needs, and different management demands.

Unit tests should be fast, sleek and isolated.  They're the Formula 1 racers of the test world.  They whiz around the track faster than you can say Continuous Integration!

Integration tests should be comprehensive, robust and realistic.  They're methodical and avoid taking shortcuts.  If it takes 5 minutes to simulate a particular kind of transaction then they will spend 5 minutes doing it for real.  (To be fair, I'm talking about black box functional regression tests here.)

The Right Tool for the Job

Where I work, we have lots of automated integration tests.  These are built by the QA Automation Team in C# using MbUnit, Gallio and WatiN.

These tests are largely intended to displace manual QA activities for acceptance testing purposes.  Consequently they are integration tests of the highest order: they interact with the web site and other system components as nearly as real people would.  But much faster and more accurately.

Still, they take a long time to run.  Running them interactively is completely out of the question.  Running them serially is painfully inefficient.  Running them only on demand wastes a valuable opportunity to generate Continuous Feedback from the system and to spot non-deterministic bugs.

Moreover these tests really belong to the QA arm of our organization.  The operators are QA, Operations and Engineering staff.  The consumers are all of the above plus Program Management and other stakeholders.

So all told, our integration testing needs are vastly different from our unit testing needs.  So we need different tools.

If There Is No Suitable Tool, Build It

For the past few months I have been thinking about building a tool to assist with deploying, running and managing automated test cases.

There are a lot of very interesting problems to solve, particularly in the user interface and overall workflow model.  Consider that many users of the system will not be test authors themselves.  Results must be meaningful, reliable, and easy to interpret.

Here's a sketch of what I have in mind for this new addition to the Gallio tool chain: Archimedes.

Archimedes will incorporate:

• A distributed test runner.
• A web-based management console.
• A repository for historical test data.
• A deployment engine.
• A scheduler.
• A notification service.
• A reporting, trending and statistics module.
• A data feed and web service for mash ups.
• Documentation.

Archimedes will leverage and extend the Gallio platform.  It will work with all supported test frameworks such as MbUnit, NUnit, xUnit.Net, MSTest, NBehave, and others yet to come.

Sound Like Fun?

Reply in comments or send an email to the Gallio-Dev Mailing List.

Technorati Tags: gallio,mbunit

Where is Gallio Alpha 3?

Earlier I announced that Gallio Alpha 3 would be released over the weekend.  As it happens, I ended up spending that time away from the computer instead.  (Yay!)

The good news is that I've been taking advantage of the extra time to fix a few more bugs and to add a few new features that some people have asked for, such as support test assembly configuration files.

Here is my tentative roadmap going forward:

• Alpha 3: Bug fixes and improvements to reports, hosting, compatibility, data-driven tests, error reporting (annotations), and the GUI.
• Alpha 4: New constraint framework and asserts for MbUnit v3.  Will begin shipping .Net 2.0 and .Net 3.5 flavors with more goodies.  Should also include some significant improvements to the GUI feature matrix and workflow.
• Alpha 5: Last chance to make major breaking changes to the model.  (optional)
• Beta: Feature complete platform, frameworks, and tools.  Publish the Gallio Book, tutorial material and other resources.
• Final: Finishing touches.

The target iteration cycle will be about 3 weeks to 1 month each.

I also have another project in mind to begin...

Technorati Tags: gallio,mbunit

Gallio Report Navigation

Today I made a few usability improvements to the Gallio Test Report.  The most noticeable of these is the addition of a navigation bar.  (www.gallio.org)

The navigation bar shows you an overview of the contents of the test report.  More specifically, it contains stripes that represent failing, inconclusive or skipped tests.  So if you have a lot of tests, you can quickly zip down to the ones you care about.

This feature was modeled after the ReSharper text editor error stripes in Visual Studio.

If we click on the stripe that represents the failed test, we are taken to it.  It is automatically expanded if needed along with any of its ancestors in the report.

Expect more stuff like this to come.

Note: Gallio Alpha 3 will be released this weekend with this feature, some bug fixes and much more.  I was previously going to call it an Alpha 2 Update but that got too confusing.  Release early, release often, right?

Technorati Tags: gallio,mbunit

Fun with ReSharper Highlighters

Gallio has an annotation mechanism for associating errors, warnings and informational messages with code elements that make up tests.  It is primarily used to tell the user when a test is malformed.

Naturally, the best way to present these annotations to the user is through integration with the IDE.  Here is a screenshot of the Gallio plugin for ReSharper doing just that.

Above we see an error because a single method can't have both the [Test] and [SetUp] attributes applied to it at the same time.  (Admittedly the error message here is a bit vague right now.)

What's cool is that we can inform the user about problems before the tests are even compiled!

Note: This feature will be in the upcoming Alpha 2 Update release soon along with a bunch of other improvements and bug fixes.

Technorati Tags: gallio,mbunit