A couple weeks ago I posted that the new beta with R# 5.0 and VSTS 2010 support would be available. Unfortunately I forgot to go back and post an update with a link.
Here’s the link: http://ccnet.gallio.org/Distributables/
Interim release notes: http://www.gallio.org/wiki/doku.php?id=release_notes
Happy testing!
This is just a short note to say that we will be releasing Gallio v3.2 Beta later this week with support for Visual Studio 2010 RTM and ReSharper 5.0.
While trying to figure out why running a single test with Gallio causes ReSharper 5.0 to throw up the test failure in a message box (which it never did it before), I found this sneaky little bug.
public abstract class RemoteTask : IEquatable<RemoteTask>
{
public string RunnerId { get; set; }public override bool Equals(RemoteTask other)
{
return Equals(other as RemoteTask);
}public bool Equals(RemoteTask other)
{
return other != null && RunnerId == other.RunnerId;
}public override int GetHashCode()
{
return RunnerId.GetHashCode();
}
}
Can you spot the bug?
Whenever we construct a Dictionary<TKey, TValue> without providing an explicit IEqualityComparer<TKey> in the constructor (which is most of the time), the .Net framework automatically provides one for us. Namely, it uses EqualityComparer<TKey>.Default.
EqualityComparer<T> is one of those really useful generic classes that is really easy to overlook. Its job is to choose an appropriate implementation of IEqualityComparer<T> for any given type T. Here are the cases it considers.
All of this magic is designed to do one thing: minimize the number of boxing conversions and casts required to compare objects.
Did you notice that RemoteTask is abstract? That tells you that it is intended to be subclassed.
To ensure correct equality and hash code semantics, a subclass must override at least the following two methods:
(Note: It is not necessary to override object.Equals(object) because all it does is delegate to IEquatable<RemoteTask>.Equals(RemoteTask). Due to covariance of the equality relation, this code will work fine for all subclasses as long as they override IEquatable<RemoteTask>.Equals(RemoteTask) with their extra logic.)
Unfortunately RemoteTask’s implementation of IEquatable<RemoteTask>.Equals(RemoteTask) is not virtual! Consequently subclasses cannot override it. No matter what they do, they are stuck with an implementation that only considers the RunnerId field when determining equality.
As a matter of fact, all subclasses of RemoteTask override object.Equals(object) but not IEquatable<RemoteTask>.Equals(RemoteTask). That’s really bad because the methods will probably return different results! Any code that uses IEquatable<RemoteTask> to compare RemoteTask instances will be broken.
This bug has been in ReSharper for a few releases now but it only just reared its ugly head due to an internal refactoring.
Previously, ReSharper performed a lookup of tasks using linear search, something like this:
private List<KeyValuePair<RemoteTask, UnitTestElement> tasks;
public UnitTestElement GetTestElement(RemoteTask task)
{
foreach (RemoteTask candidate in tasks)
{
if (object.Equals(task, candidate.Key))
return candidate.Value;
}
return null;
}
Notice that this code calls the object.Equals(object, object) static helper function which performs a null check then determines equality using object.Equals(object). In this case, subclasses that just override object.Equals(object) will be fine.
For efficiency, ReSharper now performs a lookup of tasks using a hash table, something like this:
private Dictionary<RemoteTask, UnitTestElement> tasks;
public UnitTestElement GetTestElement(RemoteTask task)
{
UnitTestElement element;
tasks.TryGetValue(task, out element);
return element;
}
What’s new here is that under the hood, the Dictionary<RemoteTask, UnitTestElement> uses EqualityComparer<RemoteTask>.Default. That in turn compares values using IEquatable<RemoteTask>.Equals(RemoteTask)…
Oh yeah, that’s the method that subclasses can’t override…
The net result is that the hash table considers all values to be equal as long as they have the same RunnerId, regardless of their type or whatever other properties they may consider in their implementation of object.Equals(object).
For the most part, this is only a problem when there are hash code collisions. If the hash codes are always distinct then it won’t matter to the hash table that the equality comparer it is using returns true more often than it should…
In other words, the fact that this code appears to work at all today is a matter of pure luck given that hash code collisions are fairly rare (for well designed hash functions, at least).
If you implement IEquatable<T> in a class that is intended to be subclasses, make sure to make it virtual! Also make sure the subclasses override it as needed!
Earlier today I wrote about the fact that I need to rebuild the Gallio build server. Then I went downstairs to get ready to leave for work and I promptly twisted my ankle.
So… now lying in bed with an ice pack strapped to my ankle, this seems like as good a time as any to start planning.
Gallio currently ships with plugins for several 3rd party tools. It supports the .Net CLR 2.0 and 4.0 on 32bit and 64bit Windows platforms and aims to eventually support Mono on Windows and on Linux. All told there are quite a few configurations to be tested.
Here’s what we need:
Here are some nice things to have:
Observation:
Most components are binary portable and should only be compiled once but may need to be tested more than once depending on how the configuration of the environment affects the component.
Hermetic means “sealed.”
A hermetic build environment is one which encapsulates all of its dependencies so that the result of the build is entirely reproducible.
Think for a moment about the dependencies of your own build process.
The more dependencies you control, the more reproducible will be the build.
The idea of establishing a hermetic build environment is to ensure that the dependencies are tightly controlled. If I change line 101 of Foo.cs to fix a bug, I need to know that the next build will be identical to the first except that it will contain my specific bug fix because that is the only change in the source tree.
Let’s suppose I upgraded the compiler in place. Now it’s possible that due to some changes in the compiler, the same source code that I compiled yesterday might produce different output today. That’s really bad. It means that I could introduce new bugs otherwise unchanged versions of my code just by recompiling it!
One solution is to indicate in the source code exactly which version of the compiler should be used. That way if I recompile yesterday’s code, then I will use exactly the same version of the compiler that was originally specified yesterday. The only way to introduce a change would be to commit a change in the source tree indicating that a new compiler version should be used instead.
The configuration of a hermetic build environment is well-known and consists of fully specified dependencies. The only way to change the output of the build is by changing something in the source.
Crucially, the hermetic build environment should not be permanently altered by the build process itself. For example, running tests shouldn’t leave extra temporary files floating around that remain for the next build to trip over. The environment should be pristine!
How do we implement something like this?
In Linux, we might run the whole build inside of a very carefully constructed chroot directory assuming that we don’t care about the kernel version or that we control it via other means.
In Windows, it’s a lot harder because important configuration information could in principle reside anywhere in the file system or system registry. For example, what really happens when we install a new version of .Net? Hmm… A more practical concern for Gallio is what happens when we upgrade to a new ReSharper beta, for example.
The case of the ReSharper dependency is interesting. To compile Gallio’s ReSharper extensions, it would be sufficient to have all of the ReSharper libraries checked into the source tree. However, to test those extensions we actually need a working installation of ReSharper, which itself requires Visual Studio, which itself requires many things. Checking in libraries isn’t enough. We’re going to need a virtual machine snapshot.
A build process can be described as a sequence of gathering and scattering operations.
Here’s a fairly straightforward example:
In general, tasks gather stuff from the source tree or from the output of other tasks, scatter a bunch of derived artifacts around, then eventually gather them together again into a form that can be consumed by subsequent tasks.
We can represent the dependencies as a linear sequence of steps (as outlined above) except that we would be better off if we could take advantage of the inherent parallelism of certain stages. For example, there’s no reason we couldn’t be compiling and testing the ReSharper 4.5 extension at the same time as the ReSharper 5.0 extension as long as both processes ran independently (in different virtual machines). Instead we should model dependencies as a graph of parallel or sequential tasks.
To make this work, we will need a continuous integration tool that coordinates task dependencies elegantly.
Here’s what the solution will look like for us.
Because we need to build and test many components on Windows, the most robust and perhaps simplest way to establish a hermetic build environment will be to use a suite of virtual machines.
To that end, the virtual machine host will maintain a collection virtual machine snapshots. Each snapshot will have a unique name, like “Win7 x64, VS 2010, R# 5.0.1161.9 (Snapshot 1.0)”.
There are many virtual machines that we could use, but for our purposes we will use VirtualBox. VirtualBox is free and performs well although I have found it to be somewhat unstable at times. Even so, it’s perfect for what we need.
We’re going to have a lot of snapshots. Every time we tweak one of the virtual machines, we will create a new snapshot and then check in a change to the source tree to use the new snapshot for builds of subsequent revisions. Fortunately VirtualBox supports differencing disk images and snapshot trees. As a result, we only need to store the differences between snapshots. What’s more, we can easily reset the virtual machine to a previous snapshot state just by throwing away the latest differencing disk image which is very fast.
Over time, we may choose to throw away some snapshots when they are no longer relevant for practical purposes since there will come a time when we will never need to rebuild most old revisions. In the meantime, gigabytes are cheap and plentiful, time is not!
How many virtual machines will there be? I’m not sure… there may be as many as a dozen.
The build process will be broken down into a graph of tasks. Each task will specify dependencies on other tasks whose output it consumes. We’ll use a continuous integration build manager to run the tasks and report progress.
There are many different continuous integration tools that we could use, but for our purposes we will use TeamCity. TeamCity does a very good job of managing dependencies and the professional edition is available for free and does everything we need.
Interestingly, because we will be using virtual machine snapshots as part of our hermetic build environment, we will not use TeamCity’s built-in support for cross-platform builds.
JetBrain’s recommended approach for cross-platform builds is to run multiple TeamCity build agents on multiple machines with different configurations. However for that to work, the machine has to persist the state of the built agent. The problem is that we will always wipe our virtual machines back to their snapshot state after each build. So if the agents ran inside the virtual machine they would lose their state changes completely between builds which would probably make TeamCity somewhat unhappy.
Another problem with running separate build agents inside each virtual machine is that we would have to keep all of the virtual machines running all of the time. Otherwise, the TeamCity primary server would not be able to connect to the build agents. Running so many virtual machines (dozens?) would be a prohibitively expensive waste of resources. Moreover, we would need to keep a lot of build agents running and it costs money to buy licenses for more than 3.
Our solution will be to run TeamCity build agents only on the virtual machine host (or in a virtual machine with access to the host). Each build task will be responsible for starting up the appropriate virtual machine, invoking a remote command inside the virtual machine, gathering results from inside the virtual machine, and then shutting down the virtual machine (discarding changes to its state).
All we need is one TeamCity build agent! That one build agent can manage as many virtual machines as we like.
In practical terms, we’ll probably run 3 TeamCity build agents. That way we can run 3 tasks in parallel to improve performance while still fitting within the license constraints of the free TeamCity Professional Edition.
We’re also going to need a few other things.
Err, right… I guess we need to build Archimedes too…
The build server has been in an unhappy state for quite some time now (registry corruption due to frequent crashes of the VirtualBox VM). Now it finally seems to have gone over the edge.
As a result I’ll be rebuilding the build server sometime in the next week. I think I'll set the new one up a bit differently if I have time.
Gallio ships with plugins for ReSharper 3.1, 4.0, 4.1, 4.5 and 5.0. The trick here is that in order to build and test those plugins, all 5 versions of ReSharper need to be installed. Ouch! (Ok, we will probably be dropping 3.1, 4.0 and 4.1 soon…)
Of course Gallio also ships with plugins for NCover, TypeMock, AutoCAD, TestDriven.Net, Visual Studio 2008, Visual Studio 2010, etc…
Imagine all of those components installed on the same VM? Yeah… it’s a little scary. Fortunately most of them do not conflict with the others so it mostly works out fine.
Ideally we should just build a bunch of VMs with different groups of dependencies and build them all independently. That way we can add new dependencies at will without worrying quite so much about how they interact (or worry about the 12 hours it will take to reinstall everything if the VM blows up).
We should really be testing on at least the following platforms:
Basically that means we’ll need a whole bunch of VMs for various configurations. Tricky.
Right now there are a bunch of things we don’t test rigorously in an automated way. Here are a few of them:
I spend a lot of time before each release manually testing this stuff. It just doesn’t scale.
All we really need are a few tests to get sufficient confidence that the system isn’t broken in really boneheaded ways. For example, Visual Studio shouldn’t pop up a dialog complaining about the Gallio add-in encountering an error during initialization. (Usually that means the installer is broken somehow.)
All it should take for a release is for us to spend a couple of minutes writing release notes, adding some more documentation to the wiki, and pushing a big red button to promote the current nightly build to stable.
Paradoxically, the first step in this process is that unstable builds need to be more visible.
What I mean is that we should maintain consistently up to date release notes and documentation for our pre-release builds too. We should get more people using those pre-release builds and perform more sanity-checking with automation so we’re not chasing last-minute bugs.
When it comes time to release, all we should have to do is post a new stable build label to the website and spam the newsgroups. :-)
Duh.
With my new job at Google, I’ve found it harder and harder to spend as much time as I would like on Gallio and MbUnit. It’s hard to find a good solid block of time to work on stuff without too many other distractions so I’ve been spending my time on little infrastructure projects.
The theory is that if I can build a big enough lever… then I can pretend there really are 32 hours in a day. :-)
Most of the Gallio and MbUnit web site content including documentation resources are now published to the web automatically by our Continuous Integration build server.
In other words, any committer can change the website just by checking in some code, and waiting 3-5 minutes for the web site to be built and the changes to propagate.
This kind of setup is really cool. Seriously seriously cool. So cool I know Oren has blogged about it.
Given my time resources are dwindling (and will even more once I start working on Android stuff), automating the whole pipeline is golden. I can practically pretend that it’s real-time. Hit commit and move on to another task.
Let me be blunt. The Gallio documentation is woefully incomplete.
The main problem is that I expected that the Gallio Book writing to progress more rapidly than it has so I didn’t spend much time developing alternative documentation resources.
Of course, there have been many offers to help write content for the book but little has materialized so far. That’s just how Open Source is sometimes. Lots of people want to help but there is always a barrier to contribution. In this case I’ve lowered the barrier a bit by automating the book pipeline from commit through to publication on the web but writing high quality documentation still takes time and preparation (especially in book form).
So…
Say hello to quick & dirty documentation by the masses for the masses.
The new Gallio Wiki is here!
It’s a little empty... Please help fill it up. :-)
Also, if you are interested, the old MbUnit v2 Wiki is still available on the Internet Archive on the Gallio website. We should probably try to port some of this content over, or something.
A couple of years ago, Ben Hall invested quite a lot of time getting Sandcastle and DocProject to play nicely together and assembling the documentation.
Unfortunately when we moved the MbUnit.com website a couple of years ago, we lost some content. Specifically, we lost the documentation site.
Oh… we still had the code to generate the documentation site, of course, but the specific versions of Sandcastle and DocProject that we used were no longer available and the new versions were not compatible with the old code.
Great. Bit rot. For two years, I was sitting on a pile of documentation that I couldn’t compile.
Last night I decided to bite the bullet and try again. It was a total nightmare! The whole ordeal took me maybe 12 hours. I spent a good 8 hours trying to upgrade to a newer version of DocProject and getting nowhere fast.
Edit: I should point out that I don't blame my problems on DocProject or Sandcastle themselves. The real problem is that I didn't really understand how it all worked so I wasted a lot of time trying stupid things that failed.
Today I finally gave up and ported the essential parts of the documentation to the latest version of Sandcastle and the Sandcastle Help File Builder. It’s a little worse for wear but it’s still readable.
Anyways, now the old MbUnit v2 documentation is online again here: MbUnit v2 Documentation.
Maybe we can port some of that content to the wiki or book and update it for MbUnit v3…
Edit: Posted new link to old MbUnit v2 wiki content.
Today we are releasing Gallio v3.1 Update 2. This releases fixes several problems on x64 platforms and includes support for Visual Studio 2010 and .Net Framework 4.0 Beta 2.
Download here: http://www.gallio.org/Downloads.aspx
Documentation here: http://www.gallio.org/Docs.aspx
Earlier release notes: v3.1 update 1, v3.1, all versions
I solved a minor mystery today.
Gallio internally uses the .Net XmlSerializer class to load plug-in metadata and save reports. You probably know all about XmlSerializer already, but maybe you don’t know just how badly using it can impact application start-up performance.
The heart of the issue is that XmlSerializer uses code generation to perform serialization and deserialization. Specifically, XmlSerializer generates some fresh C# code, compiles it out of process with csc.exe, then loads the resulting assembly. This work takes time. Seconds… feels like forever to a user. The generated assemblies are not cached across runs so there is a significant performance penalty on every launch.
There is a way to avoid this cost: pre-generate serializers assemblies and distribute them with the application.
To pre-generate a serializers assembly, you are supposed to use SGen.exe a bit like this:
SGen.exe /assembly:MyAssembly /type:MyRootXmlType
This command will emit MyAssembly.XmlSerializers.dll. All you need to do then is copy it next to MyAssembly.dll and you’re done. The cost of code generation is gone from every launch.
Two problems:
So let’s say you do all of this work and you run your program and you still see csc.exe starting up while your program runs.
To find out why, add the following to your application’s config file. (eg. MyApp.exe.config)
<configuration>
<system.diagnostics>
<switches>
<add name=”XmlSerialization.PregenEventLog” value=”1” />
</switches>
</system.diagnostics></configuration>
Then run your program again and look at the Windows Event Log. There should be some information in there to help you out. (Another thing to try is fuslogvw.exe.)
Here’s the message that I saw:
Pre-generated serializer ‘Gallio.XmlSerializers’ has expired. You need to re-generate serializer for ‘Gallio.Runtime.Extensibility.Schema.Cache’.
This message is telling me that I violated rule #2 above: the serializers assembly must always be kept in sync with its parent assembly!
Why? I’ve got fancy build scripts…
XmlSerializer verifies that the serializer assembly is in sync with its parent assembly by comparing the module version id of the parent assembly with an id that it previously cached when it generated the serializer assembly.
Let’s check whether the assemblies are in sync using Reflector.
Gallio.dll is the parent assembly, it has a module version id of “fbe34432-817a-46dd-832f-3e5bc679ecff”.
Gallio.XmlSerializers.dll is the serializers assembly, it has a special assembly-level attribute called XmlSerializerVersion that was emitted during code generation. It expects the parent assembly id to be “3c420916-3f3c-45cb-a79c-9a4bbc81014a".
Pedantic note: An assembly can have multiple modules each and each module has its own id. So the XmlSerializerVersion attribute actually contains a comma-delimited list of all module version ids of the parent asssembly sorted in increasing order.
Ok, so XmlSerializer thinks the pre-generated serializers assembly is out of sync because the module id is out of sync. That seems bananas because my build scripts always regenerate Gallio.XmlSerializers.dll whenever Gallio.dll is recompiled.
Well, my build scripts do one more thing: they run ILMerge on Gallio.dll to internalize certain dependencies that might otherwise get in the way of running unit tests. (Bad things would happen if your unit testing tool exposed a dependency on one version of Mono.Cecil.dll whereas your tests depended on a different version.)
When ILMerge internalizes dependencies, it has to regenerate the assembly. As a result, the new freshly merged Gallio.dll gets a brand new ModuleVersionId. Unfortunately we pre-generated Gallio.XmlSerializers.dll before running ILMerge so it is now out of sync.
All we need to do is regenerate Gallio.XmlSerializers.dll after the ILMerge. Problem solved.
Today we are releasing Gallio and MbUnit v3.1 Update 1. This is mainly a bug fix release with a few new little goodies.
Download here: http://www.gallio.org/Downloads.aspx
Documentation here: http://www.gallio.org/Docs.aspx
Earlier release notes: v3.1, all versions
Reminder: The Visual Studio Test Tools integration requires Visual Studio 2008 SP1 or Visual Studio 2010 to work. If you still don't see tests in the Test View then ensure that your project has the necessary ProjectTypeGuids and that the Gallio add-in is enabled in the Visual Studio Add-In Manager.
