Monday, April 20, 2009

Underfoot

Every now and then we talk about the "overhead" associated with implementing better practices.

For example, designing a stratified architecture requires some up-front planning to separate the tiers, identify roles, decide on the interfaces, establish conventions, and resolve cross-cutting concerns.  During implementation, there will almost certainly be additional code required to marshal messages across tiers, additional configuration required to glue them together and additional infrastructure required to host them.   Quite probably the team will also need to learn new skills and adopt other ancillary design/implementation practices at the same time to make it all work out.  This effort all takes time, effort, and money.

Likewise adopting TDD/BDD/DbE, using continuous integration, leveraging IoC, adopting one-way asynchronous message passing (instead of RPC), applying command-query separation, and "going SOLID" all have one-time and recurring costs... (and benefits of course.)

I think "overhead" is a misleading label for these costs.  It presumes that the costs are excess terms that could be eliminated through greater efficiency.  However these costs in fact purchase greater efficiency.  They are an investment in laying a foundation for future.  They pad the space under our feet not the space over our heads.  They let us reach greater heights!

 

Let's try this paragraph with underfoot instead of overhead.

Building this proposed multi-tier platform will have an initial 20% underfoot while we retool and lay the foundation.  Once that is done we will be 50% more efficient though we will have to spend an additional recurring 5% underfoot to keep the axles greased.

Which do you prefer?

Thursday, April 2, 2009

Announcing Gallio and MbUnit v3.0.6 Update 1.

Today we are releasing Gallio and MbUnit v3.0.6 Update 1.  This release fixes a regression in the ReSharper support and adds a couple of new features.

Highlights:

  • Added support for TestDriven.Net category filters.
  • Added support for more powerful inclusion/exclusion test filter expressions.
  • Fixed ReSharper v3.1, v4.0 and v4.1 hangs.
  • Increased NCover v1.5.8 timeout.
  • Adopted new assembly version numbering scheme.

Download here: http://www.gallio.org/Downloads.aspx

Documentation here: http://www.gallio.org/Docs.aspx

Earlier release notes: v3.0.6, all versions

Changes

Inclusion / Exclusion Filters

Gallio provides a mechanism for filtering the set of tests to run.  This is typically used to select some subset, for example to run all tests except for integration tests.

However, the filter grammar used in previous versions could not correctly express an exclusion constraint.  Several people noted that filters like "not Category: Integration" did not in fact work due to a logical error introduced by a change in how Gallio applies filters.  Oops.

In this update we have added a more concrete concept of inclusion and exclusion filters by extending the filter syntax with include and exclude clauses.

Here are a few examples using Gallio.Echo.  The same filter syntax is used universally throughout Gallio so you can use them with NAnt, MSBuild, and PowerShell as well.

  • Gallio.Echo.exe MyTestAssembly.dll "/f:exclude Category: Integration"
  • Gallio.Echo.exe MyTestAssembly.dll "/f:include Type: MyFixture"
    (here the "include" is optional)
  • Gallio.Echo.exe MyTestAssembly.dll "/f:AuthorName: Jeff and Category: SmokeTests exclude Type: BrokenFixture, AnotherBrokenFixture exclude Importance: NoOneReallyCaresAbout"

TestDriven.Net Category Filters

As of v2.17, TestDriven.Net has a Visual Studio preferences page that allows the user to configure a couple of different settings.

In this update of Gallio we have added support for the TestDriven.Net category filter options.  This feature was absent in previous releases because we did not then have a way to express exclusion filters properly.

image

New Assembly Version Numbering Scheme

Plug-in authors rejoice!  I've changed how assembly version numbers are constructed in a subtle but important way.

Previously the assembly version number would change after each build.  As a result, code linked to Gallio v3.0.6.749 wouldn't work with Gallio v3.0.6.762 despite there being no significant API changes between those versions.

Now the last component of the assembly version number is always 0.  This way there can be no assembly version conflicts within a given release branch.  However, we still need a way to distinguish different builds.  So the file version number contains the full version number.

For example, in this release the version numbers are set as follows:

  • Assembly version: v3.0.6.0.
  • File version: v3.0.6.763.

As a result, you should no longer need to recompile your plug-ins except to make them work with a new release branch.

Aside: There is now a separate branch for Gallio and MbUnit v3.0.6 in case you want to check out the code and play with it.  This will also help us keep maintenance updates on track.

Wednesday, April 1, 2009

Gallio is not Alpha Anymore (And Has Not Been Since Last Year)

I sometimes receive questions from people about when Gallio and MbUnit v3 will be out of alpha.

The answer is that they have been out of alpha officially since the v3.0.4 release last year.  The current releases should be considered stable and final.  Sure, there are still bugs to be fixed and enhancements yet to be made but we do consider these releases quite acceptable for use in production systems.

So there you go.  :-)

Technorati Tags: ,

How to enable or disable Visual Studio Team Test integration

Gallio provides integration with Visual Studio Team Test so that you can run your tests using the built-in Visual Studio Test View and other tools.

However to make this work, the project file must be configured with an extra "ProjectTypeGuid" that that informs Visual Studio that the project should be considered a Test Project.  By adding or removing this element you can enable or disable support for Visual Studio Team Test.

Caveats

  • We do not install the Visual Studio Team Test integration by default anymore beginning with v3.0.6 if you perform a "Typical" install.  You will need to perform a "Complete" installation or ensure that the "Visual Studio Team System Test Runner" component under "Test Runners" is installed as part of a "Custom" installation.
  • The built-in MbUnit project templates already contain the appropriate ProjectTypeGuids, so if you create a new project that way then it should just work out of the box.
  • Some editions of Visual Studio lack support for Test Projects and will refuse to open up any project that contains the special guid!  In this case you may have to remove the offending guid so that you can work with your projects.
  • The Visual Studio Team Test integration is very rough.  It has poor performance due to several issues including the fact that Visual Studio wants to populate its list of tests in the foreground after compiling a test project.  So the GUI may hang for some time after each build if the Test View is open and you have a big test project.  I really hope they fix this in Visual Studio 2010.
  • At this time, the integration only works with MbUnit v2, MbUnit v3 and xUnit.net.  Unfotunately csUnit and NUnit do not support the use of an abstract reflection policy so they are not supported by this feature.  However, there does exist another Visual Studio Team Test add-in for NUnit.

In all honesty, I would recommend using TestDriven.Net or ReSharper to run your tests inside Visual Studio instead of using the Test View.

Editing Project Files

Visual Studio uses a magic "ProjectTypeGuid" to determine whether it should consider a given project to be a Test Project.  Only Test Projects are considered when populating the Test View.

And the magic guid is... {3AC096D0-A1C2-E12C-1390-A8335801FDAB}

Here's how to edit the files:

1. Open up the *.csproj or *.vbproj file in a text editor.  It's just XML (actually an MSBuild build script).

2. Within the first <PropertyGroup> element at the top of the file, look for an existing <ProjectTypeGuids> element.

2a. If you found the <ProjectTypeGuids> element, then add the aforementioned Guid to the list within if not already present.  Be sure to separate the guids with semicolons.

2b. If you did not find a <ProjectTypeGuids> element, then copy and paste in the appropriate element below.

  • C# Projects: *.csproj

    <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};
    {FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

  • Visual Basic Projects: *.vbproj

    <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};
    {F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

3. Save the project file.

4. Reload the project in Visual Studio.

5. Open up the Test View.  The tests in your project should now be populated.

It's Too Slow... How Do I Turn It Off?

If you read the caveats above, I warned you... :-)

There are three ways to turn this off.  Any one of them will suffice.

1. Remove the ProjectTypeGuid just added above then reload the project.

2. Disable the Gallio add-in using the Visual Studio Add-In Manager.  Contrary to what some people expect, this will not in any way affect the ReSharper integration.

3. Uninstall the "Visual Studio Team System Test Runner" component.  You don't need to uninstall all of Gallio for this.  Use the Add/Remove programs control panel to Change the set of installed features and look under the "Test Runners" part of the tree.

What The Future Holds...

I do have a hack in mind for improving performance by populating the Visual Studio test list asynchronously although it would run contrary to the stated API contracts.  So if the performance is a problem for you but you really really want to use the Visual Studio Team Test controls, please let me know and I'll raise the priority of this task.  Also I could use a volunteer to help make the integration more compelling overall.

Also, some time ago I started working on a different native Visual Studio add-in for Gallio called Sail.  It's not ready yet but something I've been thinking about again lately...  If you'd like to help, let me know.

Technorati Tags: ,

Debugging Tests with Gallio

Here's the rundown of how to debug.

Icarus: Click the new "Debug" button in v3.0.6.

Visual Studio Team Test: Select the tests in the Test View and click Debug.  (This assumes that your test project has the appropriate ProjectTypeGuid to be recognized by Visual Studio as a Test Project.  See next post.)

TestDriven.Net: Right-click on a test or fixture and select "Test With... Debugger"

ReSharper: Select a test in the editor or test session view and use the Debug drop-down or toolbar button as appropriate.

Echo from command-line: Run tests with /debug argument. This will automatically attach to Visual Studio (or launch it if it is not running).

PowerShell: Use the "/DebugTests" switch for similar behavior to Echo.

NAnt: Add the debug="true" attribute for similar behavior to Echo.

MSBuild: Add the Debug="true" attribute for similar behavior to Echo.

It is no longer necessary to set the test runner type to IsolatedAppDomain if you use any of the above methods to run tests with the debugger.

On the other hand if you are attempting to attach to an existing test run of Gallio that was not started using any of the debugging mechanisms described above and the test runner type is IsolatedProcess (the default) be sure to attach to the Gallio.Host.exe or Gallio.Host.x86.exe process instead of Gallio.Icarus.exe or whatever.