Wednesday, January 30, 2008

An MSTest API Rant.

I just happened to look at the MSTest API today trying to get a feel for whether there were any nicer points of integration other than wrapping MSTest.exe.

Internals All The Way Up

Along the way I happened to notice that the Microsoft.VisualStudio.QualityTools.Common assembly has 166 InternalsVisibleTo attributes on it!

All of the useful API for loading and running tests is internal.  It's possible to create new test types but it does not seem possible to manipulate the test execution pipeline at all.  The good parts of the internal Tmi class are locked away and inaccessible through the public ITmi interface.  I don't see any other API that provides access even indirectly to the LoadTests methods except for the console application.  Disappointing.

Alright Microsoft, I get the message: Keep out.  Fine!

Avoid MSTest

If you are currently considering adopting MSTest internally, I strongly recommend you choose more open/extensible tools instead.  Microsoft is deliberately curtailing the ability of third-party tools integrators to extend their offerings in all but the most trivial ways.  This will ultimately limit the kind of support you will be able to receive long-term from the developer community.

Vendor lock-in is not good for anyone.


Chad Myers said...

I'm not so sure it's as malicious as much as it is ignorant. MS has a penchant for making everything internal and private in their APIs and I think it's mostly to prevent people from hurting themselves. This is a bad design choice. There are other ways of protecting people from hurting themselves without blocking off a bunch of useful functionality.

I was browsing through Reflector at some ASP.NET stuff (including the new MVC stuff) and was frustrated with the number of internal, sealed, and private members and classes.

I wrote this blog post which is related to what you're talking about, you may find it interesting:

Callum Hibbert said...

Hi there,

It can be done:

Link in the post to my CodePlex project hosting some extensions to MSTest. More contributors welcome.



Jeff Brown said...


Neat trick using ContextBoundObject.

However, your post does not address the concern I raised here. I was specifically complaining about the fact that it is not possible to host MSTest inside of another tool.

The only (legal) option seems to be running MSTest.exe which is quite unsatisfactory if we want to do things like enumerate tests for presentation in a GUI.

Obviously there are possible workarounds using deep reflection, but I'm worried I'll get slammed by Microsoft for circumventing deliberate limitations of the product.

This leaves users of MSTest locked into the framework since no one can deeply integrate MSTest into a third-party product short of reimplementing and emulating it.