Sunday, September 30, 2007

Reading XML documentation comments

So today I decided to have a little fun and see what it would take to read XML documentation comments associated with code members in assemblies that are currently loaded.  It took a little doing to get it completely correct but it's done and fully tested now (including gnarly cases involving generic parameters).

There are a few interesting uses for performing reflection on XML documentation from within a running application.  For example here is a test case that reads some of its data from its own XML comments.  It's pretty nifty.  I'm not sure I would actually recommend this usage pattern because I find embedded resources and custom attributes are sufficiently effective and robust.

What I have in mind has more to do with enhancing MbUnit Gallio reports and GUI content.  The XML documentation simply becomes another piece of metadata that you can see or manipulate.  But I won't put it past others to come up with other interesting uses of this feature.

Gotchas: The MSDN documentation regarding how member names are computed for XML documentation comments is incomplete and incorrect -- particularly as regards generic type arguments.  I think I've got it nailed though.

Friday, September 28, 2007

The runaway product

A couple of days ago I read The Mythical Business Layer on Worse than Failure.

There's nothing at all shocking in there.  However, the article did provoke me to think about a few cases of over-engineering I've seen (and participated in) where features were added to the underlying abstractions that were never exposed to end users due to "limitations in the UI."  *ahem*  Those "limitations" existed because the core development team lost sight of the product and introduced excess flexibility complexity in the system that was completely irrelevant to the business.  The UI team had a more stringent specification to work from so they could not go quite so far astray.

A story that sucks

I think one of the ways this happens goes a little bit like this:

Business Analyst: Our Vacuum product is doing great.  We've got 13% market penetration in the Light-Duty Household market segment but only 2% of the Families With Children segment.  I think we can do better than that.
Product Manager: How about adding a feature to provide more power for cleaning up tough spots?
Marketing Manager: Yeah, we could totally sell that to those people.  Let's call it the Ultra-Suck option.
Business Analyst: I think that might appeal.  Let me go crunch some numbers...

Time passes...

Project Manager: Ok, so we'll be implementing the Ultra-Suck option for our Vacuum product.  Here's how it will look.  We'll add this button over here under the handle that provides a 10% power boost.  We'll also beef up the thermal cut-offs on the motor in case of overheating.
Engineering 1: Ho hum.
Project Manager: This is a cool feature!  Heck, maybe in the next release we can add multiple boost settings and an adaptive power meter...
Engineering 2: Okay.
Engineering 1: !!

Time passes...

Engineer 1: I'm bored.  Another stupid Vacuum feature.  The business guys just don't understand what people want.
Engineer 2: Heh.  But did you hear about that adaptive power meter?  That sounds pretty cool.
Engineer 1: Yeah.  I guess since we should plan for it then.  We'll need a way to measure the current load and scale the output power to multiple levels.
Engineer 2: Maybe.  But there's no point trying to measure the load now if we aren't adaptively tuning the output power.
Engineer 1: Ok.  But it's pretty easy to add support for scaling the power output.  Right now we need need 100% and 110%, but later we'll probably want a range from 50% to 120%.  The circuit won't be that much different.  We can just hardwire the Ultra-Suck button to the 110% level.
Engineer 2: I'm not so sure...

Time passes...

Project Manager: Is the Ultra-Suck button done?
Engineer 1: No.  I'm having trouble with the circuit that selects a range of output power levels.
Project Manager: Don't we just need two levels?
Engineer 1: Yeah but later on we might need more for the adaptive power meter feature.
Project Manager: But right now we only need two.  Get it done.

Eventually the project schedule slips due to "unforeseen difficulties" in the implementation.  The fanciful features of the anticipated future model may never come to be.  Meanwhile the product has acquired a lot of added complexity and has shown no net gain from all of this work.

The moral

While this story is somewhat contrived in the world of mass-produced physical consumer goods where unit production costs are very high, it happens all too often in software.  Somewhere along the way, the original narrowly focused business objective is lost.  "Just a little tweak here, a little tweak there."  But each tweak raises the cost of development, testing and maintenance.

That's not good!  There is zero business value to any of this extra work until it becomes relevant to the product.

Aside: I'm lying a bit when I say there is zero business value.  After all, the extra complexity may help to provide motivation to get things done.  Good people are hard to replace so there should be some allowance for keeping them happy.  However, I strongly prefer reserving 20% time for such projects in lieu of allowing the core products to become unnecessarily complicated.

kick it on

Tuesday, September 25, 2007

How to delete Google Gears offline data

Google Reader jammed on me in offline mode today.  Try as I might, it wouldn't switch back to online mode even after I let it run for an hour apparently "synchronizing".  Restarting the browser and clearing caches didn't help.

After a couple of Google searches I did not find any information regarding how to manually clear out Google Gears offline storage.  So here's one way... (probably not the best way)

  • First close your browser.
  • You'll find the Google Gears offline data store for Firefox inside your Firefox profile directory.  On my machine, that folder is "C:\Documents and Settings\Jeff\Local Settings\Application Data\Mozilla\Firefox\Profiles\wb2qohxr.default\Google Gears for Firefox".
  • The offline data store contains a couple bookkeeping files ("localserver.db" and "permissions.db") and individual directories for each domain name with offline storage.  Just delete the appropriate directory.  In my case it was "".
  • Restart your browser and away you go...

Works fine now.

Saturday, September 22, 2007

Release early, release often.

The reaction to James' Newkirk's announcement of XUnit (looks interesting!) has reminded me of a very important principle: Release early, release often.

Sometime last year I created some spiffy data-driven extensions for MbUnit that took it far beyond RowTest.  Actually, MbUnit supports a whole suite of data-driven testing features but there were still a few things I wanted to have that didn't exist.

Keep it simple

One of goals was to provide simplified support for pulling data from files.  I wanted a simple declarative form that would load data from CSV or Xml files with a minimum of fuss.  Ideally it would map values based on the actual field, property or parameter names in the tests so that very little in the way of data binding glue would be required in the common case.  It should also be easy to define new data sources (and it was...).

Here's an example using CSV:

public class AuthenticationTest
    public string ServerName;

    public string ServerPort;

    public void Login(string userName, string password, bool isValid)
        Server server = new Server(ServerName, ServerPort);
        bool success = server.Login(userName, password);
        Assert.AreEqual(isValid, success);
Where we have two embedded resources.
ServerName, ServerPort, Owner, Domain, 9999, operations, test
UserName, Password, IsValid
jeff, letmein, true
jeff, knockknock, false
jim, letmein, false
jim, knockknock, true
"", "", false

Execution of the test is pretty straightforward.  The test fixture's ServerName and Password fields are initialized with the contents of the matching columns of the server-config.csv file.  Then the Login test method gets run once for each row in credentials.csv.  Data binding occurs automatically including conversions from strings to other types (like booleans).


A second goal was to simplify retargeting of tests to multiple environments without recompilation or manual munging of files.  Like most enterprises, we have separate development, test, staging and production environments.  The configuration of each environment varies in a few particulars such as in domain names and in the pre-populated data set.

To capture environment-based configuration, I created a new batch of attributes like this:

public string ServerName;
Each new file now specifies configuration for a particular environment.

The environment to target is selected at runtime either by popping up a dialog box for the user to choose one (the selection is saved for the duration of the test session) or by grabbing it from an environment variable.  (Remark: MbUnit Gallio will include support for passing external values to tests directly from the command-line or from the GUI test runner to handle just this sort of case with less hackery.)

The moral of the story...

I thought this API was pretty cute so I published the (non-proprietary) code to the MbUnit mailing list.  However, since I was already thinking about how to perfect the feature in MbUnit Gallio, I refrained from polishing the code and contributing it back to the MbUnit v2 framework itself.  That was a mistake.

If I had polished the code, other people would be using it now and no doubt I'd have received plenty of useful feedback to apply towards MbUnit Gallio.  Oh well.  So here is the code for these simple data-driven extensions for MbUnit v2, completely unretouched.

With that in mind, we're gearing up to cut the first alpha release of MbUnit Gallio soon...  Release early, release often.

kick it on

Wednesday, September 19, 2007


Validation is when someone else takes your vaguely sketched idea, embraces it, simplifies it, improves it, and realizes it better than you could ever have even imagined on your own.

Sunday, September 16, 2007

Shedding fallacies.

Work, health, or happiness: pick two three.

It's time...

Saturday, September 15, 2007

Tests that last.

Roy Osherove asks on his blog if you have ever written throw away tests (TATs) and whether you have since learned to write tests that last (TTLs).

Sure, I have written TATs and learned from the mistake.  Now I mostly write TTLs and they stay that way because I maintain them.  Sadly there have also been a couple of situations where I've transferred code ownership to others who did not practice unit testing and allowed the test suites to rot... *sigh*

My approach to writing TTLs is pretty simple.

Rule 1: Don't overcommit to your unit testing effort early on.

Ok...  Before the Agile folks put my head on a platter please let me clarify.

I am often engaged in designing architectural concerns such as APIs, processes, conventions and infrastructure.  These concerns all have associated contracts.  I wouldn't sign a contract in blood without careful negotiation first; would you?  You would do more research and you would haggle.  So don't overcommit testing resources to a hastily sketched out system that might change tomorrow.

Architectural concerns have a very high initial design burden.  The cost of changing your mind can be huge!  On the other hand, big design up front is risky.  This seems almost at odds with Agile development.  It isn't.

Ideally we want to partition the design process so that we can obtain confirmation of the soundness of core ideas incrementally.  However, because the core ideas all need to work together smoothly we must reserve the right to change our minds.  To get the feedback we need, it helps to sketch out sample clients that exercise a representative subset of the current prototype.  It's even better if the sample clients are executable because it provides more concrete evidence that we're on the right track.  Oh look, we just wrote integration tests!

What about unit tests for a new API?  Well, eventually we'll want heaps of those.  But early on, they aren't all that helpful.  Moreover since the design is still rather volatile, you run a risk of generating tons of TATs.  This is especially true for white-box tests.  Defer writing expensive and risky tests until you're confident they won't need be thrown away.  (But do remember to write them as soon as you're sure! In many situations that just means waiting a couple hours until you're done thrashing with ReSharper.)

Rule 1 (corollary): Write integration tests before most unit tests.

Unit tests are usually very tightly coupled to the code you're writing.  Volatility in the code under test can translate into major headaches and wasted effort.  In particular, reserve white box tests for last.

Rule 2: Don't aim for 100% code coverage of your first draft.

It's almost never right the first time.  Wait until well after you've first integrated a component into the system and has been thoroughly proven out.  Code coverage metrics are good for gaining more confidence that your implementation is correct; however, they are not of much use figuring out whether your design satisfies the requirements.  You'll probably end up refactoring your first draft beyond recognition so don't waste time checking it too deeply.

Rule 2 (corollary): Don't write white box tests until the implementation has stabilized.

White box tests are useful.  Don't let them turn into TATs because you wrote them prematurely.

Rule 3: Write your tests clearly.

All of the usual tricks apply:

  • Test code should be soluble.
  • UseVeryClearNamesThatMightEndUpBeingALittleLong.
  • Don't verify more than one behavior in each test.  (Try not to have too many assertions in a single test.)
  • Limit the usage of mock objects to generating stubs and setting up relatively straightforward expectations for dependencies.
  • Clearly document how the test harness is being set up and what interactions are taking place.
  • Include detailed messages in your assertions to describe expected behavior.
  • Don't do anything surprising in your tests.
  • Avoid introducing side-effects across test runs.
  • Try to isolate the tests as much as possible from one another.
  • Don't depend on too many white box implementation details.  Do cover the corner cases of your algorithms but try not to depend on exactly how the computation will proceed or in what order certain intermediate results will be produced.
  • Avoid hardcoding exact string equality comparisons except in trivial cases.  If you are calling a function that formats a text message containing some information then there can be a fair amount of looseness in the formatting.  Look for the presence of expected information by searching for matching substrings.
  • Avoid hardcoding hidden policy configuration that might change.  For example, today an anonymous email service might only let you send 5 emails per hour before a spam killing rule is activated; tomorrow it might be 2.  Instead of hardcoding "5" into your test, try to obtain the constant from the same source as the actual implementation.

Rule 4: Invent your own rules!

Conventions are good.  Learn what works and apply that insight routinely!  Then you'll be sure to get tests that *really* last.

MbUnit Gallio, for example

I'm currently architecting MbUnit Gallio incrementally.  If you were to download the source now, you'd notice there aren't all that many unit tests.  Two of the tests currently fail!  Have I lost my mind?  Shouldn't I develop a test framework using TDD?  Perhaps...

The problem is that I ran out of storage space in my brain.  There are too many things going on inside Gallio for me to have any confidence getting it right the first time as a pure gedankenexperiment.  Pretty diagrams and written notes were helpful at the high level but not so much below that.  I needed to write actual code (or at least design interfaces) so that I could observe the interplay between the various high level concepts and adjust them accordingly.  It's been very rewarding.  As the design process progresses, I've managed to isolate and solidify several components to enable others to work in parallel while I iterate over other fuzzy details.  It's getting there!

I've written integration tests at several different levels of integration within the system.  There's enough meat to get confidence that the design is moving in the right direction.  Those integration test failures I mentioned earlier were fully expected because parts of the system are still stubbed out!  Meanwhile I've been trying to progressively write unit tests for components that have solidified.  But here I've encountered the limits of my ability.  I can't do architecture, design, code, tests and project coordination all at the same time.  So as a team we shall work together to increase our testing velocity.  (Which leaves me feeling irrationally guilty because I prefer to take personal responsibility for thoroughly testing everything that I code.  Actually, I need to learn how to delegate more of the detailed design and coding work. Despite being intensely interested in these tasks I need to release my investment in them so that I can focus on maintaining the velocity of the project as a whole.  I am learning to scale up to larger projects.  It's not easy.)

I'm confident that almost all of these tests will last because we are working to understand the implications of the system that we are designing and building.  And we care enough to get it right (eventually)!

Friday, September 14, 2007

Pile it on...

I can't remember who sent me the link to Ralf's blog but thank you very much! This stuff is fantastic!

I think I might just have to go back and read through his entire history...  His writing style is incredibly lucid and compelling.

Being a protagonist

I had a thoroughly enjoyable conversation with my boss today (hi Ray!) wherein he asked me what I felt best characterised the sentiment of my blog.  Without hesitation, I answered: "Antagonistic!"  And... I was momentarily taken aback.  How valuable are my rants?  Is that really what I want to share with the world?


At the risk of starting another rant, I would like to share my thoughts from this evening on being a protagonist...

What I want

I am a very passionate individual.  More than anything I want to be a force for goodness, balance and positive change.  Sure, part of achieving that goal requires thinking critically about the world and recognizing opportunity and means for improvement.  But it is a very small part.

I believe that the larger part of the challenge is to play an active role in making a difference.  Sometimes that means playing a minor supportive character assisting others who are engaged in a great epic.  Other times it means playing the humble protagonist who stands up and gathers others to uphold what is right and make the world more beautiful.  (Cue the Lord of the Rings soundtrack.)

Nobody likes a backseat driver

I could natter on endlessly about all sorts of software engineering p-r-o-b-l-e-m-s. *ahem*  Who asked for my advice anyways?

It makes no sense.

If I am not a protagonist and I am not acting like kindly Tom Bombadil to support others on their own quest, then what is my role?  There are two possible answers.  I could be a nobody whose only purpose is to contribute local colour to the story.  Or... I could be an antagonist who is distracting or perhaps even actively interfering with the major players.

Neither option is very satisfying!

Leadership, not Wizardry

Gandalf was not a good leader.  Sure, he set the story in motion but then he foolishly abandoned those he had rousted at their great peril.  In his arrogance, he assumed his own objectives were of such great importance that he neglected those he had gathered so that he could personally undertake his mission.  Consequently he very nearly failed to achieve his role in helping to fulfill their Great Purpose.  Thinking great thoughts is not enough!

Ok, so Gandalf was an idiot.  Not the only one...

I confess I have acted like a Wizard too.  I have falsely assumed the countenance of Great Responsibility.  I have meddled.  I have not dealt the cards fairly among those with whom I have shared responsibility.  I have upheld ludicrous ideals that ultimately accomplished nothing except to frustrate others who were better grounded in reality.

Pure foolishness.

Being a protagonist

I am working on MbUnit Gallio with a great team of people.  I add value to the team with architectural insight, programming skills and friendship.  However, I fear that I have also held us all back while I sat in a corner settling "complicated design issues."  That is not leadership; it is arrogant bullshit!  I am certainly not the only visionary.  I am not the only one who can finesse details.  Besides, sometimes my methods are crude and clumsy.  I make mistakes.

That said, I believe we are building a great product!  We are innovating.  We are discovering surprisingly natural solutions to nagging problems.  We are deeply invested in quality.  We care.

I am a protagonist but not the only one.  I must do my part to help keep the team on track and moving forward at a sustainable velocity despite my own weaknesses.  My personal opinions and ideals matter only insofar as they enable us to achieve our common goals.  We share responsibility in different capacities but in equal measure.

We are in this together.

P.S. For the record, I did enjoy the Lord of the Rings... even Tom Bombadil.

Thursday, September 13, 2007

Soluble Code

I had a discussion with someone today that reminded me of this post by Scott Bellware. Go read it: Soluble Code.

Specifically we were discussing design conventions and how they help (or hinder) the people that follow them (or don't). Once again I had to make the point that code is read far more often than it is written. It's important to have solid conventions in place so that we can understand each other and work towards common goals. Otherwise the end product will have all of the permanence of the Tower of Babel.

I've noticed a strange cognitive dissonance with some programmers. On the one hand they claim to value code clarity. But on the other hand when they get stuck they labour under some misapprehension that it really won't matter what goes on behind the curtain because no one else will ever see it; or so they believe. These seem to be the same programmers who have difficulty grasping the Law of Leaky Abstractions and who cannot be trusted to design a sensible API with more than 3 functions in it because they'll try to make it "simpler" by squishing all of the behavior together with as many "helpers" as possible.

Trying to hide ugly and complex code is just a sure-fire way to ensure that you're stuck with it for life because no one else on the team will be interested in maintaining it. So don't do it. It makes me sad because I have only occasionally been successful in reaching out to these programmers and teaching them to think more critically about what they're building.

Of course I too suffer from occasional delusions. I like to believe the fiction that insoluble code will eventually get ejected from the system (like dietary fiber). But it does seem to take quite awhile to digest sometimes...

P.S. Oh yes, and No Broken Windows!

Read more code!

By now most of you have probably read on SlashDot that QNX has opened part of the QNX Realtime OS source code.

Naturally I'm jumping on the opportunity to read more code written by others.  I find it fascinating, particularly with mature projects.  For example, here's the QNX kernel entry code for x86 (written in assembler of course).

Fascinating!  I can read it but I don't understand half of what it's doing.  What more could I expect after looking over a few source modules for less than half an hour?  Too little context.  But oh so much fun!

When I was a kid a several time TI user group members gave me boxes of old TI stuff including books like TI99/4A Intern, schematics, maintenance manuals and reverse-engineered specifications.  I can't count how many hours I spent absorbing all of that knowledge.  I was happy as a clam!  And I eventually put much of it to good use in various software and hardware hacks...

Here's some advice I often offer to programmers:  Read as much code written by other people as you possibly can.  Think critically about what it does, the problems it solves, how it was written, how it was tested, the team that created it, and the context in which it was born.  Use that insight to improve your own work.   (Besides, if you don't enjoy reading code, how are you going to survive doing it all day long?)

Corollary: Also read lots of books about code...

Wednesday, September 12, 2007

Semi-modal offline support

My life on the go

When I'm not working from home I commute to the office on public transit. On the way, I read lots of books (ok, and other fun stuff too. I also use my laptop to read blogs (too many to list), check email and write code.

Aside: Installing Visual Studio on my laptop was probably the best thing I have ever done for my family life. If I can take my code with me then I can keep hammering away at bugs while I'm communiting instead of hanging around at the office endlessly. I'd rather be sitting on my patio anyways with the WiFi.

The problem is that despite my GPRS connection (yeah, I wish I'd waited and gotten the EVDO model) I do go underground four times during my commute. So connectivity is a bit spotty. Consequently I'm in the habit of loading up FireFox with tons of tabs so that I have plenty of reading material for the whole trip.

Google Reader

Recently I switched to using Google Reader for reading RSS feeds. (Previous I had been aggregating the feeds using LiveJournal.) Google Reader now has a nifty offline support feature. When you switch it to offline mode it will download "up to 2000 items" and save them to your local disk. But there are problems...

The items have to be downloaded all at once. If it takes too long and you lose your connection or cancel the download, tough bananas. (Ironically, I also found the installer for the offline support plugin to be very flaky. If you're not online when you run it, it hangs on startup and eventually errors out.)

In search of graceful degradation

So what I'm going to pick on today is the notion that offline support requires a discrete mode.

Internet connectivity is always at least a little unreliable. Sure, most of the time your web application can issue an XML/HTTP request to some service provider and get back a response at most a few tenths of a second. But sometimes it will take much longer or it will fail. What should the application do?

So first off, forget about the statelessness of the web: that was only true back when we were serving up plain HTML documents. Today's web applications truly exhibit distributed state. The client has some of it and the server has the rest. Often client development is simplified by minimizing the amount of state it retains locally at the cost of responsiveness and robustness.

If you think this is contrived, think about all of the forms-based web applications you've used that are constantly doing round-trip calls to the server to submit changes made to the client's state to the server and figure out what to do next. Pretty horrible stuff! AJAX smooths the experience somewhat but the same kinds of things are going on underneath. It's all client-server.

So what happens when I pull the plug on my DSL connection while I'm writing a blog post. Well, Blogger periodically saves drafts to the server for backup purposes. But I can still make edits to the local state. Yet if my browser crashes for some reason, I'll lose them all because none of that state is persistent. (I'm speaking from experience here.)

Some years ago I read A Fire Upon the Deep. Vinge's universe places fundamental limits on the level of technological sophistication of different regions of the galaxy. At one point in the story there's a description of ships from the Beyond gracefully degrading as they enter the Slow Zone. Communication systems didn't fail outright but they compensated for narrowing bandwidth by reducing the quality. IIRC they adapted even to the point of transcoding conversations with talking heads and text-to-speech. That's the ideal...

The many shades of connectedness

Being offline means that there exists distributed state in a system that cannot be accessed or reconciled for an indefinite period. In between online and offline are many degrees of communication (un)reliability. Each degree of unreliability requires a corresponding measure of aggressiveness to tolerate. Here's a scale:

  • Perfect: The application's, reliability, bandwidth and latency requirements are fully met. Packets are never dropped. The plug is never pulled. You could run a Telnet session over UDP with complete confidence. This world does not exist!
  • Typical: There are occasional problems but they are easily compensated for by routine error correction protocols with at most minor glitches.
  • Unreliable: It becomes harder to mask errors. An application can still generally assume the problems are transient but it may become practically unusable unless it does at least some local caching.
  • Sporadic: User state will be lost or left dangling in mid-air unless the application compensates with local state management.
  • Offline: I pulled the plug! The application is on its own with whatever client-side wits it has left. In many cases, this is the kiss of death for an application and the user has to resort to typing up notes in Notepad until connectivity is restored.

A web application is ultimately at the mercy of the network. It can be enhanced to support disconnected operation, as with Google Reader, but there are significant complications. Just the same, why treat disconnected access as a special mode? Why not think of it as a very aggressive caching strategy with robust client-side storage? And if that's the case, why not enable it all of the time?

In other words, why doesn't Google Reader always cache feed items to my local disk whenever it can. I could certainly tell it to prefetch more data because I'm going offline for an extended time. Moreover if the synchronization process fails part way through, I should still be able to access whatever content I do have available. Why does offline support need to be modal? Why does Google Reader need to behave any differently from a desktop RSS client when disconnected from the network?

Thinking about web-enabled applications

I used the word enabled in web-enabled to effect a fundamental change of perspective. The web today is an excellent content delivery platform. Just the same, it is suboptimized for application delivery. On the web today, application and content are tightly enmeshed with one another but in reality they're very different. When I go offline, I don't mind if some of my content becomes inaccessible but I would still like to be able to use my web-enabled application to manipulate local content. That can't work so long as the application actually lives in a faraway server farm and I'm just interacting with served up views of its state when I'm online.

Ideally I want the web to deliver rich client applications whose code gets been cached on my local machine. I want applications that blur the line between the desktop and the web (including conforming to some sensible UI conventions). These applications would be enabled by the web and tightly bound to network services but they're not shackled to it. They can stand on their own albeit with reduced functionality. Ultra-portable applications: they run anywhere.

That's what I want!

Doom, death and destruction!

So who will be first to provide a compelling solution for lightweight rich-client application delivery over the web? Google Gears, Apollo, Silverlight or someone else?

Dynamic method interception with Mono.Cecil

For a couple months I've had an idea jangling in my head to use Mono.Cecil to instrument libraries to enable runtime dynamic interception of any method call (not just virtual ones). For example, this ability would be useful for extending Rhino.Mocks to support mocking static methods. It could also be used for tracing into unmodifiable code.

Of course, generally when you want to do these things you're doing something wrong. Let's set aside those objects for the moment. I've been curious to see whether the principle was sound and how much work it would be.

I tried hard to suppress it and focus on more important things. I was pretty successful at it. But I broke down over the weekend while I was writing a wee little code generator to help keep MbUnit Gallio Assertions all nice and consistent (I really wish C# had hygienic macros!). All that code gen on the brain was driving me to distraction. So I gave it a shot.

The results

1. The principle is sound.
2. It took about two days of heavy-duty hacking to get a proof of concept (while taking care of a bunch of other business). Pretty much what I expected but more than I'd hoped.

I've checked in code that uses Mono.Cecil to perform runtime interception in either of two different ways:
1. Statically instrumenting the assembly to include interception code that can be hotwired on demand. (easy)
2. Loading the assembly into the dynamic assembly context and patching method bodies on demand using MethodRental. (hard)

Moving forward

Generally the profiler API is better suited for dynamic interception but there are interesting scenarios where we can't use it (like in production code). So I can imagine there would be significant value to various projects if a variety of interception strategies were offered under a common simplified instrumentation interface. I'm sure the .Net AOP enthusiasts would put it to good use.

It was a good experiment but not immediately useful. (That's what hobbies are for...)

In any case, I don't plan on doing much of anything with this code anytime soon since I've got tons of other stuff on the go. Please let me know if you find this code of use or are interested in contributing to its continued development.

Monday, September 10, 2007

Multithreaded testing

Karl Seguin wants Better multithreading support needed in .NET. Yeah, I miss Java's thread-safe collections and fancy synchronization primitives too. I sometimes feel like my working vocabulary is being stunted for lack of a proper built-in CyclicBarrier class. I'm somewhat tired of reimplementing these kinds of things.

But I'd like to focus on the last thing Karl said:

At the top of my wish list though is support for multithreading testing. Multithreaded applications might be hard to write, but they are a nightmare to test. It's easy enough to run multiple threads over your code, but I haven't yet seen (which doesn't mean it doesn't exist), something that lets you transparently test specific race and deadlock conditions. Was a waiting thread properly signaled? Was a deadlock properly resolved? So far my approach has been simply to put huge loads on the application and hope problems surface.

Developing and testing concurrently accessed shared objects is hard! That's why CSP was invented. Just the same, the difficulties can't be swept under the rug.


In these situations we really want to be looking for race hazards and deadlocks. We want to find possible interleaved execution paths that cause the contract of the shared object to be violated. It would be helpful to have a semi-automated tool to search for these cases. (Particularly if it also knew about asynchronous interruptions like ThreadAbortedException.) However, to guide the analysis we probably need a way to express the desired non-deterministic behavior as part of the shared object's contract. I know this kind of work has been done many times over using theorem provers over process calculi. I've yet to see one applied to ordinary code outside of a research environment.


But there's more to it than that. Besides correctness we are often also interested in performance. After all, since we decided to optimize throughput or bandwidth through increased parallelism we should really get some measure of success. Was the added complexity worth it? How do we know our fancy concurrently accessed shared object won't introduce an insurmountable bottleneck despite our efforts? It is hard to create a realistic load profile for synthetic benchmarks. Operations involving lots of I/O with external systems are especially nasty.

Ultimately I think the hardest part of performance tuning is the interpretation of the results. Moreover, performance optimization is a dynamic feedback process. As the system evolves previous observations become invalid. The best tuned algorithm at one time can suddenly lose its edge as the underlying platform evolves.


I generally avoid premature optimization during development. Quite often what happens is that the performance of the simplest possible design is in fact quite acceptable. It's much easier to verify the correctness of a simpler implementation!

As for optimization, instead of using a raw performance metric like CPU utilization, I think in terms of scalability metrics. "Will this process handle the current 100 transactions per second? Will it need to be scrapped to handle the anticipated 1000 transactions per second?" Reframing the problem helps me avoid wasting effort on useless micro-optimizations. It keeps me focused on choosing a good algorithm first. Once that's done I'll start looking at cheap low-level improvements if the cost/benefit ratio is good enough.

Writing it down

As if deciding what to test and how wasn't hard enough, actually implementing a multi-threaded test is a real bear. You need to set up threads, catch unhandled exceptions, introduce synchronization barriers, handle non-deterministic events, tear it down, and somehow yield diagnostic output regarding the sequence of actions that actually took place so you can figure out what went wrong. Phew! I dread it every time.

Fortunately it doesn't have to be that hard. A couple of months ago I wrote down an idea for an MbUnit extension for "agent-driven testing" to help with all of this bookkeeping. The idea is think of each concurrent process that participates in a test as an agent. A test can spawn any number of agents of various kinds that each run in their own thread. If an agent terminates abnormally, the entire test including its agents is aborted with a failure result. Therefore agents can write to the execution log and evaluate assertions concurrently. Moreover, agents can access the shared state of the fixture. They can also communicate and synchronize via signals, condition variables and barriers.

What distinguishes this idea from other great multi-threaded test helpers out there is that it will leverage more of the test framework and consequently will be much more expressive. I'll say more about this later...

P.S. I made up the term "agent-driven testing". I'm not quite satisfied with the name. Surely someone else has come up with a better one.

P.P.S. I wish we could easily do run-time instrumentation of code in .Net. It'd be exceedingly cool to build something like ConTest. I have been thinking about using Mono.Cecil to do some interesting things at runtime though.

Sunday, September 9, 2007

The GUI looks good but how are her conversational skills?

Ayende asks How to sell maintainability and takes a stab at demo-ware. Somehow this got me thinking about female dating personae. (Disclaimer: This is in no way intended to be sexist!)

Persona: Demo software. It's her first night out on the town. She's nervous and is working to overcome her shyness. She wears a little too much make-up. She masks her insecurity by redirecting attention to her looks. She talks a bit too fast but she avoids engaging in deep conversation. She's afraid she might look stupid because she's inexperienced. She desperately wants to feel accepted and fit into the crowd. She tries to show off but she doesn't understand the rules by which she will be judged.

Persona: Mature software. She's sitting at her favorite restaurant. She's confident and relaxed. Her clothing is a little out of style but still shows a great deal of subtlety and careful selection. She knows herself and has pride in her abilities. She discourses openly and vividly about her life and goals. She takes suggestions seriously and is not easily flustered by the judgments of others. She is comfortable as she is. She knows she is valued by others and feels no need to be flashy except when it pleases her.

Revenons à nos moutons

Building maintainable software takes a long time and a lot of effort. It takes real commitment to build a productive relationship with the code base. It takes careful planning, patience and persistence. So in my mind the only way to sell maintainable software is to sell the value of that commitment. After all, the business organization will be "stuck" with it for a long time: it should invest the time and energy to ensure the relationship will be pleasant and fruitful even as the world turns and changes.

Kind of like marriage... ;-)