Thursday, December 6, 2007

More Code Submission Tips for Job-Seekers

If you apply to an automation testing position, please make sure your code submission is at least syntactically correct.  I do not really expect perfect code but try to keep the overall bug count to a minimum.  You have plenty of time to polish your code submission before you apply, please use it wisely.

Buggy code will not in itself result in disqualification.  But it will provide some interesting fodder for the phone screen.  I will ask how you tested this code.  I will also ask you to walk me through some of it to see whether you can find a few of the problems on your own.  This is not much different from the whiteboard coding exercises you'll get during an on-site interview.

I'm not a mean guy; I want you to succeed.  So you really need to put your best foot forward.

Tuesday, December 4, 2007

Onwards! Upwards!

I'm doing some thinking about my future.  So this post is about me, my desires, my fears, my conceit.

Thinking Out Loud

Knowledge is power.  Now that my employer is being acquired some quantities have been fitted to previous unknowns: I have more knowledge.  This gives me more power.

Where to go from here?

Running the Numbers

Now I know exactly the (cash) value of retaining my current position and can compare it against that of pursuing other opportunities at any given point in time.  Clearly this analysis neglects the intangibles: professional growth, loyalty, potential.

About those intangibles.  I value and respect my coworkers.  I value my role as a lead.  I value the resources I have to work with.  I value my freedom in choosing compelling projects.  I value the cooperation.  I value the teaching opportunities.

But unfortunately I have little faith that my current trajectory will lead me to achieve my long-term goals.

Who am I?

I'm a software developer, a teacher, a friend.  An entrepreneur?  A leader?

I marketed and sold my first serious application program when I was 13, Term 80.  Sure, it's a bit laughable now (only 20,000 lines of assembly with lots of rough spots) but it's hard to ignore all of the supportive letters that I received from people who sent me $15 for each copy.  It's also hard to ignore all of the enthusiasts who still apparently use it!  It's 15 years old for pete's sake!

That means a lot to me!  I built something people wanted.

It's also hard to ignore the fact that I seem to be able to motivate others to achieve their best.  Passion is infectious.  I don't hog center stage: I showcase the excellent work of my peers.  I teach and I learn.  I learn and I teach.  Mentoring is one of my favorite activities.  (And I married a teacher.)

But I've been off doing these things on the side.  I haven't participated much in the wider community.  Why didn't I go to DevTeach?  I certainly wanted to.  I did go to ALT.Net in Texas and had a thoroughly mind-blowing experience.

Meanwhile I'm working with some really smart people on Gallio and MbUnit trying to build a diverse and inclusive community of toolsmiths, testers, and the people who love and work with them.

Returning the Favors

When it comes down to it, I'm a very social guy.  I think I can do a lot of good with others and for others.  I believe I do this everyday on a small scale.  But I want a bigger force multiplier to work with.  I want to positively impact the lives of as many people as I can.

I owe a great debt of gratitude to those who have supported me, challenged me, disciplined me, and led me on to greater heights.  I feel the best way to repay it is to pass it along.  How do you create leaders?  I don't know, but I keep trying.

Entrepreneurship

I'm drawn to entrepreneurship.

My father runs his own business.  One of my uncles runs his own business.  Several of my friends and peers run their own business.  And I've always wanted to run my own business too.  I certainly have no shortage of ideas.  I've given plenty of them away and I've even marketed a few...

When it comes down to it, starting a business does not seem all that difficult.  Choose your partners, set an initial heading, start moving in that direction, get funding, build a product, see what sticks.  Work hard, be persistent, and see what happens.  Worst case?  You fail and try again.

That's really no different from how I developed my current competence in software development.  I spent my whole childhood on it.  My intuition as an architect came from building systems end-to-end out of diverse parts.  My persistence came from repeated failure and eventual success.

It's not about getting rich.  It's about acquiring the resources to invest in my dreams.  It's about attracting talent.  It's about bringing out the best in myself and in others around me.  And it's about doing damn cool things!

Long-Term Goals

Here they are:

  • 6 months: Get out into the world.  Teach.  Lead.
  • 1 year: Become a father.
  • 2 years: Start a company or join in at ground level with someone else.
  • 5 years: Be the founder, chief technical officer, or architect of a successful startup company.
  • 10 years: Help someone else get off the ground.
  • 20 years: Retire.  Keep learning.  Keep investing in the new leaders of the world.

Realizations

I am but an egg.

While I continue to refine my skill as a software developer, my current position is mostly teaching me about leadership, management, and business.  It affords me access to the insight and expertise of others in domains I have not yet mastered.  The workings of the business are being laid bare.  I'm reading books about management.  I'm learning to effectively delegate tasks to others.  I'm studying politics.  I'm learning to be a little selfish.  I'm learning to release the outcome.

However, I realize this position will never enable me to achieve my goals.  I will need to move on.  Others who mean well have tried to convince me otherwise but I recognize how they are protecting their immediate interests.  Just the same, I know they will support me if I left just as they do now while I stay.

The longer I stay, the more I can learn (and the more my options vest.)  But the longer I stay, the more I defer investing in new possibilities.  There is value in persistence and repetition.  There is value in change and novelty.  So hard to choose.

I could easily go elsewhere with a higher salary and with new challenges.  I'm sure I could learn a lot there too.  It would be fun.  I might have more influence.  I might have more guidance.  I might grok more of what I need to know.  Or I might not.  It's a gamble.

Cutting Loose

It feels like I'm just waiting for my next guide to shove me out of the nest.

Eventually I have to cut loose, find a partner, and try to fly...

When?  Now?  6 months from now?  A year from now?  Surely not much longer than that...

Help?

Code Submissions.

When you send code submissions to me along with your resume, don't just submit your most recent school project and send it to me.  Please spend the extra time to clean it up!

I want to see the quality of work that you are capable of producing.  I want to see your best work.  I want to see that you care enough to make a good impression on me.

Tips:

  • Don't express frustration in comments.
  • Remove debug output.
  • Add documentation.
  • Include test cases.
  • Include build / deployment scripts and other infrastructure if you are sending me a whole project.
  • Refactor it.
  • Write your code for a human reader, not for a compiler.
  • Use a consistent uniform style throughout.  I can tell a lot about your experience based on where and how you use whitespace.  I want to see that you are aware of the overall effect and can adopt good coding habits.
  • If you can, don't send me a school project.  Send me something else you built for fun on your own time and have worked on for a couple of months / years.  Even if it's incomplete.  Heck, maybe you can spend a day and write some really cool little demo application to show off what you know.
  • Send me something that you think is cool.  Don't try to guess what I will think is cool.  I think lots of stuff is cool...
  • Don't overdo it.  Ultimately I will spend about 10-15 minutes analyzing you code.  I'm skimming through it taking a random sample of what seems interesting.  To increase the likelihood of me stumble upon the juicy parts, try to keep the overall size of your submission pretty small (< 10,000 lines).  Or maybe you can offer hints as to your favorite modules.
  • Don't rush!  I'm quite happy to wait an extra day for you to put together a compelling portfolio.  It's worth your time and mine.

In the end, I want you to choose what you feel is your best work -- even just a little part of it -- and send it to me.  This is your portfolio.  I know you're still learning and growing as a developer.  Show me an achievement that you are proud of.

Because in the end, you can bet I will try to push the current limits of your ability.  I will work with you to create an environment in which you can excel as a full member of my team.

So give me a real taste of what you can do.

Monday, December 3, 2007

Wanted.

Smart, passionate, hard working software engineer seeking similarly inclined ne'erdowells to go out for beers, brainstorm, and found a successful start-up company.

Poster lives in the San Francisco Bay Area.

Yes, I'm serious about finding partners for a start-up!  Know anyone?

Thursday, November 29, 2007

Gallio / MbUnit v3 Alpha 1 Release

Today we decided to cut an early preview release of the Gallio Automation Platform and MbUnit v3.

Download the combined installer from here.

Alpha 1 marks a significant milestone: much of the groundwork is in place.  It doesn't look like much yet, but there's a whole lot going on inside.  We can run MbUnit v2, MbUnit v3, NUnit v2 and xUnit.Net beta 2 tests in a variety of ways, generate reports, write plugins, and produce installer packages with integrated documentation.  We can also do nifty things like embed attachments, such as WatiN screenshots, into reports.

However, this release is primarily of interest to fellow toolsmiths.  We do not recommend using it in a production environment at this time.  This is a work in progress.  Some parts are already quite polished but there are still many loose ends.  Many planned features are absent.  We appreciate your interest and support in our efforts!

Alpha 2 will consolidate what we've accomplished with Alpha 1 to deliver a genuinely useful product.  We plan to incorporate more work from NBehave, NUnit and other teams we are bringing on board.  In addition, we'll pick up the pace on the development of the end-user tools (particularly the GUI) and of the framework components.  This is also when MbUnit v3 will begin to blossom.

And we'll take it from there...

If you are interested in learning more about the Gallio Automation Platform and MbUnit v3, please join us on the Gallio-Dev or MbUnit-Dev mailing lists.

P.S.: I'll try to get some screenshots and a walkthrough up soon.

Tuesday, November 27, 2007

Close but not quite...

I was pondering different mechanisms for adapting some existing reflection-based code to transparently perform reflection against a variety of targets: types loaded in the current AppDomain, types whose metadata has been read from disk, and types that are purely defined by an in-memory AST.  (The reason is to support running MbUnit tests inside of ReSharper.)

The xUnit.Net guys encountered this same problem.  They chose the obvious solution of defining a few new interfaces such as ITypeInfo and to write wrappers that accept .Net Type instances or ReSharper ITypeElement instances.

Sure that's a fine approach.  Can we do better?

It turns out that the .Net class library partly supports a retargetable reflection model via subclassing.  You can actually subclass Type, MemberInfo, and other types and override most of their implementation.  The Reflection.Emit library uses this feature extensively.

There's just one hitch: you can't subclass Assembly or ModuleReflection.Emit does this but you and I can't because the constructors are internal and much of the behavior is non-virtual.

Oh duck typing, where are you?

*sigh*

Wednesday, November 7, 2007

Beautiful Code

I've been reading Beautiful Code on and off during my commute and in the bathtub.

I had high hopes for this book.  I'm somewhat disappointed.  I found some several chapters quite good, others a bit mundane, a few were somewhat tedious to read, and a couple espoused what I consider to be risky design practices.

Nevertheless, I did enjoy the diversity of the examples and opinions.  It's always good to read what other coders have to say, even when we disagree.

There's much more to be said about beautiful code.  I haven't dug into the Beautiful Code web site much yet but at a glance it looks pretty good.  And there's always lots of stuff to read in other books, blogs, and mailing lists.

My first piece of advice to other programmers is always: "Read more code."  So if you're thinking about reading this book, I'd say go ahead.  It can't hurt.  ;-)

Friday, October 26, 2007

A Loopback Mail Server for Functional Regression Testing

So for functional regression testing purposes, we need to be able to create tons of user accounts on our system.  Every user account needs an email address.  And of course, our system sends out notification emails to users so we need to test that stuff too.

No Frills

Ok, so I want to write tests that can submit an email address to the application during user registration and then go read that user's mailbox for any notifications that have been received.

Here's the constraint: I can't instrument our application and stub out its email sending code with a stub.  Remember, I'm writing functional regression tests here.  Some of those will be smoke tests that we want to run in a live deployed environment.  So the emails have to be real.

I don't want to use the corporate mail because of administrative overhead.  I am certainly not interested in asking Ops to create thousands of test email accounts and ensuring their contents get purged on a regular basis.  So we've got to have our own SMTP and POP3 server just for testing purposes.

And there are other requirements:

  • Zero-administration.  I can install the server on some machine and forget about it.
  • Zero-configuration.  I never have to worry about creating user mail boxes.
  • Self-cleaning.  It'll purge stale data automatically.
  • XCOPY deployable.  I can just drop it on a server and maybe run a script to install it as an NT Server and I'm done.
  • Lightweight.  It has such a negligible footprint that it's well below the radar in terms of resource utilization.
  • Robust.  It keeps going, and going, and going...
  • Cheap.  No per-user licensing, please!

Buy or Build?

A little Googling turned up nothing useful in this vein.  (Which surprised me!)  So we can't buy our way out of the problem.  Darn.

Ok, build it.  I'm sure we can cobble such a solution together using sendmail and a mess of perl scripts but I just don't have the time or inclination to dive into that kind of arcana.  (Besides, they both make my eyes bleed.)

Can I do it in a couple hours with off-the-shell components?  Yes!  It's not free, but it's definitely cheap enough.

Introducing the Loopback Mail Server

Last night I spent a couple of hours and put together a simple SMTP / POP3 mail server as described above.

Get the code here: LoopbackMailServer.zip.  As mentioned above it does depend on the CleverComponents .Net suite so it's not quite free but it does the job.

Here's what it does:

  • The Loopback Mail Server stores mail messages in a folder called Mail within the application directory to contain the user mail boxes.  Messages are kept around for a maximum of 7 days and are deleted automatically during periodic cleanup sweeps every 4 hours.  (These parameters can all be configured if desired.)
  • The SMTP server accept email sent by any user (no authentication is performed) to any recipient.  Mail for the recipient is stored as flat files in an automatically created folder named based on the recipient's email address.  The recipient's email address is only lightly validated to ensure that it is a valid file name.
  • The POP3 server accepts connections from any user (no authentication is performed) and provides the ability to retrieve and delete mail messages.  The POP3 user's name must be a valid email address which gets mapped back to the user's mail box folder on disk.  If there is no such folder then the user's mail box is considered empty.  Otherwise the files within the folder are enumerated and presented as the mail box contents for that user.
  • The server can be run standalone by running the Standalone.bat file.  In that case a cheesy little debugging dialog appears.
  • More commonly, the server is installed or uninstalled as an NT Service by running the Install.bat or Uninstall.bat file.

The code archive includes a Readme.txt file to explain how the program is used.  It also includes some rough sample code for sending/retrieving emails for testing purposes.

Using the Loopback Mail Server in a test

Here's what a typical test might do:

  • Create a WatiN IE browser instance.
  • Using WatiN, register a new user with a unique email address like user@loopback.mydomain.com where the loopback.mydomain.com domain has appropriates CNAME and MX records to point at the host where the Loopback Mail Server is running.
  • Using a POP3 client within the test (there's one included in the CleverComponents library), retrieve all emails for user@loopback.mydomain.com.
  • Verify that the user received a notification from the system regarding registration.
  • Extract the registration verification link from the body of the email notification.
  • Using WatiN, navigate back to the site and complete the registration process.

You'll notice we didn't need to tell the Loopback Mail Server anything about user@loopback.domain.com.  The system just sent an email to the account and it just worked.  Likewise we won't need to do anything to delete the account when we're done with it.  The Loopback Mail Server will take care of purging emails and mail folders 7 days after they are created (by default).

Tada!

Experimenting with the Designer

Just for kicks since it was such a simple project I decided to see how much I could use the designer to build it in a RAD-friendly way (shh!).  The experience was something of a mixed bag.

Overall I found the discoverability and responsiveness of the designer to be quite poor.  I was often left scratching my head as to how to accomplish quite simple data binding tasks or as to why the components I needed weren't showing up in the toolbox.

Overall, using the designer was often more of a hindrance than a help.  Even on the GUI layout of the debugging dialog, I encountered a lot of friction.  I still can't imagine using it for anything but the simplest applications.  *sigh*

Free and Open-Source Alternatives

Unfortunately I wasn't able to find any actively developed and maintained free open-source SMTP and POP3 server implementations for .Net.  It's possible I just missed them.  Moreover it didn't need to be .Net at all.  It could have been Java, or C, or even Perl for all I care.

If you are interested in writing a mail server like this without any commercial dependencies or if you know of another one that is already available, please let me know!  I'd love to use it and bundle it up as some kind of function regression testing package add-on for MbUnit or STAF.

kick it on DotNetKicks.com

Tuesday, October 23, 2007

But the xUnit Book Says...

The xUnit Book says a lot of things.  But it's not the last word on automated testing, not by a long stretch.

Steven Micham posted this comment in relation to my blog post about using dependency injection in a test framework:

I've also just skimmed through the xUnit Test Design Patterns book and the criticism of NUnit and therefore MbUnit with the [TestFixutreSetup] is exactly that the shared state is a 'bad thing' for constructing isolated test environments on a per-test basis.

Since this injection is basically shared state, how do you answer that?

You've got a good point.  So how do I answer that?

All that glitters is not unit testing.

Well, [TestFixtureSetUp] is obviously not a great foundation for building isolated test environments on a per-test basis.  We have [SetUp] for that.  However, there are cases when fixture-level isolation is desirable.

Here are a few:

  • The tests use resources that are expensive to initialize or to dispose.  For example, the tests might need to start / stop a partially stubbed out service with a high initialization overhead.
  • The tests involve a common computation that results in a lot of state that we want to verify systematically.  For example, the tests all consume the same input data that is produced by compilation.  Each test might examine this data in a different way so you get soluble, focused assertions across independent portions of the state instead of a single massive test with dozens of unrelated assertions within.
  • Or, when you're not writing unit tests...

Smells like integration testing.

Mmm... the musty odor of integration tests.  Reminds me of mothballs, old sweaters and damp wool.

Every test you write integrates some parts of the system.  The more you integrate into the set of components under tests, the more expensive your test will be, and the more difficult it will be to write.  Eventually you find yourself writing system tests (functional regression tests) and you wonder where all of the colour in the world went.

There's a bit of a change of scenery as you leave the comfortable realm of pure unit testing with mocks, stubs, and hard lines.  Integration testing is a vast desert: raw, wild, dangerous.  You can get hurt out there until you learn to adapt.

It's remarkably easy to encounter situations writing unit tests where you cannot exercise as much control over the environment as you'd like.  For example, the component under test might do some heavy lifting with the filesystem.  Often you can refactor your way out of these situations by stubbing out behavior or moving the heavy-weight concerns elsewhere where they are easier to test.  However, there is a point of diminishing returns.

So at some time you'll look at your suite of mocks and in desperation say: "I guess I'll need to create a real file on disk and make sure it gets deleted after the test finishes."  Ok.  So you do that and put the necessary code in [SetUp] and [TearDown].  And then you need a few more files...  Oh no, you've crossed over into integration testing.

Maybe you can refactor to improve the design.  Go ahead and try it!  Or... maybe not.  Crud.

Now what?

If you're going to write integration tests, do them for real!

It's a fact: integration tests are qualitatively different from unit tests.  The most dangerous thing you can do is to look at your tests and think: "these are just slightly ugly unit tests."  They're not really unit tests anymore.  If you try to treat them that way you'll just end up with "really ugly integration tests" and which is really bad.

So what do you do?

Well, first you want to be using a testing framework that degrades nicely when you reach the limits of white box unit testing.  As you write integration tests you're going to end up breaking a lot of unit testing rules.  You need a tool that can keep up with this transition.  Or you get a completely different tool.

In practice, MbUnit and NUnit both degrade fairly well for integration testing.  That's where being able to selectively violate test-level isolation by introducing shared fixture-level concerns is really handy.  Using a completely different tool is probably not worth the pain until you start doing something very different, like functional regression testing.

But... to help keep things straight, be sure to keep your unit tests and integration tests isolated.  Spend extra effort documenting those integration tests.  Intent revealing test method names might not be enough.  You'll need real documentation: possibly even paragraphs!

And that's ok.  We're not really doing unit testing anymore.  Even if we used a unit testing framework to implement the tests.  We crossed over (and it didn't hurt much).

I do care what the xUnit Book says.

Yes, I do care.  But the xUnit book is about unit testing.

The extra features MbUnit provides that other frameworks like xUnit.Net do not are there precisely to give programmers the freedom to selectively break the unit testing rules but not the tool.

MbUnit is intended to be used for more than just by-the-book unit testing.  So we offer support and guidance for those activities too.  We just don't expect the tool to enforce our recommendations to the exclusion of all else.

There's other good stuff out there.  Trust your instincts.

Constructor dependency injection in a test framework!

Nonsense?  Hardly!  (Translation: I'm doing it already and it's very useful.)

After reading the plans for xUnit.Net beta 2 that involve a cumbersome IUseFixture<T> interface type, I decided to pass on some of my ideas for MbUnit v3 in hopes that they will be of value to others.

Defining the problem

Problem: I've got some common state or behavior that I want to share across multiple tests within a fixture (or perhaps across multiple fixtures).

Answers:

  • With MbUnit v2 or NUnit, just put the state on your fixture class and use [TestFixtureSetUp] and [TestFixtureTearDown].
  • With MbUnit v3, you can do that or you can create "mixins" that are defined using dependency injection on the fixture class and are first-class participants in the test suite.  Mixins are standalone classes that augment fixtures with custom behavior via attributes like [TestFixtureSetUp] or [SetUp] as usual but they can also contribute new tests, data sets, and other goodies.  Using a mixin is as simple as declaring a property, field, or constructor parameter of the mixin type on your fixture (or on another mixin).  Look for this feature in MbUnit v3 - Alpha 2 in a month or two.
  • With xUnit beta 2, your test class should implement IUseFixture<T> and put the necessary logic in T's constructor and Dispose.  But there are drawbacks to this approach.  One of them is that it results in additional boilerplate for providing setter methods to inject the shared instance.  Another is that the setter methods are obviously called after the constructor runs so you can't consume the shared instance as part of any constructor-based initialization.
  • Something else?

Constructor dependency injection to the rescue!

Another way to think of the problem is to imagine we are trying to inject services into the test fixture.  The service initialization and disposal can be taken care of by the framework per-test or per-fixture as required.  Let's see what constructor dependency injection can do here...

// Just an ordinary test fixture...
[TestFixture]
public class SomeTests
{
    private readonly IE ie;
    private readonly FixtureGlue glue;

    // The framework can use constructor dependency injection
    // to provide instances of a WatiN IE browser wrapper
    // and some custom FixtureGlue type as needed since they
    // are concrete types.  They also both happen to implement
    // IDisposable so that the framework can take care of
    // disposing them too.
    //
    // So to inject concrete components like this into a test,
    // all we really need is to declare a constructor with the
    // right signature.
    public SomeTests(IE ie, FixtureGlue glue)
    {
        this.ie = ie;
        this.glue = glue;
    }

    // Some ordinary test...
    [Test]
    public void MyTest()
    {
        ie.GoTo("http://www.google.com");
        ie.TextField("q").TypeText("MbUnit");
        ie.Button("iBtn").Click();

        Assert.AreEqual("http://www.mbunit.com/", ie.Url);
    }
}

// Encapsulates some common behavior we want to run before
// and after all tests.
public class FixtureGlue : IDisposable
{
    public FixtureGlue()
    {
        // configure something in the environment
    }

    public void Dispose()
    {
        // tear down whatever we did
    }
}

However, we haven't said anything about when we get new instances of IE and FixtureGlue.

Fine-tuning the semantics

Should we create new instances of the IE and FixtureGlue types for each test or share them across all tests within the fixture?

In MbUnit, the expected answer would be probably to share these services because the fixture instance is reused all tests so its constructor will only run once (although in principle we could make this configurable).

In xUnit.Net, the situation is a little different.  xUnit.Net always constructs a new instance for each test.  So by default the test author will probably assume that the injected services are not reused.  However, that's precisely what we want...

To facilitate local reasoning about the tests, we can offer a custom attribute that is applied to the constructor's parameters to explicity state what's going on.  For xUnit.Net, we can add a [Fixture] attribute that states that the value of the associated parameter should be reused for each test in the fixture.  This usage of the word "fixture" seems pretty consistent with xUnit.Net nomenclature.

So our revised constructor declaration looks like this:

public class SomeTests
{
    // Initialized with a new instance of IE that is created
    // and disposed for each test and a shared instance of
    // FixtureGlue that is reused across all tests in the
    // fixture and disposed when finished.
    //
    // So we can see that by default services are recreated
    // each for each test unless the [Fixture] attribute appears.
    public SomeTests(IE ie, [Fixture] FixtureGlue glue)
    {
        // as before...
    }

    // as before...
}

Obviously this idea can be taken much further.  Dependency injection can be used for parameterizing tests in plenty of other interesting ways and with more or less customization of the underlying process...

Spoiler: MbUnit v3 does all sorts of nifty stuff with dependency injection, particularly related to data-driven testing and to increasing the reusability of common test fixture extensions well beyond what can be achieved by subclassing.  Dependency injection has already proven to be immensely beneficial to me in the form of custom MbUnit v2 extensions for data-driven testing.  The plan is to go much much further.

kick it on DotNetKicks.com

Thursday, October 18, 2007

Pain points

Roy wrote a great post about an anonymous company that apparently has serious communication, leadership and accountability problems.

Generally speaking, there are various “pain points” that currently prohibit the departments from either getting the full data they need, influence the current releases so they contain the correct feature set or be more productive in their work.

In a strange turn of events, yesterday I met someone on the way home who seem to have similar problem in her line of work.  The discussion centered primarily around the cost and apparent ineffectiveness of manual regression testing being performed "over there" in the outsourced QA departments.  I found it somewhat distressing overall.  Dollars and units of time were being weighed against an apparent need for more leadership, better internal processes, and improved flow of communication.  There may also be room for architectural improvement since it sounded like routine changes to products, unit price, and supply chain were requiring excessive regression testing and had a high defect rate.  I have too little technical information to guess what's going on there.

I'm all for work smarter not harder but I don't quite understand how work cheaper and faster is supposed to play out.  Fortunately the person I encountered was interested in improving internal processes to get there.  However I received a somewhat neutral response when I suggested leadership might also be required.  Moreover it's pretty clear that the outsourced workers are not really considered full members of the organization.  Not good but not too surprising.

All in all, I enjoyed listening to a 10,000ft summary of one person's internal perspective on the organizational difficulties of a large multi-national company.  I found that my outlook was somewhat more nuanced and biased towards the people involved whereas hers was rather categorical and biased towards the business demands.  That in itself was interesting.

My intention is not to judge, but these observations bear more thought.  I'm not accustomed to thinking on such scales.

I want an office with a sofa!

Monday's enhancement to my home office was a sofa.  It's just like the one pictured on the right but I opted for a deep red fabric instead of the cream color.  The ends fold down into sofa, chaise and cot configuration.

Anyways, I'm sure I can find more attractive sofas but this one was pretty much what I was looking for and at a price I was willing to pay.

No more clutter pile of boxes and dust bunnies in the corner of the room!

Now I've got a comfortable place to sit and think or to snuggle up with my wife and watch a movie on the flatscreen.  Ahh!  Now if only I could get one of these at the office...

Once upon a time I worked in an environment with lots of natural light, open space, and a central lounge that was actually in the middle of the space rather than tucked off in some undesirable corner.  Surprise, surprise, some of the most insightful conversations would occur on or around the couches in this conveniently located lounge space...

False sharing opportunities across modules.

I guess I'm really just writing this for me.

In one of my current projects, I have a case where two contributors on separate occasions have moved a class from one module to another (to increase reuse).  In each case I provided an explanation of why it was in its original location and requested that it be put back there again.  (I'm considering sticking a remark on the class to prevent further tampering...)

The explanation is a little long-winded.  Essentially it boils down to the fact that any apparent reusable potential of this particular class is illusory.  It declares the command-line options accepted by a tool along with help text and whatnot.  This is true despite the fact that several tools may offer similar options.  The problem is that the options aren't really identical.  The actual parsing code has been factored out to a common utility class and is reused.

Thinking...

Anyways, this got me thinking.  How could two very smart guys independently make the same mistake.  I think it has to do with modular design considerations that aren't always so obvious.

When new modules are created, it always happens that some things must reside inside the module and others must reside outside.  So module boundaries do constrain reuse and for good reason: it's to protect the encapsulation.  Opportunity for sharing the insides of modules should be viewed with suspicion until proven worthwhile lest they introduce undesirable coupling.  Perhaps more subtle is the case where the sharing opportunity exists only because of accidentally and rather arbitrary circumstances.  Watch out for those.

It's worth keeping in mind that redundancy in the small scale is often not very relevant to the overall architecture.  Outright duplication can exist across and within modules due to such diverse reasons as: isolated development groups, greatly varying rate of change or maturity of modules, historical carry-overs, inefficient cross-team communication, licensing constraints, conformance requirements, political tension, inexperience, carelessness.

Respecting my promise...

I had prepared a longer diatribe on this topic entitled DRY is not DAMP (Don't Repeat Yourself is not Disregard All Modular design Principles).  But I remember my promise to try not to be antagonistic...

Just the same, I'd like to point out one pet peeve of mine that recurs frequently in UI toolkits.  It's common for the Check Box and Radio Button controls to be represented by classes derived from a common ancestor like CheckBoxOrRadioButton.  After all, both controls essentially consist of a toggle-button and a label.  Right?

No, wrong. I believe the sharing in this case is illusory.

At best, I believe the shared implementation concerns should be captured by delegation rather than inheritance so they don't clutter up the API.  After all, Check Boxes and Radio Buttons have widely varying semantics.  The controls are not really substitutable anyways.  This is part of why I feel the common strategy of using inheritance to express composition of UI components is generally misguided.  (Or maybe I just like SWT more than WinForms.)

Interfaces should not leak implementation details to the surface simply by virtue of having exploited some opportunity to reuse a little code.  Each module should be considered a cohesive unit with a clear relationship to the whole and unburdened by inherited concerns that are of no relevance to its users.

In fact, sometimes there are very good reasons for skipping over an apparent opportunity to reuse code because it would do more harm than good.

Saturday, October 13, 2007

Gallio Progress

There's something deeply satisfying about this screenshot. Here we can see MbUnit v3, MbUnit v2, NUnit and xUnit.Net all side-by-side in the same test runner. Not bad.

Edit: I just noticed that I mislabeled the xUnit.Net framework in there. It should be using the official name of the project rather than Xunit. That's fixed now.

Friday, October 12, 2007

Software Automation Wizard in San Francisco

Ingenio is hiring software automation wizards.

Self-Operating Napkin (Rube Goldberg) On a selfish note, I'm looking for someone to work with me to use and grow our test automation platform and solve challenging integration testing problems.  There is a lot of creative programming involved as we write and maintain these tests and the systems that run them.


Software Automation Wizard in San Francisco

Do you enjoy writing high-quality code using the latest tools?  Do you hate bugs?  Do you like solving challenging system integration problems?

Ingenio's automation team is looking for engineers to develop a comprehensive and maintainable test suite for Ingenio's many products and diverse features.  We'll bring you up to speed with the latest techniques writing automated tests in C# with MbUnit, WatiN, and other tools.

Our system consists of a mixture of web, telephony and data platforms.  It takes a lot of creativity to make sure all of the pieces work together to provide a compelling end-user experience.  There is no out-of-the-box solution.  In fact, we have contributed extensively to several open source projects to help solve this very problem.

What we will look for in you:

  • You love to program!  You have most likely written stuff in various languages.  You already know at least one object-oriented programming language and are interested in learning C# (if you don't already know it and a dozen others).  You have opinions on why one language can be better than another for some task.  You’re not afraid of reading the manual and experimenting to learn about more powerful language features.
  • You love to program so much that you have a 4 year degree in computer science or engineering.
  • You have a passion for software testing.  Breaking things to figure out how they work is something you have done for a long time – even before you thought you could get paid for it.  You have probably sent complaints to the various web sites you use when you do something odd on the site and it doesn’t work as expected.  Maybe you already have some experience writing tests using MbUnit, JUnit, WatiN, WatiR, Selenium or other tools.
  • You have great communication skills.  You will often work with other engineers, program managers, and business people.  Conversation comes easy to you no matter who you’re talking to.
  • You are highly motivated.  In fact, perhaps you haven’t even heard of WaitN but by the time you have contacted us you have already written some cool stuff with it.

The Social Enterprise

A couple of days ago, Ayende wrote:

Enterprise Software - A common term used to refer to systems that usually run a core part of a business.  Often those are mission critical systems, which interact with many other systems in the organization in many interesting ways.  Enterprise software is considered hard because it often need to mix technological capabilities with the business logic, and be flexible to handle changing conditions as the business evolves.

I feel there's something very important missing from this definition.

Where are the people?

Enterprise software is hard not just because of gnarly technical system integration issues related to core business processes but because it has to be very carefully tailored to fit the social environment of the enterprise.

I'd argue that the social challenges of enterprise software greatly outweigh the technical considerations.  If the system does not somehow relate to people, it is not an enterprise system.

Enterprise systems generally incorporate a very large installed base of internal and external stakeholders that have widely varying perspectives and needs.  These people are accustomed to a particular conception of the business operating environment.  They have developed a specialized vocabulary to describe their doings.  They have adopted work habits to get them through the day.  To observers they often appear to behave in strange, mysterious and chaotic ways.  Changes run the risk of inducing systematic paralysis, shock, mass-hysteria and cannibalism.

We're going to build systems for them?!?  You must be out of your mind.

The stakeholders of enterprise systems are tightly coupled to the system itself.  They are a part of the machine and they exhibit a sensitive dependence on its behavior (and are often quite resistant to change).  Building and maintaining enterprise systems involves performing a carefully unscripted dance around the social, political, financial, legal and quasi-religious spheres of the organization.

The actual code seems almost irrelevant!  "Business logic" is a formalized contortion of the whimsical demands of a fractious collective of fanatics, individualists, providers, consumers, and parasites.

The true horror of enterprise systems can only be revealed in the actions and sentiments of people.  They are your business.  That's what's so hard.

Sometime I'll get around to posting my ALT.Net reflections...

Friday, October 5, 2007

ALT.Net Conference.

I'll be heading down to Austin, Texas for the ALT.Net Conference this weekend.  I'm very much looking forward to meeting everyone and seeing what you're all up to.  I'm also planning to give a little demonstration of MbUnit Gallio work in progress.

Should be lots of fun!

Sunday, September 30, 2007

Reading XML documentation comments

So today I decided to have a little fun and see what it would take to read XML documentation comments associated with code members in assemblies that are currently loaded.  It took a little doing to get it completely correct but it's done and fully tested now (including gnarly cases involving generic parameters).

There are a few interesting uses for performing reflection on XML documentation from within a running application.  For example here is a test case that reads some of its data from its own XML comments.  It's pretty nifty.  I'm not sure I would actually recommend this usage pattern because I find embedded resources and custom attributes are sufficiently effective and robust.

What I have in mind has more to do with enhancing MbUnit Gallio reports and GUI content.  The XML documentation simply becomes another piece of metadata that you can see or manipulate.  But I won't put it past others to come up with other interesting uses of this feature.

Gotchas: The MSDN documentation regarding how member names are computed for XML documentation comments is incomplete and incorrect -- particularly as regards generic type arguments.  I think I've got it nailed though.

Friday, September 28, 2007

The runaway product

A couple of days ago I read The Mythical Business Layer on Worse than Failure.

There's nothing at all shocking in there.  However, the article did provoke me to think about a few cases of over-engineering I've seen (and participated in) where features were added to the underlying abstractions that were never exposed to end users due to "limitations in the UI."  *ahem*  Those "limitations" existed because the core development team lost sight of the product and introduced excess flexibility complexity in the system that was completely irrelevant to the business.  The UI team had a more stringent specification to work from so they could not go quite so far astray.

A story that sucks

I think one of the ways this happens goes a little bit like this:

Business Analyst: Our Vacuum product is doing great.  We've got 13% market penetration in the Light-Duty Household market segment but only 2% of the Families With Children segment.  I think we can do better than that.
Product Manager: How about adding a feature to provide more power for cleaning up tough spots?
Marketing Manager: Yeah, we could totally sell that to those people.  Let's call it the Ultra-Suck option.
Business Analyst: I think that might appeal.  Let me go crunch some numbers...

Time passes...

Project Manager: Ok, so we'll be implementing the Ultra-Suck option for our Vacuum product.  Here's how it will look.  We'll add this button over here under the handle that provides a 10% power boost.  We'll also beef up the thermal cut-offs on the motor in case of overheating.
Engineering 1: Ho hum.
Project Manager: This is a cool feature!  Heck, maybe in the next release we can add multiple boost settings and an adaptive power meter...
Engineering 2: Okay.
Engineering 1: !!

Time passes...

Engineer 1: I'm bored.  Another stupid Vacuum feature.  The business guys just don't understand what people want.
Engineer 2: Heh.  But did you hear about that adaptive power meter?  That sounds pretty cool.
Engineer 1: Yeah.  I guess since we should plan for it then.  We'll need a way to measure the current load and scale the output power to multiple levels.
Engineer 2: Maybe.  But there's no point trying to measure the load now if we aren't adaptively tuning the output power.
Engineer 1: Ok.  But it's pretty easy to add support for scaling the power output.  Right now we need need 100% and 110%, but later we'll probably want a range from 50% to 120%.  The circuit won't be that much different.  We can just hardwire the Ultra-Suck button to the 110% level.
Engineer 2: I'm not so sure...

Time passes...

Project Manager: Is the Ultra-Suck button done?
Engineer 1: No.  I'm having trouble with the circuit that selects a range of output power levels.
Project Manager: Don't we just need two levels?
Engineer 1: Yeah but later on we might need more for the adaptive power meter feature.
Project Manager: But right now we only need two.  Get it done.

Eventually the project schedule slips due to "unforeseen difficulties" in the implementation.  The fanciful features of the anticipated future model may never come to be.  Meanwhile the product has acquired a lot of added complexity and has shown no net gain from all of this work.

The moral

While this story is somewhat contrived in the world of mass-produced physical consumer goods where unit production costs are very high, it happens all too often in software.  Somewhere along the way, the original narrowly focused business objective is lost.  "Just a little tweak here, a little tweak there."  But each tweak raises the cost of development, testing and maintenance.

That's not good!  There is zero business value to any of this extra work until it becomes relevant to the product.

Aside: I'm lying a bit when I say there is zero business value.  After all, the extra complexity may help to provide motivation to get things done.  Good people are hard to replace so there should be some allowance for keeping them happy.  However, I strongly prefer reserving 20% time for such projects in lieu of allowing the core products to become unnecessarily complicated.

kick it on DotNetKicks.com

Tuesday, September 25, 2007

How to delete Google Gears offline data

Google Reader jammed on me in offline mode today.  Try as I might, it wouldn't switch back to online mode even after I let it run for an hour apparently "synchronizing".  Restarting the browser and clearing caches didn't help.

After a couple of Google searches I did not find any information regarding how to manually clear out Google Gears offline storage.  So here's one way... (probably not the best way)

  • First close your browser.
  • You'll find the Google Gears offline data store for Firefox inside your Firefox profile directory.  On my machine, that folder is "C:\Documents and Settings\Jeff\Local Settings\Application Data\Mozilla\Firefox\Profiles\wb2qohxr.default\Google Gears for Firefox".
  • The offline data store contains a couple bookkeeping files ("localserver.db" and "permissions.db") and individual directories for each domain name with offline storage.  Just delete the appropriate directory.  In my case it was "www.google.com".
  • Restart your browser and away you go...

Works fine now.

Saturday, September 22, 2007

Release early, release often.

The reaction to James' Newkirk's announcement of XUnit (looks interesting!) has reminded me of a very important principle: Release early, release often.

Sometime last year I created some spiffy data-driven extensions for MbUnit that took it far beyond RowTest.  Actually, MbUnit supports a whole suite of data-driven testing features but there were still a few things I wanted to have that didn't exist.

Keep it simple

One of goals was to provide simplified support for pulling data from files.  I wanted a simple declarative form that would load data from CSV or Xml files with a minimum of fuss.  Ideally it would map values based on the actual field, property or parameter names in the tests so that very little in the way of data binding glue would be required in the common case.  It should also be easy to define new data sources (and it was...).

Here's an example using CSV:

[DataTestFixture]
[TestFixture]
public class AuthenticationTest
{
    [CsvResource("server-config.csv")]
    public string ServerName;

    [CsvResource("server-config.csv")]
    public string ServerPort;

    [DataTest]
    [CsvResource("credentials.csv")]
    public void Login(string userName, string password, bool isValid)
    {
        Server server = new Server(ServerName, ServerPort);
        bool success = server.Login(userName, password);
        Assert.AreEqual(isValid, success);
    }
}
Where we have two embedded resources.
server-config.csv
ServerName, ServerPort, Owner, Domain
mymachine.test.example.com, 9999, operations, test
credentials.csv
UserName, Password, IsValid
jeff, letmein, true
jeff, knockknock, false
jim, letmein, false
jim, knockknock, true
"", "", false

Execution of the test is pretty straightforward.  The test fixture's ServerName and Password fields are initialized with the contents of the matching columns of the server-config.csv file.  Then the Login test method gets run once for each row in credentials.csv.  Data binding occurs automatically including conversions from strings to other types (like booleans).

Integrate!

A second goal was to simplify retargeting of tests to multiple environments without recompilation or manual munging of files.  Like most enterprises, we have separate development, test, staging and production environments.  The configuration of each environment varies in a few particulars such as in domain names and in the pre-populated data set.

To capture environment-based configuration, I created a new batch of attributes like this:

[DevCsvResource("server-config.dev.csv")]
[TestCsvResource("server-config.test.csv")]
[StageCsvResource("server-config.stage.csv")]
public string ServerName;
Each new file now specifies configuration for a particular environment.

The environment to target is selected at runtime either by popping up a dialog box for the user to choose one (the selection is saved for the duration of the test session) or by grabbing it from an environment variable.  (Remark: MbUnit Gallio will include support for passing external values to tests directly from the command-line or from the GUI test runner to handle just this sort of case with less hackery.)

The moral of the story...

I thought this API was pretty cute so I published the (non-proprietary) code to the MbUnit mailing list.  However, since I was already thinking about how to perfect the feature in MbUnit Gallio, I refrained from polishing the code and contributing it back to the MbUnit v2 framework itself.  That was a mistake.

If I had polished the code, other people would be using it now and no doubt I'd have received plenty of useful feedback to apply towards MbUnit Gallio.  Oh well.  So here is the code for these simple data-driven extensions for MbUnit v2, completely unretouched.

With that in mind, we're gearing up to cut the first alpha release of MbUnit Gallio soon...  Release early, release often.

kick it on DotNetKicks.com

Wednesday, September 19, 2007

Validation

Validation is when someone else takes your vaguely sketched idea, embraces it, simplifies it, improves it, and realizes it better than you could ever have even imagined on your own.

Sunday, September 16, 2007

Shedding fallacies.

Work, health, or happiness: pick two three.

It's time...

Saturday, September 15, 2007

Tests that last.

Roy Osherove asks on his blog if you have ever written throw away tests (TATs) and whether you have since learned to write tests that last (TTLs).

Sure, I have written TATs and learned from the mistake.  Now I mostly write TTLs and they stay that way because I maintain them.  Sadly there have also been a couple of situations where I've transferred code ownership to others who did not practice unit testing and allowed the test suites to rot... *sigh*

My approach to writing TTLs is pretty simple.

Rule 1: Don't overcommit to your unit testing effort early on.

Ok...  Before the Agile folks put my head on a platter please let me clarify.

I am often engaged in designing architectural concerns such as APIs, processes, conventions and infrastructure.  These concerns all have associated contracts.  I wouldn't sign a contract in blood without careful negotiation first; would you?  You would do more research and you would haggle.  So don't overcommit testing resources to a hastily sketched out system that might change tomorrow.

Architectural concerns have a very high initial design burden.  The cost of changing your mind can be huge!  On the other hand, big design up front is risky.  This seems almost at odds with Agile development.  It isn't.

Ideally we want to partition the design process so that we can obtain confirmation of the soundness of core ideas incrementally.  However, because the core ideas all need to work together smoothly we must reserve the right to change our minds.  To get the feedback we need, it helps to sketch out sample clients that exercise a representative subset of the current prototype.  It's even better if the sample clients are executable because it provides more concrete evidence that we're on the right track.  Oh look, we just wrote integration tests!

What about unit tests for a new API?  Well, eventually we'll want heaps of those.  But early on, they aren't all that helpful.  Moreover since the design is still rather volatile, you run a risk of generating tons of TATs.  This is especially true for white-box tests.  Defer writing expensive and risky tests until you're confident they won't need be thrown away.  (But do remember to write them as soon as you're sure! In many situations that just means waiting a couple hours until you're done thrashing with ReSharper.)

Rule 1 (corollary): Write integration tests before most unit tests.

Unit tests are usually very tightly coupled to the code you're writing.  Volatility in the code under test can translate into major headaches and wasted effort.  In particular, reserve white box tests for last.

Rule 2: Don't aim for 100% code coverage of your first draft.

It's almost never right the first time.  Wait until well after you've first integrated a component into the system and has been thoroughly proven out.  Code coverage metrics are good for gaining more confidence that your implementation is correct; however, they are not of much use figuring out whether your design satisfies the requirements.  You'll probably end up refactoring your first draft beyond recognition so don't waste time checking it too deeply.

Rule 2 (corollary): Don't write white box tests until the implementation has stabilized.

White box tests are useful.  Don't let them turn into TATs because you wrote them prematurely.

Rule 3: Write your tests clearly.

All of the usual tricks apply:

  • Test code should be soluble.
  • UseVeryClearNamesThatMightEndUpBeingALittleLong.
  • Don't verify more than one behavior in each test.  (Try not to have too many assertions in a single test.)
  • Limit the usage of mock objects to generating stubs and setting up relatively straightforward expectations for dependencies.
  • Clearly document how the test harness is being set up and what interactions are taking place.
  • Include detailed messages in your assertions to describe expected behavior.
  • Don't do anything surprising in your tests.
  • Avoid introducing side-effects across test runs.
  • Try to isolate the tests as much as possible from one another.
  • Don't depend on too many white box implementation details.  Do cover the corner cases of your algorithms but try not to depend on exactly how the computation will proceed or in what order certain intermediate results will be produced.
  • Avoid hardcoding exact string equality comparisons except in trivial cases.  If you are calling a function that formats a text message containing some information then there can be a fair amount of looseness in the formatting.  Look for the presence of expected information by searching for matching substrings.
  • Avoid hardcoding hidden policy configuration that might change.  For example, today an anonymous email service might only let you send 5 emails per hour before a spam killing rule is activated; tomorrow it might be 2.  Instead of hardcoding "5" into your test, try to obtain the constant from the same source as the actual implementation.

Rule 4: Invent your own rules!

Conventions are good.  Learn what works and apply that insight routinely!  Then you'll be sure to get tests that *really* last.

MbUnit Gallio, for example

I'm currently architecting MbUnit Gallio incrementally.  If you were to download the source now, you'd notice there aren't all that many unit tests.  Two of the tests currently fail!  Have I lost my mind?  Shouldn't I develop a test framework using TDD?  Perhaps...

The problem is that I ran out of storage space in my brain.  There are too many things going on inside Gallio for me to have any confidence getting it right the first time as a pure gedankenexperiment.  Pretty diagrams and written notes were helpful at the high level but not so much below that.  I needed to write actual code (or at least design interfaces) so that I could observe the interplay between the various high level concepts and adjust them accordingly.  It's been very rewarding.  As the design process progresses, I've managed to isolate and solidify several components to enable others to work in parallel while I iterate over other fuzzy details.  It's getting there!

I've written integration tests at several different levels of integration within the system.  There's enough meat to get confidence that the design is moving in the right direction.  Those integration test failures I mentioned earlier were fully expected because parts of the system are still stubbed out!  Meanwhile I've been trying to progressively write unit tests for components that have solidified.  But here I've encountered the limits of my ability.  I can't do architecture, design, code, tests and project coordination all at the same time.  So as a team we shall work together to increase our testing velocity.  (Which leaves me feeling irrationally guilty because I prefer to take personal responsibility for thoroughly testing everything that I code.  Actually, I need to learn how to delegate more of the detailed design and coding work. Despite being intensely interested in these tasks I need to release my investment in them so that I can focus on maintaining the velocity of the project as a whole.  I am learning to scale up to larger projects.  It's not easy.)

I'm confident that almost all of these tests will last because we are working to understand the implications of the system that we are designing and building.  And we care enough to get it right (eventually)!

Friday, September 14, 2007

Pile it on...

I can't remember who sent me the link to Ralf's blog but thank you very much! This stuff is fantastic!

I think I might just have to go back and read through his entire history...  His writing style is incredibly lucid and compelling.

Being a protagonist

I had a thoroughly enjoyable conversation with my boss today (hi Ray!) wherein he asked me what I felt best characterised the sentiment of my blog.  Without hesitation, I answered: "Antagonistic!"  And... I was momentarily taken aback.  How valuable are my rants?  Is that really what I want to share with the world?

No.

At the risk of starting another rant, I would like to share my thoughts from this evening on being a protagonist...

What I want

I am a very passionate individual.  More than anything I want to be a force for goodness, balance and positive change.  Sure, part of achieving that goal requires thinking critically about the world and recognizing opportunity and means for improvement.  But it is a very small part.

I believe that the larger part of the challenge is to play an active role in making a difference.  Sometimes that means playing a minor supportive character assisting others who are engaged in a great epic.  Other times it means playing the humble protagonist who stands up and gathers others to uphold what is right and make the world more beautiful.  (Cue the Lord of the Rings soundtrack.)

Nobody likes a backseat driver

I could natter on endlessly about all sorts of software engineering p-r-o-b-l-e-m-s. *ahem*  Who asked for my advice anyways?

It makes no sense.

If I am not a protagonist and I am not acting like kindly Tom Bombadil to support others on their own quest, then what is my role?  There are two possible answers.  I could be a nobody whose only purpose is to contribute local colour to the story.  Or... I could be an antagonist who is distracting or perhaps even actively interfering with the major players.

Neither option is very satisfying!

Leadership, not Wizardry

Gandalf was not a good leader.  Sure, he set the story in motion but then he foolishly abandoned those he had rousted at their great peril.  In his arrogance, he assumed his own objectives were of such great importance that he neglected those he had gathered so that he could personally undertake his mission.  Consequently he very nearly failed to achieve his role in helping to fulfill their Great Purpose.  Thinking great thoughts is not enough!

Ok, so Gandalf was an idiot.  Not the only one...

I confess I have acted like a Wizard too.  I have falsely assumed the countenance of Great Responsibility.  I have meddled.  I have not dealt the cards fairly among those with whom I have shared responsibility.  I have upheld ludicrous ideals that ultimately accomplished nothing except to frustrate others who were better grounded in reality.

Pure foolishness.

Being a protagonist

I am working on MbUnit Gallio with a great team of people.  I add value to the team with architectural insight, programming skills and friendship.  However, I fear that I have also held us all back while I sat in a corner settling "complicated design issues."  That is not leadership; it is arrogant bullshit!  I am certainly not the only visionary.  I am not the only one who can finesse details.  Besides, sometimes my methods are crude and clumsy.  I make mistakes.

That said, I believe we are building a great product!  We are innovating.  We are discovering surprisingly natural solutions to nagging problems.  We are deeply invested in quality.  We care.

I am a protagonist but not the only one.  I must do my part to help keep the team on track and moving forward at a sustainable velocity despite my own weaknesses.  My personal opinions and ideals matter only insofar as they enable us to achieve our common goals.  We share responsibility in different capacities but in equal measure.

We are in this together.

P.S. For the record, I did enjoy the Lord of the Rings... even Tom Bombadil.

Thursday, September 13, 2007

Soluble Code

I had a discussion with someone today that reminded me of this post by Scott Bellware. Go read it: Soluble Code.

Specifically we were discussing design conventions and how they help (or hinder) the people that follow them (or don't). Once again I had to make the point that code is read far more often than it is written. It's important to have solid conventions in place so that we can understand each other and work towards common goals. Otherwise the end product will have all of the permanence of the Tower of Babel.

I've noticed a strange cognitive dissonance with some programmers. On the one hand they claim to value code clarity. But on the other hand when they get stuck they labour under some misapprehension that it really won't matter what goes on behind the curtain because no one else will ever see it; or so they believe. These seem to be the same programmers who have difficulty grasping the Law of Leaky Abstractions and who cannot be trusted to design a sensible API with more than 3 functions in it because they'll try to make it "simpler" by squishing all of the behavior together with as many "helpers" as possible.

Trying to hide ugly and complex code is just a sure-fire way to ensure that you're stuck with it for life because no one else on the team will be interested in maintaining it. So don't do it. It makes me sad because I have only occasionally been successful in reaching out to these programmers and teaching them to think more critically about what they're building.

Of course I too suffer from occasional delusions. I like to believe the fiction that insoluble code will eventually get ejected from the system (like dietary fiber). But it does seem to take quite awhile to digest sometimes...

P.S. Oh yes, and No Broken Windows!

Read more code!

By now most of you have probably read on SlashDot that QNX has opened part of the QNX Realtime OS source code.

Naturally I'm jumping on the opportunity to read more code written by others.  I find it fascinating, particularly with mature projects.  For example, here's the QNX kernel entry code for x86 (written in assembler of course).

Fascinating!  I can read it but I don't understand half of what it's doing.  What more could I expect after looking over a few source modules for less than half an hour?  Too little context.  But oh so much fun!

When I was a kid a several time TI user group members gave me boxes of old TI stuff including books like TI99/4A Intern, schematics, maintenance manuals and reverse-engineered specifications.  I can't count how many hours I spent absorbing all of that knowledge.  I was happy as a clam!  And I eventually put much of it to good use in various software and hardware hacks...

Here's some advice I often offer to programmers:  Read as much code written by other people as you possibly can.  Think critically about what it does, the problems it solves, how it was written, how it was tested, the team that created it, and the context in which it was born.  Use that insight to improve your own work.   (Besides, if you don't enjoy reading code, how are you going to survive doing it all day long?)

Corollary: Also read lots of books about code...

Wednesday, September 12, 2007

Semi-modal offline support

My life on the go

When I'm not working from home I commute to the office on public transit. On the way, I read lots of books (ok, and other fun stuff too. I also use my laptop to read blogs (too many to list), check email and write code.

Aside: Installing Visual Studio on my laptop was probably the best thing I have ever done for my family life. If I can take my code with me then I can keep hammering away at bugs while I'm communiting instead of hanging around at the office endlessly. I'd rather be sitting on my patio anyways with the WiFi.

The problem is that despite my GPRS connection (yeah, I wish I'd waited and gotten the EVDO model) I do go underground four times during my commute. So connectivity is a bit spotty. Consequently I'm in the habit of loading up FireFox with tons of tabs so that I have plenty of reading material for the whole trip.

Google Reader

Recently I switched to using Google Reader for reading RSS feeds. (Previous I had been aggregating the feeds using LiveJournal.) Google Reader now has a nifty offline support feature. When you switch it to offline mode it will download "up to 2000 items" and save them to your local disk. But there are problems...

The items have to be downloaded all at once. If it takes too long and you lose your connection or cancel the download, tough bananas. (Ironically, I also found the installer for the offline support plugin to be very flaky. If you're not online when you run it, it hangs on startup and eventually errors out.)

In search of graceful degradation

So what I'm going to pick on today is the notion that offline support requires a discrete mode.

Internet connectivity is always at least a little unreliable. Sure, most of the time your web application can issue an XML/HTTP request to some service provider and get back a response at most a few tenths of a second. But sometimes it will take much longer or it will fail. What should the application do?

So first off, forget about the statelessness of the web: that was only true back when we were serving up plain HTML documents. Today's web applications truly exhibit distributed state. The client has some of it and the server has the rest. Often client development is simplified by minimizing the amount of state it retains locally at the cost of responsiveness and robustness.

If you think this is contrived, think about all of the forms-based web applications you've used that are constantly doing round-trip calls to the server to submit changes made to the client's state to the server and figure out what to do next. Pretty horrible stuff! AJAX smooths the experience somewhat but the same kinds of things are going on underneath. It's all client-server.

So what happens when I pull the plug on my DSL connection while I'm writing a blog post. Well, Blogger periodically saves drafts to the server for backup purposes. But I can still make edits to the local state. Yet if my browser crashes for some reason, I'll lose them all because none of that state is persistent. (I'm speaking from experience here.)

Some years ago I read A Fire Upon the Deep. Vinge's universe places fundamental limits on the level of technological sophistication of different regions of the galaxy. At one point in the story there's a description of ships from the Beyond gracefully degrading as they enter the Slow Zone. Communication systems didn't fail outright but they compensated for narrowing bandwidth by reducing the quality. IIRC they adapted even to the point of transcoding conversations with talking heads and text-to-speech. That's the ideal...

The many shades of connectedness

Being offline means that there exists distributed state in a system that cannot be accessed or reconciled for an indefinite period. In between online and offline are many degrees of communication (un)reliability. Each degree of unreliability requires a corresponding measure of aggressiveness to tolerate. Here's a scale:

  • Perfect: The application's, reliability, bandwidth and latency requirements are fully met. Packets are never dropped. The plug is never pulled. You could run a Telnet session over UDP with complete confidence. This world does not exist!
  • Typical: There are occasional problems but they are easily compensated for by routine error correction protocols with at most minor glitches.
  • Unreliable: It becomes harder to mask errors. An application can still generally assume the problems are transient but it may become practically unusable unless it does at least some local caching.
  • Sporadic: User state will be lost or left dangling in mid-air unless the application compensates with local state management.
  • Offline: I pulled the plug! The application is on its own with whatever client-side wits it has left. In many cases, this is the kiss of death for an application and the user has to resort to typing up notes in Notepad until connectivity is restored.

A web application is ultimately at the mercy of the network. It can be enhanced to support disconnected operation, as with Google Reader, but there are significant complications. Just the same, why treat disconnected access as a special mode? Why not think of it as a very aggressive caching strategy with robust client-side storage? And if that's the case, why not enable it all of the time?

In other words, why doesn't Google Reader always cache feed items to my local disk whenever it can. I could certainly tell it to prefetch more data because I'm going offline for an extended time. Moreover if the synchronization process fails part way through, I should still be able to access whatever content I do have available. Why does offline support need to be modal? Why does Google Reader need to behave any differently from a desktop RSS client when disconnected from the network?

Thinking about web-enabled applications

I used the word enabled in web-enabled to effect a fundamental change of perspective. The web today is an excellent content delivery platform. Just the same, it is suboptimized for application delivery. On the web today, application and content are tightly enmeshed with one another but in reality they're very different. When I go offline, I don't mind if some of my content becomes inaccessible but I would still like to be able to use my web-enabled application to manipulate local content. That can't work so long as the application actually lives in a faraway server farm and I'm just interacting with served up views of its state when I'm online.

Ideally I want the web to deliver rich client applications whose code gets been cached on my local machine. I want applications that blur the line between the desktop and the web (including conforming to some sensible UI conventions). These applications would be enabled by the web and tightly bound to network services but they're not shackled to it. They can stand on their own albeit with reduced functionality. Ultra-portable applications: they run anywhere.

That's what I want!

Doom, death and destruction!

So who will be first to provide a compelling solution for lightweight rich-client application delivery over the web? Google Gears, Apollo, Silverlight or someone else?

Dynamic method interception with Mono.Cecil

For a couple months I've had an idea jangling in my head to use Mono.Cecil to instrument libraries to enable runtime dynamic interception of any method call (not just virtual ones). For example, this ability would be useful for extending Rhino.Mocks to support mocking static methods. It could also be used for tracing into unmodifiable code.

Of course, generally when you want to do these things you're doing something wrong. Let's set aside those objects for the moment. I've been curious to see whether the principle was sound and how much work it would be.

I tried hard to suppress it and focus on more important things. I was pretty successful at it. But I broke down over the weekend while I was writing a wee little code generator to help keep MbUnit Gallio Assertions all nice and consistent (I really wish C# had hygienic macros!). All that code gen on the brain was driving me to distraction. So I gave it a shot.

The results

1. The principle is sound.
2. It took about two days of heavy-duty hacking to get a proof of concept (while taking care of a bunch of other business). Pretty much what I expected but more than I'd hoped.

I've checked in code that uses Mono.Cecil to perform runtime interception in either of two different ways:
1. Statically instrumenting the assembly to include interception code that can be hotwired on demand. (easy)
2. Loading the assembly into the dynamic assembly context and patching method bodies on demand using MethodRental. (hard)

Moving forward

Generally the profiler API is better suited for dynamic interception but there are interesting scenarios where we can't use it (like in production code). So I can imagine there would be significant value to various projects if a variety of interception strategies were offered under a common simplified instrumentation interface. I'm sure the .Net AOP enthusiasts would put it to good use.

It was a good experiment but not immediately useful. (That's what hobbies are for...)

In any case, I don't plan on doing much of anything with this code anytime soon since I've got tons of other stuff on the go. Please let me know if you find this code of use or are interested in contributing to its continued development.

Monday, September 10, 2007

Multithreaded testing

Karl Seguin wants Better multithreading support needed in .NET. Yeah, I miss Java's thread-safe collections and fancy synchronization primitives too. I sometimes feel like my working vocabulary is being stunted for lack of a proper built-in CyclicBarrier class. I'm somewhat tired of reimplementing these kinds of things.

But I'd like to focus on the last thing Karl said:

At the top of my wish list though is support for multithreading testing. Multithreaded applications might be hard to write, but they are a nightmare to test. It's easy enough to run multiple threads over your code, but I haven't yet seen (which doesn't mean it doesn't exist), something that lets you transparently test specific race and deadlock conditions. Was a waiting thread properly signaled? Was a deadlock properly resolved? So far my approach has been simply to put huge loads on the application and hope problems surface.

Developing and testing concurrently accessed shared objects is hard! That's why CSP was invented. Just the same, the difficulties can't be swept under the rug.

Correctness

In these situations we really want to be looking for race hazards and deadlocks. We want to find possible interleaved execution paths that cause the contract of the shared object to be violated. It would be helpful to have a semi-automated tool to search for these cases. (Particularly if it also knew about asynchronous interruptions like ThreadAbortedException.) However, to guide the analysis we probably need a way to express the desired non-deterministic behavior as part of the shared object's contract. I know this kind of work has been done many times over using theorem provers over process calculi. I've yet to see one applied to ordinary code outside of a research environment.

Performance

But there's more to it than that. Besides correctness we are often also interested in performance. After all, since we decided to optimize throughput or bandwidth through increased parallelism we should really get some measure of success. Was the added complexity worth it? How do we know our fancy concurrently accessed shared object won't introduce an insurmountable bottleneck despite our efforts? It is hard to create a realistic load profile for synthetic benchmarks. Operations involving lots of I/O with external systems are especially nasty.

Ultimately I think the hardest part of performance tuning is the interpretation of the results. Moreover, performance optimization is a dynamic feedback process. As the system evolves previous observations become invalid. The best tuned algorithm at one time can suddenly lose its edge as the underlying platform evolves.

Trade-offs

I generally avoid premature optimization during development. Quite often what happens is that the performance of the simplest possible design is in fact quite acceptable. It's much easier to verify the correctness of a simpler implementation!

As for optimization, instead of using a raw performance metric like CPU utilization, I think in terms of scalability metrics. "Will this process handle the current 100 transactions per second? Will it need to be scrapped to handle the anticipated 1000 transactions per second?" Reframing the problem helps me avoid wasting effort on useless micro-optimizations. It keeps me focused on choosing a good algorithm first. Once that's done I'll start looking at cheap low-level improvements if the cost/benefit ratio is good enough.

Writing it down

As if deciding what to test and how wasn't hard enough, actually implementing a multi-threaded test is a real bear. You need to set up threads, catch unhandled exceptions, introduce synchronization barriers, handle non-deterministic events, tear it down, and somehow yield diagnostic output regarding the sequence of actions that actually took place so you can figure out what went wrong. Phew! I dread it every time.

Fortunately it doesn't have to be that hard. A couple of months ago I wrote down an idea for an MbUnit extension for "agent-driven testing" to help with all of this bookkeeping. The idea is think of each concurrent process that participates in a test as an agent. A test can spawn any number of agents of various kinds that each run in their own thread. If an agent terminates abnormally, the entire test including its agents is aborted with a failure result. Therefore agents can write to the execution log and evaluate assertions concurrently. Moreover, agents can access the shared state of the fixture. They can also communicate and synchronize via signals, condition variables and barriers.

What distinguishes this idea from other great multi-threaded test helpers out there is that it will leverage more of the test framework and consequently will be much more expressive. I'll say more about this later...

P.S. I made up the term "agent-driven testing". I'm not quite satisfied with the name. Surely someone else has come up with a better one.

P.P.S. I wish we could easily do run-time instrumentation of code in .Net. It'd be exceedingly cool to build something like ConTest. I have been thinking about using Mono.Cecil to do some interesting things at runtime though.

Sunday, September 9, 2007

The GUI looks good but how are her conversational skills?

Ayende asks How to sell maintainability and takes a stab at demo-ware. Somehow this got me thinking about female dating personae. (Disclaimer: This is in no way intended to be sexist!)

Persona: Demo software. It's her first night out on the town. She's nervous and is working to overcome her shyness. She wears a little too much make-up. She masks her insecurity by redirecting attention to her looks. She talks a bit too fast but she avoids engaging in deep conversation. She's afraid she might look stupid because she's inexperienced. She desperately wants to feel accepted and fit into the crowd. She tries to show off but she doesn't understand the rules by which she will be judged.

Persona: Mature software. She's sitting at her favorite restaurant. She's confident and relaxed. Her clothing is a little out of style but still shows a great deal of subtlety and careful selection. She knows herself and has pride in her abilities. She discourses openly and vividly about her life and goals. She takes suggestions seriously and is not easily flustered by the judgments of others. She is comfortable as she is. She knows she is valued by others and feels no need to be flashy except when it pleases her.

Revenons à nos moutons

Building maintainable software takes a long time and a lot of effort. It takes real commitment to build a productive relationship with the code base. It takes careful planning, patience and persistence. So in my mind the only way to sell maintainable software is to sell the value of that commitment. After all, the business organization will be "stuck" with it for a long time: it should invest the time and energy to ensure the relationship will be pleasant and fruitful even as the world turns and changes.

Kind of like marriage... ;-)