How do you deploy?

Ayende Rahien asked: How do you deploy?

In our case the CI build server produces a ZIP file containing the build image. Then we run a command on a master deployment machine which pulls the latest build ZIP (or a specified version) and uses the WSHController COM object to fire off commands on the individual servers to deploy that version, also providing additional information about the environment.

I wrote a prototype batch deployment tool using Ruby on Rails two weeks ago which will eliminate the step of remoting into the master deployment machine. This still needs to be under partial manual control because we use the same process in production but it's convenient to have all the commands in one place with some status screens. The next step is to investigate whethere more of the workflow should be integrated into it because production deployments require a significant amount of communication among teams.

The CI integration deployment eliminates the manual step and does not perform remote command execution. The CCNet instance on the build server fires a message over to force the CCNet instance on the integration server to deploy the latest ZIP'ed build image to the local machine and run the tests with MbUnit and NCover. Pretty simple.

Where it all breaks down...

The remote deployment trick with WSHController has worked fine for 5 years. Unfortunately we started having reliability issues with WSHController several months ago where it occasionally crashes or hangs. It appears to be due to a buffer overflow in its handling of environment variables. After many hours of trying to figure out how to fix or work around it, I just decided to rewrite the script using a different remote command execution tool. So yesterday I spent 2 hours rewriting the remote deployment script to use SysInternals ' PsExec instead. Then I spent the next 14 hours trying to figure out why it also crashes (C0000409) or hangs (sometimes, unless the output of the remote command is redirected to a file instead). As I can't see anything obviously wrong with my wrapper script (in Iron Python) I'm left wondering which patch, upgrade, or service pack might have caused my current woes. *sigh*

So... now I'm going to take a break, have lunch on the patio, watch the fish swim in the pond, try to do something about my sinus headache and get down to working on MbUnit v3.