Like all plans, it was simple and effective. I even wrote it up on the whiteboard in clear steps:
Javablogs- Build dependant libraries
- Build Javablogs in parallel to existing site
- Migrate data
- Run first update
- Test
- Switch over
- Post announcement
Pretty simple, huh?
Well, except I screwed it up during step two. I carefully checked the new version of Javablogs out of CVS in the user's home directory instead of the deployment directory, so the old version could continue running in parallel until we were satisfied it was stable. The only problem was... a stray symbolic link caused the new copy to be checked out directly over the running version. Byebye website.
The smart thing to do would have been to immediately back up to the old version and rebuild what we knew worked.
I think this is final proof that I am not a smart person.
A few hours frantic bug-fixing later, I get most of the site working live. And then, of course, there were the things I had to fix later from home because I didn't notice before leaving the office.
The site still isn't entirely stable, but hopefully it'll make it through the night.
Update. It didn't last the night. It didn't last even the hour after I went to bed. Luckily, though, sleeping on a problem often brings you more enlightenment than stressing over it. So as a short memo to myself, and anyone else who might read this cautionary tale:
Never deploy a new version of a large public website with the maximum database connection pool size set to four. It's unlikely to do much more than stagger along for a few minutes and then expire messily.
Update 2 Yes, that seems to have done it.
