wiredfool

Somehow, on top of all the other things that happened on vacation, something close to my worst sysadmin nightmare came up. A break in OpenSSL/SSH. It’s complicated, mission critical, and it can’t be kept away from the users, at least in the SSL case. I’d rate this a 4/5 in panic level. (a 5 would be a remote root hole in one of these services.)

Oh, wait, I haven’t talked about the vacation. Flying with a sick kid is not fun. Nor with 2. Staying in a hotel with 2 sick kids and 2 sick parents is even less fun. But it did get better after a few days.

Then, debian stable’s random number generator was found to be a little weak, so that the keys generated were extremely predictable. Trivially even. Which means that any key generated using openssl on those systems is suspect, any dsa key used on one of the systems is suspect, and everything needs to be updated quickly without locking myself out.

I did have a couple of things in my favor — while I was using dsa keys, they were generated on OSX, so they weren’t instantly bad. And I use ip address filtering on ssh where I can and fail2ban where I can’t, so attackers either get 0 or 5 chances to get in before their packets are dropped. Out of all of this, I think that there were 3 keys that didn’t need to be replaced, because they were putty generated rsa keys.

Issue #1: I’ve got enough different machines and images that I wanted to use something a little faster than one at a time to do the updating. it turns out that Capistrano is a good way to do that, but it doesn’t work on the stock OSX Tiger install, nor on my ubuntu 6.06 machine. But, eventually I figured out that it does work from MacPorts. But ou have to compile a bunch of stuff, which is a little slow on a g4/1.2ghz.

Issue #2: For some reason, there’s one essential package, libssl0.9.8 that doesn’t update well on debian without a terminal. It has a prompt for which services to restart, and will hang there if run from Capistrano. So, I had to log into all the servers and images to do the actual update.

At least Capistrano sped up the new key deployment, and will probably speed up things in the future, but for this operation, I don’t think that I netted any time savings.