wiredfool

I’ve been using spread and whackamole for IP address failover on Debian Etch, but I’ve noticed that they’re not 100% reliable in my setup, in some cases hanging when I’m trying to query who owns a node, and in others, just not picking up the ip address that’s shared between two machines.

What I’m doing involves an active machine (or vmware instance) and a hot spare running the same code and ready to take over instantly. There’s generally very little state, as these are inbound load balancers or outbound gateways. Where I have lots of state, like database, the failover mechanisims are a little different.

I’m testing out ucarp as a replacement. It has it’s good points — it’s one piece that’s reasonably easy to setup, and bad points that it’s not exactly high profile. The current version (1.5) is a released version of the snapshot (1.5_0802222) in lenny/testing and has some significant bug fixes, so I’ve pulled the source package in from lenny/testing and the distribution from the main distribution site and compiled it on etch for a quick backport. There’s one patch line that failed — the options accepted on the command line, and that appears to have been integrated in the upstream package.

The package from lenny worked reasonably well in testing, but showed some issues in production related to not being able to “Can’t do IP\_ADD\_MEMBERSHIP”, which is necessary to recieve the multicast packets. This was fixed in 1.5 final.

Ucarp is a little different than wackamole. Wackamole just grabs an ip address when it detects that it needs to, Ucarp runs a script when it determines that the ip address needs to be aquired or dropped. This was something that I had wanted with wackamole for poking at daemons and such. The debian port has integrated the control of ucarp into the /etc/network/interfaces file, so straightforward configurations can be done completely in that file.

In /etc/network/interfaces

auto eth0
iface eth0 inet static
        address 192.168.10.91
        netmask 255.255.255.0
        gateway 192.168.10.1

        ucarp-vid 90
        ucarp-vip 192.168.10.90
        ucarp-password foobar
        ucarp-advskew 1
        ucarp-master yes
        
# ucarp controlled.
iface eth0:ucarp inet static
        address 192.168.10.90
        netmask 255.255.255.0
        

The options on ucarp bear explaining:

* vid: 90 an id. ucarp will communicate with any other instance using this id, so this needs to be consistent between the master and the spare
* vip: the address to be managed
* password: A shared password between the instances
* advskew: Lower numbers will win the election of which instance should be the master
* master: if this instance should startup as the master

This approach seems to work well for some tests, the failover time for pinging is 3-4 seconds total. But for apachebench and funkload, it seems that once they get an arp of the machine that’s getting taken over, that one stays resident until the next invocation of the app.

One of the things with webapps now is that they load a ton of extra stuff, javascript, css, images, static pages. Lots of extra stuff. The work site has something like 15 includes of one variety or another on each page, and a bunch more that are loaded as needed on one or two pages. That’s a lot of latency.

One way to fix it is caching and expires headers, another is to minify and compress the javascript and css, and another way is to just get the files near the user so that they can load all the items without round trips to the server. Big sites do this with a CDN, a content distribution network. But two drawbacks of that approach are that they’re not from the same site for origin purposes, and you generally have to pay for a CDN, and that’s out of the reach of smaller sites. On the other hand, they do it without you noticing, and gears does require confirmation and trust.

Google Gears has a local server and cache space that are intended for use in taking applications offline for later syncing with the cloud. But it’s just as useful for serving bits in conjunction with the online version. The important thing is that everytime there’s a request for one of the files in the cache, gears will serve it locally. That’s a lot less latency.

Best of all, it’s pretty easy to setup. You include the gears_init.js file, create a local server and assign it a manifest. The manifest controls everything that gears will download once and cache, until the version number in the manifest changes.

The javascript changes, cribbed from wordpress’s version of this and the google docs.

<script type='text/javascript' src='gears_init.js'></script>".
<script type='text/javascript'>
       if ( 'undefined' != typeof google && google.gears ) {
             var localServer = google.gears.factory.create('beta.localserver');
              var store = localServer.createManagedStore('cdn-store');
              store.manifestUrl = 'gears-manifest.txt';
              store.checkForUpdate();
       }
</script>

And the manifest file:

// site-manifest.txt
{
  "betaManifestVersion": 1,
  "version": "1",
  "entries": [
    { "url": "gears_init.js" }
	{ "url": "common/prototype.js" },	
	{ "url": "common/scriptaculous.js?load=effects" },	
	{ "url": "common/effects.js" },	
	{ "url": "and so on and so on...." }
  ]
}

I’m not sure that this is a total game changer for the general web, but for sites that are apps where the users both trust the site and spend a significant amount of time there, it’s a win. This is already being used for the admin interface of WordPress 2.6. I can see it being used in a lot of places with rich interfaces where the users are highly invested in the site and latency is a concern.

Of course, it’s possible but it’s not quite as easy as saying mv *.txt *.html.

So that I remember this:
for i in \`find *.txt\` ; do mv $i ${i/txt/html}; done

references: here and here

I’ve seen a dramatic uptick in signups to this blog, from what look like junk emails @ gmail, .ru, and other places. I’m a little confused, since there aren’t even attempts to spam me. The only thing I can think is that someone is building up a stash of wordpress logins for the next time that there’s a sql injection attack that can be performed by a logged in user.

What I’d really like to do is add a field to the signup page, simply asking: Why?

But then, some robot would probably try to convince me that it’s human.

Somehow, on top of all the other things that happened on vacation, something close to my worst sysadmin nightmare came up. A break in OpenSSL/SSH. It’s complicated, mission critical, and it can’t be kept away from the users, at least in the SSL case. I’d rate this a 4/5 in panic level. (a 5 would be a remote root hole in one of these services.)

Oh, wait, I haven’t talked about the vacation. Flying with a sick kid is not fun. Nor with 2. Staying in a hotel with 2 sick kids and 2 sick parents is even less fun. But it did get better after a few days.

Then, debian stable’s random number generator was found to be a little weak, so that the keys generated were extremely predictable. Trivially even. Which means that any key generated using openssl on those systems is suspect, any dsa key used on one of the systems is suspect, and everything needs to be updated quickly without locking myself out.

I did have a couple of things in my favor — while I was using dsa keys, they were generated on OSX, so they weren’t instantly bad. And I use ip address filtering on ssh where I can and fail2ban where I can’t, so attackers either get 0 or 5 chances to get in before their packets are dropped. Out of all of this, I think that there were 3 keys that didn’t need to be replaced, because they were putty generated rsa keys.

Issue #1: I’ve got enough different machines and images that I wanted to use something a little faster than one at a time to do the updating. it turns out that Capistrano is a good way to do that, but it doesn’t work on the stock OSX Tiger install, nor on my ubuntu 6.06 machine. But, eventually I figured out that it does work from MacPorts. But ou have to compile a bunch of stuff, which is a little slow on a g4/1.2ghz.

Issue #2: For some reason, there’s one essential package, libssl0.9.8 that doesn’t update well on debian without a terminal. It has a prompt for which services to restart, and will hang there if run from Capistrano. So, I had to log into all the servers and images to do the actual update.

At least Capistrano sped up the new key deployment, and will probably speed up things in the future, but for this operation, I don’t think that I netted any time savings.

I can never remember this when I’m looking for it.

If you have a signing key (not the primary) and an encryption key (the default key) for gpg, then when you need to sign a new key, the command is:

gpg --default-key ###### --sign-key ######

With the new webkit with safari 3.1, I’ve noticed one really annoying Mail.app regression — not the one that Daring Fireball found.

When pasting in ascii text, into a plain text email, all formatting is lost, where by formatting, I mean whitespace is collapsed.

Quite often I paste in the results of sql queries, and they’re typically aligned in columns with a monospaced font. Always worked until the upgrade, now I get everything on one line with different amounts of whitespace.

Pain in the ass really.

The upgrade instructions for this version are pretty much the same as any other version.

So, what that means is that you install it, then wait 1-4 days, then install an urgent security patch?

I’m probably going to give it a shot tonight. Dunno if i’ll regret it in the morning though.

I had the need to check a debian install for corruption recently when a newly installed and configured server started crashing under heavy IO activity. These days, when debian stable is kicking out kernel oopses, it’s most likely bad hardware, and if random wierd stuff is happening, look at the memory first. And surely enough, they found bad memory in the server. Since bad memory can hose a lot of things, I wanted to check the installed packages to make sure that there wasn’t any latent corruption.

I’d done this before on RedHat systems with a rpm command (rpm -Va verifies all installed packages against the manifest md5 sums), but dpkg doesn’t have an equivalent command.

But there is a package, debsums, that does exactly what I needed.

sudo apt-get install debsums
sudo debsums -l

Should show a listing of all the packages where there’s no hash on file.

debsums −−generate=nocheck −sp /var/cache/apt/archives Generates sums from the installed packages that are still in the cache.

Run sudo debsums -l again, This shouldn’t list any packages this time — if there are still some, you may need to redownload some with the command: sudo apt−get −−reinstall −d install `debsums −l`.

Then: sudo debsums −ca should give a listing of all the binaries that are different from the installed versions.

** Warning, as the man page notes, this is more a check for corruption and not a substitute for a malicious activity checker like tripwire.

As will become somewhat obvious over the next few posts, I’ve been setting up new database servers, and have crossed a sort of a threshold. IO bandwidth completely outstrips the ability of a single core to do anything interesting or useful with it. I’ve never had a database server that was not IO bound, and to be fair, this one probably isn’t once you factor in all the cores. I’ve also never had an 8 core system before.

This is a dual quad core server with low power chips, from Intel. The individual core’s aren’t speed daemons, but for the most part, I’m looking at lots of parallel operations so I’d rather have the parallelism than the individual core speed. The drive system though, it’s a LSI Megaraid with 512MB battery backed cache and 6 15K SAS drives in a raid 10 configuration. It’s good for better than 200 MB/sec continuous reads or writes, and about 130MB/sec per character writes. 1000 seeks/second. (and 450mb/sec reads on a 12 drive raid 10, and 200 meg/sec single character writes) Fast. Nice for DB loads. Nice for just about anything that needs speed more than space.

However, the parallelism rears it’s ugly head. This is the wide finder problem that Tim Bray has been talking about.

Gzip only runs on a single core, and while it’s throughput varies, it’s in the range of 4MB/second/core. If we could use all the cores, we still couldn’t keep up with the drives, but at least we’d be able to zip large backups reasonably fast.

Ssh, one core. Though, I’m seeing 45 MB/second/core there, it should be nearly possible to saturate the drive system and the dual gig-ethernet with the combined processor power. One of the use cases on the server is to blat the whole database from one to the other using rsync when reestablishing the secondary for fail over. 4x Faster would be nice.

Database dumps. Database Restores. Processor constrained again. Even on the old machine, a 3 yr old dual opteron, dumps are processor constrained at least when writing to a network or reading a toast table. This is annoying, since one non-negotiable bit of downtime is the time to dump from the old machine and load on the new one. It turns out that we can speed things up here to some extent by dumping each table on it’s own, with several in parallel. My tests are showing that for a database that’s dominated by 3 tables, the time to dump and reload becomes dominated by the time to dump and reload the largest table, rather than the entire database. Here that’s about a 25% savings, as there’s one massive table, one simply large, and quite a few under a hundred megs.

Recreating the indexes is another place where we redline one core, and hardly stress the drives. I think that I can probably manually split the index generation into separate threads, but I’d have to be careful to not split individual tables into different files.

These aren’t ordinary operations for a running database, but they are operations that tend to happen during down time when everyone’s looking at the clock and getting nervous. And it’s only going to get worse on the hardware side, since there’s no indication that anyone’s going to be rolling back to big honkin single cores. And the data’s certainly not getting smaller. So, it’s time for the software to get a little smarter.