wiredfool

On of my main linux machines was running Ubuntu 6.06LTS, which is a long term support release from the middle of 2006. It was getting a little long in the tooth (firefox 1.5 anyone?) and that was getting tiresome.

So yesterday, I bit the bullet and did the upgrade to the newest long term support release, from last spring. Following the directions worked well enough, except for a couple of packages in the openoffice 2.4 release that I had installed. There were a set of them with circular dependencies, so they couldn’t be removed, which in turn were blocking a newer bugfix of openoffice 2.4, in turn, that was blocking the package manager from doing anything useful like updating emacs from version 21 to 22.

Apt is a good package manager, but sometimes it gets into states that aren’t easily resolved. If you get conflicting messages about something being installed, that’s a good sign that you’re there.

The normal solutions didn’t work :

sudo apt-get -f install
sudo apt-get -f remove
sudo apt-get dist-upgrade

Were all hopeless.

What finally worked was downloading two upgraded packages directly (libhsqldb-java adn openoffice.org-base) and installing them manually via dpkg, and then clearing out all the circular references with:

sudo dpkg -r openoffice.org-kde-integration
sudo dpkg -r –force-all openoffice.org-core01
sudo dpkg -r –force-all openoffice.org-core02
…

Dpkg can force things that apt-get can’t, or won’t do. After that, sudo apt-get upgrade -f dist-upgrade was enough to make everything work properly.

Finally, there’s a package archive for the shiny new openoffice 3 packages, but it takes some digging to find the archive key and the keyserver.

##repo for open office3
deb http://ppa.launchpad.net/openoffice-pkgs/ubuntu hardy main
deb-src http://ppa.launchpad.net/openoffice-pkgs/ubuntu hardy main

gpg –keyserver keyserver.ubuntu.com –recv-keys 60D11217247D1CFF
gpg -a –export 60D11217247D1CFF | sudo apt-key add –

It’s running leopard, and it’s new, fast and all that. Which is all different from what I’ve had before.

– It’s fast. It’s amazing what going from 2.5 yr old laptop tech to current desktop tech will do, even if I didn’t get a quad core in it. And 2×24″ monitors is nice for programming.

– The screen is not as good as my Dell 2407WFPHC, even when calibrated. There’s a distinct shift to the color cast of the backlight from side to side, and the backlight intensity varies as well. It may be a subtle angle thing, I’m not sure I can tell. I’m also not getting it to be anywhere near as warm as the calibrated Dell, even when I’m using the same target profile for both monitors.

– Waking the external display from sleep goes to static. It’s the first time I’ve seen a machine snowcrash.Even if it does just mean that I have to powercycle the display.

– Fast user switching and terminal and multiple monitors don’t play well together. All the terminal windows end up on the main display when switching back. Maybe I should just use xterms, since they don’t have the same problem.

– iTunes stops playing when I swap users using fast user switching. PITA when I just want to check my mail. Also, I can’t keep a phone call on gizmo through switching users. What I’d really like to do is dedicate one ‘space’ to each user, and just switch without login issues and such. Not sure that’s possible though.

– I can’t control the headphones and the speakers separately. I’d love to do something complicated to the sound routing like: Alerts-> speakers. Incoming Phone call rings -> both. Music -> switchable. But most importantly, incoming phone rings always to the speakers, no matter what, so that I can hear it if I’m not at my desk. And it’s not like it’s exactly easy to find the headphone jack, since it’s behind the computer.

I’m a moderator on a really big flickr group, and we’ve been running out of human power to do the moderation. There’s just too much traffic. So, My mind goes to automation and whatnot, and I’m winding up building something to help.

It’s essentially a webapp, in python.

I’m reminded that while the names have changed, we’re really no better off than we were 4 years ago in the python webapp space. There are tons of frameworks, incorporating any number of competing templating engines, object-relational mappers, and other middleware. It’s just that they’re now a bit more rails influenced. I’d use my own templating and ORM, but it’s all work related now, and I don’t want that dependency now. This will wind up being open-sourced at some point soon. And it’s a good learning experience to work on one of the more modern packages for a while, even if it does make things harder in the short run.

And they still don’t deploy easily on commodity hosting. You’d think that the Rails Revolution would helped that part, as it’s pushed fcgi and proxying to places where id hasn’t gone before.

But really, if I write PHP, it just works on any webhost. You’d be hard pressed to find a webhost these days that can’t do a minimal php/mysql website. I farted with python for two evenings on Dreamhost, and kept running into strange problems that might be the framework, might be the web adapter, and might just be the shared environment. (fcgi must be group writable? really?) So far, I’m 5 evenings into it total. So 2 evenings is a pretty good setback in the scope of this project.

More to come on this project later.

For reasons that may be obvious, sometimes I want my servers to tell me something, and get my attention in a manner a bit more direct than via email or rss. (If I’m busy, I’m not checking my email. If I’m checking my rss, I’m not busy) And for reasons of geography, sms messages don’t work. Well, cells don’t work, so I don’t actually carry my cell, so. Well, It’s just not effective.

So, I need a couple levels of notification above trawling email or logs. One to interrupt me with bad but not fatal problems, and one to get my attention if I need to know now.

The lower level, interrupt me, for bugs that really shouldn’t exist, problems with important customers, or things that I need to be bothered about until I fix them. The more severe stuff is akin to the raid array losing a drive or deciding to take an impromptu vacation.

So. The best thing I’ve found for notification for things that I want to know is Growl. It pops up a little notification on screen, politely, and it can receive messages over the net. Unfortunately, it’s UDP, and there are a couple of NAT boxes between me and the servers. If it was TCP, it would be a simple matter of remote port forwarding with ssh. I always have a connection open, so it’s just a matter of adding yet another port through there. But it’s UDP, and ssh doesn’t forward UDP.

There are solutions out there using netcat (here and here and so on) all of which tie in a few netcat processes into the tunnel to listen on UDP, send TCP through the tunnel, Listen on TCP on the other end, and forward UDP to the end destination.

That should work, but what I was finding was that the netcat listener on the far end would listen for exactly one UDP connection, and then stop. Once the initial socket closed, nothing further would get through. A few hours of mucking around with this while dealing with other interrupting issues and I had enough. 10 minutes later, I had a pair of python servers that would do the UDP and TCP listening. So far they’ve been far more reliable than the netcat solution.

This runs on the far end:

#!/usr/bin/env python
import SocketServer
from socket import AF_INET, SOCK_DGRAM, SOCK_STREAM, socket
import sys

GROWL_PORT=9887
TCP_PORT=19887
DEST=’192.168.10.98′

class growl_udp_handler(SocketServer.DatagramRequestHandler):
def handle(self):
addr = (“localhost”, TCP_PORT)
s = socket(AF_INET,SOCK_STREAM)
s.connect(addr)
s.sendall(self.rfile.read())
s.close()

u = SocketServer.ThreadingUDPServer((‘0.0.0.0’,GROWL_PORT), growl_udp_handler)
u.serve_forever()

and this on the local side.

class growl_tcp_handler(SocketServer.StreamRequestHandler):
def handle(self):
addr = (DEST, GROWL_PORT)
s = socket(AF_INET,SOCK_DGRAM)
s.sendto(self.rfile.read(),addr)
s.close()

t = SocketServer.ThreadingTCPServer((‘0.0.0.0’,TCP_PORT), growl_tcp_handler)
t.serve_forever()
I’m hoping that this is reliable enough that I spend more time debugging the subject of the notifications rather than the notifications themselves.

This has been a missing part of my photo workflow for a while:

Daring Fireball: Writing AppleScripts That Dynamically Target Either Safari or WebKit.

My workflow involves opening a page of pics in safari, looking at bigger ones, then dumping the whole lot into lightroom. And that only works in Safari, because I haven’t gotten around to making Webkit work.

While trying to setup a key for the last keyboard macro in emacs (from Steve Yegge’s emacs page), somehow I got my f key bound to a game of 5×5 rather than binding my f5 key to the macro.

Makes things difficult. Rather than doing the easy thing by quitting and restarting, I made myself fix it. There’s a useful function called M-x describe-key which will give you the binding for a key. From that, I found that letters were bound to self-insert-command, and that the f was indeed bound to something else entirely.

So evaling this: (global-set-key "f" 'self-insert-command) gave me back my f key. Of course, I had to copy and paste in the effs in that command.

Python’s ducktyping is very useful. It’s good to be able to generally treat something as a file, or a string even if it’s not exactly a file or a string. But when there are methods with the same signature(*) whose parameters change meaning, then you’ve got a problem.

When you have a string, you can use the string.translate(mapping) call to translate characters — to do things like force all whitespace to be a real space, or drop everything in the 8 bit zone. Unicode strings have a similar method, unicode.translate(mapping).

Except that the mapping is totally different for strings and unicode strings. The string translate takes a 256 character string, and the unicode one takes a hash. This leads to the error:
TypeError: character mapping must return integer, None or unicode
which is especially fun when you don’t know if you have a string or a unicode string, and stuff just worked before.

This is a working solution for the unicode case. Instead of mapping = string.maketrans('\r\n\t',' '), you need mapping = dict(zip(map(ord, u'\r\n\t'), u' ')) .

(*) well, technically the type would change, but you don’t see that in a non-statically typed language.

We’ve occasionally had issues with a database machine hanging at boot waiting for
a PepperC Usb Mass storage device. Turns out that it’s part of an IPMI
card for supermicro motherboards, and for some annoying reason, it was
asserting itself as the root device.

This is a big server, with a raid card, a bunch of drives and a couple of logical arrays off of that card. It’s meant to boot off of one of the arrays, on /dev/sda.

What is happening is that the SCSI and the USB are probed in parallel and there’s a race condition as to which one responds first and therefore gets the sda designation. The next gets sdb, then sdc, and so on. Normally, drives and partitions are referred to by that
designation. Normally, the order is static, and everything is good.
But if something takes longer than normal, perhaps due to an extra
drive in the case or something, then the other one has a chance to
take over.

And then you’re trying to boot off of something that you don’t expect,
like the newly inserted big drive for backups, or the PepperC device. (Good thing that the IPMI card has KVM over ip. It’s nice when a piecce of equipment solves aas many problems as it causes. )

So, the other way to refer to drives is by uuid, a universal id. All
drives and partitions have a unique 128bit id, which makes it a bit harder to have some interloper of a device register a second or two early and mess everything up. This stable root device (AKA UUID) walkthrough worked for me with Debian Stable (etch) with the caveat that I had to mkswap again, as my swap partition didn’t have a uuid associated with it originally.

Incidentally, this IPMI card also will prevent booting if its event log is full by displaying a ‘Press F1 to continue’ prompt on the console. I’m at the point where I’m not sure if the IPMI card helps or hinders reliability, and I’m likely not to put any in machines in the future.

On a fast connection to S3, I consistently get socket errors (socket.error: (104, ‘Connection reset by peer’)) when I’m pushing in big files. (big being ~ 250mb or so). Totally reproducible with the same file uploaded. Turns out that people have been having this issue for a while, and the solution is to throttle the tcp parameters so that it doesn’t overrun a tcp window somewhere.

This has the solution. In etc/sysctl.conf:

# Workaround for TCP Window Scaling bugs in other ppl’s equipment:
net.ipv4.tcp_wmem = 4096 16384 512000
net.ipv4.tcp_rmem = 4096 87380 512000

Recently we shifted the core of the webapp at work to run on lighttpd/php5 connected by fcgi from the standard apache1.3/mod_php4. There have been a couple of cases where people using the POST api methods have consistently gotten a connection closed error with lighttpd version where everything worked properly on apache.

This is the post:

POST /api.php HTTP/1.1
Content-Type: text/xml
Host: www.example.com:443
Content-Length: 19742
Expect: 100-continue
Connection: Keep-Alive
X-Forwarded-For: 10.0.0.0

<TransactionRequest>
...

and this is the response from lighttpd:

HTTP/1.1 417 Expectation Failed
Connection: close
Content-Length: 0
Date: Thu, 30 Oct 2008 19:35:29 GMT
Server: lighttpd/1.4.13

and the response from Apache:

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Date: Thu, 30 Oct 2008 19:37:26 GMT
Server: Apache/1.3.34
X-Powered-By: PHP/4.4.4-8+etch6
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/xml

<?xml version="1.0" encoding="iso-8859-1"?>
...

To make matters worse, it appears that pound, our front end load balancer, doesn’t log the error, but it does show up in the lighttpd logs as expected.

So, what’s the problem? It’s the expect header. Lighttpd doesn’t support it, and sends back a 417 Expectation failed error if that header is included. It’s a known WontFix bug for the 1.4 series, and the 1.5 series has been expected for a long while now.

The thing is, there are a lot of command line http clients or libraries out there that send this header, and it’s distressing that lighttpd simply can’t work with them.

And then, what’s the fix?

I’m afraid that it’s goign to be to route around lighttpd for now, either by replacing with apache2 or ngnix. There’s an outside possibility that I’ll be able to filter the header on the load balancer, but that’s not looking likely right now.