wiredfool

For reasons that may be obvious, sometimes I want my servers to tell me something, and get my attention in a manner a bit more direct than via email or rss. (If I’m busy, I’m not checking my email. If I’m checking my rss, I’m not busy) And for reasons of geography, sms messages don’t work. Well, cells don’t work, so I don’t actually carry my cell, so. Well, It’s just not effective.

So, I need a couple levels of notification above trawling email or logs. One to interrupt me with bad but not fatal problems, and one to get my attention if I need to know now.

The lower level, interrupt me, for bugs that really shouldn’t exist, problems with important customers, or things that I need to be bothered about until I fix them. The more severe stuff is akin to the raid array losing a drive or deciding to take an impromptu vacation.

So. The best thing I’ve found for notification for things that I want to know is Growl. It pops up a little notification on screen, politely, and it can receive messages over the net. Unfortunately, it’s UDP, and there are a couple of NAT boxes between me and the servers. If it was TCP, it would be a simple matter of remote port forwarding with ssh. I always have a connection open, so it’s just a matter of adding yet another port through there. But it’s UDP, and ssh doesn’t forward UDP.

There are solutions out there using netcat (here and here and so on) all of which tie in a few netcat processes into the tunnel to listen on UDP, send TCP through the tunnel, Listen on TCP on the other end, and forward UDP to the end destination.

That should work, but what I was finding was that the netcat listener on the far end would listen for exactly one UDP connection, and then stop. Once the initial socket closed, nothing further would get through. A few hours of mucking around with this while dealing with other interrupting issues and I had enough. 10 minutes later, I had a pair of python servers that would do the UDP and TCP listening. So far they’ve been far more reliable than the netcat solution.

This runs on the far end:

#!/usr/bin/env python
import SocketServer
from socket import AF_INET, SOCK_DGRAM, SOCK_STREAM, socket
import sys

GROWL_PORT=9887
TCP_PORT=19887
DEST=’192.168.10.98′

class growl_udp_handler(SocketServer.DatagramRequestHandler):
def handle(self):
addr = (“localhost”, TCP_PORT)
s = socket(AF_INET,SOCK_STREAM)
s.connect(addr)
s.sendall(self.rfile.read())
s.close()

u = SocketServer.ThreadingUDPServer((‘0.0.0.0’,GROWL_PORT), growl_udp_handler)
u.serve_forever()

and this on the local side.

class growl_tcp_handler(SocketServer.StreamRequestHandler):
def handle(self):
addr = (DEST, GROWL_PORT)
s = socket(AF_INET,SOCK_DGRAM)
s.sendto(self.rfile.read(),addr)
s.close()

t = SocketServer.ThreadingTCPServer((‘0.0.0.0’,TCP_PORT), growl_tcp_handler)
t.serve_forever()
I’m hoping that this is reliable enough that I spend more time debugging the subject of the notifications rather than the notifications themselves.