techblog 

I recently spent some time enabling my Ruby DNS library (Dnsruby) to use EventMachine.

Dnsruby comes with its own (pure Ruby) event loop. This allows all the I/O to be done in one thread, the event loop, while the client code runs in its own thread. However, given the limitations of Ruby’s threads and select, and the variety of behaviours that exist on different platforms, it’s hard to get a pure Ruby system to work perfectly everywhere.

EventMachine gets round this issue by coding the guts of the I/O in C++ (or Java). The Ruby client code calls out to the native extensions to actually do the I/O, and all the Ruby code lives in one single thread, thus eliminating any irritating shared data issues and avoiding the whole Ruby thread quagmire.

So, I thought it would be nice for Dnsruby to be able to use the EventMachine library, when it was installed on the local platform. It would also make it easy for EventMachine users to use Dnsruby to make DNS queries from within their EventMachine loop.

Here are a few notes about the development. Continue Reading »

Nominet’s nameservers are hosted by various Internet organisations (ISPs, IXPs, and others), but managed by Nominet staff. The zones are updated dynamically by a process which responds to DNS-affecting Automaton requests. This process runs at least once a minute, and the larger zones can change as rapidly as that. All of the eleven authoritative nameservers are monitored to ensure that the zones are up to date, within a small margin to allow for the propagation of changes to the various locations.

On the evening of Wednesday 10 October our monitoring system determined that one of them, ns7.nic.uk, was falling behind. This was brought to my attention by the on-call engineer. His first concern was whether this was something that we should include in our new Service Announcements RSS feed. Rather than deal with this issue, my focus was on stopping the nameserver from responding with potentially incorrect information. I feel that a DNS server which responds with an incorrect answer whilst claiming to be authoritative is worse than one that does not respond at all. Our resilience to nameserver failure is good enough that we can sustain an outage on one nameserver quite easily. My instruction was to prevent this nameserver from responding to DNS requests.

Once the nameserver process had been stopped on ns7.nic.uk we had to address whether this was something worthy of a Service Announcement. My decision was no, it wasn’t. The Service Announcement feed is for events that will affect, or are currently affecting, a large number of Nominet customers. DNS is a transparent service. The lack of DNS, or serious degradation of the service, would be enough to warrant an announcement, but not the loss of a single node.

So, what caused the failure? The company providing hosting for ns7.nic.uk has several different transit providers. It appeared as if one of these started blocking outgoing traffic from port 53 over both TCP and UDP at around 21:00 UTC. This had the effect that the nameserver was receiving notifies from other nameservers, but its update requests were being blocked at the border of the hosting company’s network. At the time of writing we are still not clear why this block was put in place, but we are chasing it. The hosting company tore down the BGP session with that transit provider and normal service was resumed. We are still in negotiations with all parties, so for now the session remains down.

I’ve been using Subversion for a good few years now, but I’ve never encountered this subtlety until now. Suppose your repository is at https://blah.co.uk/repository and currently stands at revision 1000. You would like to look at a file which is situated at oldproject/trunk/deep/down/in/the/bowels/interesting.txt as it was at revision 500. But in between then and now the whole of the tree under oldproject has been removed. My first attempt was to do this:

svn co -r500 https://blah.co.uk/repository/oldproject/trunk/deep/down/in/the/bowels/

But that doesn’t work, because that URL is no longer valid. At this point it looks like you’ll have to check out the entire repository at revision 500, which could take a very long time.

The answer to this problem is that you need to do is to use the special @ syntax to tell subversion at which point in the history to first look up the URL:

svn co https://blah.co.uk/repository/oldproject/trunk/deep/down/in/the/bowels@500

Note that the -r flag is not needed, but if provided it could be used to trace the history of this code back even further.

I’ve just had to configure several new power distribution units . The serial ports were hooked up to our terminal servers as normal but then I wanted to configure them all at the same time as the interface is menu driven. There is the well known Cluster SSH program but this wouldn’t work as I was connecting through to the terminal server then through to the individual port using a local port. I was typing ssh user:port@server and cluster ssh was interpreting this incorrectly and trying to ssh to port@server - i.e. the user it was trying to connect as was the port number. Rather than try and modify cluster ssh (it’s just a perl script underneath) I looked around for alternatives and found Keyboardcast. A quick aptitude install later and the job was done. You can either fire up new terminals with the tool or select existing ones to work with. The only downside I found was that it wouldn’t work with tabs in Gnome terminal. An excellent tool.

The instructions for a recent update to my Apple Intel MacBook Pro laptop asked me to boot from the installation disk after applying the update and run the check disk utility. I retrieved an OSX installation disk from our IS department, put it in the drive and, following the instructions, restarted the system whilst keeping the “C” key pressed. The DVD drive whirred away - but the system booted from the hard disk.

I then tried to reboot the system by selecting the DVD from the “Startup Disk” panel in the “System Preferences” utility and clicking on the “Restart” button. Again the DVD drive started up as the system restarted, but once again the laptop booted from the hard disk. I restarted the system for a third time, keeping the “Options” key (the left-hand “alt” key) pressed during the process. This brought up a screen displaying the two boot choices, the hard disk and the DVD drive. I selected the DVD drive, pressed “Return” … and the system again booted from the hard disk.

Thinking that the disk was faulty, I obtained another installation disk (we have a number of MacBook Pro laptops here) and tried that, but with the same results.

It was only a chance conversation with one of our staff who is an ex-Apple employee that solved the problem. To boot from an install disk, you need a disk that holds a version of the operating system equal to or later than the one with which the computer was delivered. So if your system was delivered with OSX 10.4.9, it won’t boot from a 10.4.5 disk. Presumably this is because of version compatibility issues between the firmware in the laptop and the drivers on the install disk. When I obtained an install disk with the right version of OSX, the laptop booted from it without any problems.

It would have saved a lot of time and trouble had the system displayed an error message when it detected the version mismatch.

This probably isn’t news to many but I only recently noticed that Oracle AQ can manage JMS queues. We don’t use a full blown J2EE application server so our options when using a JMS broker is to either have a standalone server/cluster (more work for SAs) or to embed the JMS server in to our middleware (more work for me). By using the database we add no new dependencies or software so this is a pretty neat solution. Another big advantage for us is that it looks like we would be able to create messages inside PL/SQL procedures but I may be getting ahead of myself here.

There isn’t much to it but it was a lot harder to get to the same point than with the other JMS brokers I have been testing. There is a lot of documentation to go through and it is all fairly general. I also couldn’t find a good example so I’ve created a quick howto… Continue Reading »