techblog 

I like cycling and walking. What I also like to do is record where I have been. My Nokia N95 has built-in GPS, but it is pretty poor reception. Besides, the software I use to record where I have been, AFTrack, doesn’t work with the internal GPS. So I now use a GPS receiver.

AFTRack is fine as a mechanism for recording GPS tracks, but I wanted to have a way of viewing them after the event. Of the various formats available for export I have only played with KML and GPX so far. Google Earth is the natural app for viewing KML files, and very good it is too. But there is a great site, Trailguru, which accepts GPX uploads and shows you altitude and speed profiles for the whole track. The intention of the site’s creator, Tim Park, is to build a database of tracks through a wiki interface. This will enable you to plan routes based on existing track segments, and to average over repeated track uploads, reducing the effects of GPS ‘jitter’.

You can see my first upload, a Sunday bike ride with my son, here.

The UKOUG have an annual conference every year in Birmingham, and this year they are celebrating their 25th Anniversary. The conference attracts internationally renowned speakers and provides a wealth of information to DBAs of all levels of expertise.

This year has seen a huge number of abstracts submitted for a quite limited number of speaking slots, in fact there are speaking slots for only 30% of submitted abstracts.

This year I’m delighted to say that Nominet have had 2 abstracts accepted for the conference and is another example of Nominet’s commitment to sharing and exchanging information.

A lot has been said in the last few weeks about the need to randomise the source port used to issue DNS queries to mitigate the risk of cache poisoning attacks.

The need to randomise the 16-bit Query IDs was recognised long ago, and is implemented widely.  However investigation of the DNS traffic hitting Nominet’s .uk name servers shows that even now Query ID randomisation is not 100% correctly implemented!

A snapshot was taken of all incoming traffic to one of the .uk authoritative name servers for a 10 minute window around 12:00 BST (11:00 GMT) for each of the last six days, and then filtered to exclude any client that issued less than 500 queries in total across those sampling periods.

The results below showed 16 clients were found to be only using Query IDs in the range 0 - 16383 (i.e. 14 bits) and another two only using the range 0 - 32767 (15 bits).

Servers using the full 16 bits for Query ID have a typical standard deviation in excess of 17,000.

It’s likely that those using only 14 bits are running some version of MS Windows, and have not been patched following Microsoft’s Security Bulletin MS08-020 published in April this year. This bug was originally reported to Microsoft by my colleague Roy Arends.

Further investigation of the data even found one low volume client only using Query IDs in the range 0 - 127, a mere 7 bits of entropy!

For some time now, I’ve been convinced of the inherent evil of threads. I’ve gone to some lengths to mitigate their use in projects over the last few years. For example, my dnsjnio package uses java.nio to conduct many thousands of concurrent DNS queries in just a couple of threads. This package has been pretty stable for the last couple of years, and I know that it is in reasonably heavy use at several sites (at least).

So, I was very surprised to discover it has a nasty race condition!

I was alerted to this by a very helpful user. I’m still in the discovery phase, so certainly no fix yet, I’m afraid. However, the problem seems to be a race condition between a timer thread (which looks after query timeouts), and the select thread, which handles the I/O. A separate thread is used for timers because so many queries may be outstanding at any one time - and the select thread is busy enough handling I/O to worry about ordering timeouts. If a timeout is generated immediately after data has been received, but not yet promulgated to the user, then a connection can be closed twice, leading a NullPointerException (oops!).

Of course, some sort of lightweight Actors implementation would have been ideal here - in Erlang, I would have an Erlang process maintain the timeout queue, and the select() function ask it for the next timeout before starting the loop. And with a message-passing system, it would be simple to detect that the close had already been processed, as messages are processed sequentially.

Away from the details of this particular issue, I think there’s a broader point to make. It seems that even a relatively simple project, with lots of use, that has been designed to avoid the evils of threading, is still likely to contain some nasty threading bugs.

I will be coding future projects to avoid threading altogether. Well, that’s maybe a bit extreme. But I’d certainly want to design all my threads around blocking queues, with no other means of interaction.

My favoured design for my next project involves a collaboration of many communicating sequential processes - one on each core. Each process may be written in whatever language is best for the functionality of that process - C for speed-critical areas and the JVM or Erlang for most others. Of course, once you design your system like this, it’s then easy to move some components onto the network.

Some might invoke the generic tenth rule, and say that I may as well use Erlang for all of the nodes. I would agree, if there weren’t such a small amount of communication required between the nodes (for this project). And if my need for speed in certain areas wasn’t so acute.

I shall also be looking for other ways to learn from the functional languages - for example, enforcing immutability (perhaps with something like this).

If anyone is particularly interested in this, I’d recommend this paper which I was pointed to recently. I agree with almost everything in it - especially the recommendation to avoid C where at all possible.

Anyway, enough rambling - I’d better get back to fixing my nasty threading bug!

Last week I attended an ITIL v3 Foundation Training course. Non technical training courses usually leave techies a little bored and uninspired but for me this definitely was not the case. All the stories I have heard from colleagues in the past is that ITIL is a bad thing, has huge overhead and costs huge amounts of money to implement. From what I was taught last week I think this is a reflection on the way ITIL has been implemented and not a criticism of ITIL itself. One of the cornerstones of what we were taught last week is to pick up the pieces from ITIL that are relevant to your business and change/apply them to your needs, this seems like a very good philosophy to me.

Anyone who has done much looking into these kind of frameworks will tell you the cost is high mainly due to the expansion in human resources which can be huge and it is certainly true that looking through the course notes (and if you are really sleepless the ITIL books themselves) that it seems to be a job creation scheme, there are new roles all over the place, however they are just that ‘roles’ many of these tasks can and should be taken on by somebody who has another job and can dedicate a little of their time to this ITIL role to ensure that things are running smoothly from an ITIL perspective. Of course this depends on the size of the organization in question, but in the case of Nominet I would imagine that one person could indeed take on several of these roles, the important thing is that the roles and responsibilities are clearly defined.

ITIL version 3 has expanded from the previous version to now include Service Strategy and Service Design and as these were new topics there was quite a lot of focus on them in the three day course. These subjects however I believe are of more interest to a management and/or software development audience, my interest mainly was on the Service Operation angle of the course as it is in that area that I have spent the last 13 years of my working life. I was very interested in ITIL’s take on Incident, Problem and Change Management, these are all critical areas within the operation of any critical service (IT or Non IT) and doing these efficiently in a repeatable manner as ITIL suggests is very important for any service based organisation, to be fair I was already aware of this but ITIL does enable you to easily spot any weakness in the way you operate and most importantly use the experiences of other people.

A third person has now asked me about using Dnsruby to write an authoritative nameserver. It seemed worth jotting some notes here.

A nameserver at its most basic simply provides a mapping service between names and addresses. Of course, the RFCs that specify DNS have more than that to say on the subject, even at the start. And, in the 25 years since the first DNS specifications, there’s been a whole load more! For example, an authoritative nameserver might support views, ACLs, different flavours of zone transfers, dynamic updates, etc. And then there’s the cryptographic extensions : TSIG, DNSSEC and so on.

If you want the latest, fully RFC-compliant authoritative nameserver, then your best bet is definitely just to download BIND or NSD. It will run a lot faster than anything you can do with Ruby, and will support most of the currently fashionable DNS features. You won’t even need to debug it! ;0)

However, I don’t think that the people asking about Dnsruby are necessarily after a fully-featured nameserver : “I would like to get some kind of simple ruby dns server running, authoritative-only, for development purposes as it can interact nicely with existing ruby infrastructure and i have no need for high volume”

Of course, you could ask why you’d need to use Dnsruby at all - after all, if only an A record is to be requested, how much DNS functionality is required to respond? I guess there are two reasons :

1) Code reuse. Even if only a few lines of the parser in Dnsruby are used, it still beats re-implementing (and debugging) them.

2) Future extensions. You might not think you’ll want that fancy new feature, but who knows how things will change in the future? The Dnsruby library already provides support for may DNS features (most notably those related to security).

One issue is that I wrote Dnsruby as a client-side library. So, although it supports zone transfers, it only supports initiating and receiving them - not serving them. Currently, support is only there for verifying DNSSEC signatures (although very little code would be needed in order to support signing packets). Similarly, there is currently only support for sending queries; there is no listening server. And there is certainly no data store (or cache).

Anyway, the question was : “If I could have dnsruby sit on a machine (using EM - awesome!) and receive dns requests, and I could put some hook code in there somewhere to look it up in a DB or YAML file or whatever, it sounds like I could do that, but I just can’t find where to begin.”

I think most of the answer is in the question! The main tasks would be to write a server to listen for incoming queries, and some kind of storage to hold the data you want to serve (be that in memory or on disk).

I’d use EventMachine to sit on (presumably) port 53 listening for incoming UDP queries. You should really also listen for TCP queries (especially if you want to support zone transfer). When you receive something, call Dnsruby to decode it, and look up the response in your data store.

As for the storage, you’d need to decide if it should support updates, or whether it can be read-only. You could use the Dnsruby::RRSet class to hold resource records which have the same label, class and type. When your server receives a DNS request, do something like :

def NameServer.createResponseToIncoming(packet)
    message = Dnsruby::Message.decode(packet)
    question = message.question()[0]

    # If the question is a Dnsruby::Update or an AXFR/IXFR transfer request then deal with it separately
    # if (message.header.opcode != Dnsruby::OpCode.Query) ....

    # Find the records which were requested
    rrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, question.qtype)
    rrset.each {|rr| message.add_answer(rr)}

    # And get the NS records for the authority section
    nsrrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, Dnsruby::Types.NS)
    rrset.each {|rr| message.add_authority(rr)}

    # We also want the NS A records for the additional section
    nsrrset.each do |ns|
         rr = DataStore.findArecord ForNS(ns)
         message.add_additional(rr)
    end

    # Make sure that we are authoritative for the response! Otherwise, return REFUSED
    message.header.rcode = Dnsruby::RCode.NoError # Or Refused

    # Now encode the packet and return it for the server to respond with
    response = Dnsruby::Message.encode(message)
    return response
 end

[As you can see, Dnsruby is not doing a great deal for you here. Indeed, it would be possible to do away with most of the requirement for understanding DNS : you could store the data in wire format, thus removing the need to decode/encode the message. However, you’d still need to construct the encoded data store, and you’d have problems if you decided to secure your zone with DNSSEC with this approach]

If anybody does end up implementing a namserver with Dnsruby, I’d be happy to consider adding the code to the Dnsruby library! ;0)

Hopefully by now many of you will have read of the new and serious vulnerability in DNS servers that allows them to be spoofed easily and bypassing detection systems.  We at Nominet are privy to the details of this vulnerability and can assure you that it is every bit as serious a threat as being portrayed in the various advisories being released by CERTs around the world.

Whilst it is based around a known vulnerability in DNS (the 16 bit ID field) it enables spoofing of a caching resolver with a very small number of packets, far less than might trigger any normal detection system.  It can also be triggered remotely by various techniques, so an attacker does not need query access to your resolver to exploit the vulnerability.

The full details of the vulnerability will be released at BlackHat on August 6th, which gives a four week window for the upgrade of _all_ caching resolvers.   This is something that we strongly urge you to do and a process that we have already begun.  To be clear, this only applies to caching resolvers, not authoritative servers.

The main CERT vulnerability note is here (with links in it for every product)

If you use BIND then details of the patched versions are here.

If you use Microsoft DNS server then details are here.

If you wish to use a caching resolver that is built from the ground up for security, and already implements the mitigation technique, then please consider Unbound.  If you use djbdns then that too already implements the interim mitigation technique of source port randomisation.

It should be clear to all those who work with DNS on a daily basis that the only true mechanism to prevent DNS spoofing techniques is DNSSEC and we all need to begin taking that seriously.

So please use this short window to upgrade your caching resolvers and iron out any issues that might arise before exploits of this vulberability are seen in the wild

Over the past few months I have been working with Iris Explorer to develop visualisation applications. Iris Explorer uses a module based approach to application development, where tasks are split into individual modules which can be then wired together. The modules represent processes on the data and the wires between them represent the flow of the data between the modules. A simple drag and drop interface is provided for adding and linking the components of an application, or map, together.

Iris Explorer is distributed with a set of standard modules for accomplishing a wide variety of standard tasks such as, reading data, writing images, and standard visualisation techniques, such as plotting histograms. An example of an Iris Explorer map is shown below:

I have been working on creating a 3D graph to represent the volume of queries to our WHOIS system for each hour of a given date, or range of dates. This could largely be accomplished using the standard module set avaialble with Iris Explorer, however there were no modules for the labelling of the axes of graphs which were particularly suited to the task, which meant I had to write my own custom module.

Iris Explorer provides the ability to write your own modules, so it is entirely possible to add custom processes and functionality to an application. The method for developing these custom modules is a two step process, first of all the the layout of the module is defined using the Iris Module Builder, secondly a user function, written in C/C++, is added.

The Module Builder provides a GUI to creating the module window, its data input and output ports and specifying the flow of data through the module internally. This is a logical process where you simply specify the type of input data which is accepted as well as the format of the output data. The appearance of the module itself and the user interface of the module are also specified using a drag and drop WYSIWYG style editor.

The second stage, the user function, controls the processing of the data. So in the case of the Graph Labelling module I have written this would extract the number of dimensions in the data set and number bars of the graph accordingly, as well as creating and positioning axis text labels.

The Iris Explorer Map produced collects data for a given date range, and split this into hourly segments to give a clear picture of the volume of queries to our systems at any given hour of the day. An example of the results of this are shown below: