techblog 

A third person has now asked me about using Dnsruby to write an authoritative nameserver. It seemed worth jotting some notes here.

A nameserver at its most basic simply provides a mapping service between names and addresses. Of course, the RFCs that specify DNS have more than that to say on the subject, even at the start. And, in the 25 years since the first DNS specifications, there’s been a whole load more! For example, an authoritative nameserver might support views, ACLs, different flavours of zone transfers, dynamic updates, etc. And then there’s the cryptographic extensions : TSIG, DNSSEC and so on.

If you want the latest, fully RFC-compliant authoritative nameserver, then your best bet is definitely just to download BIND or NSD. It will run a lot faster than anything you can do with Ruby, and will support most of the currently fashionable DNS features. You won’t even need to debug it! ;0)

However, I don’t think that the people asking about Dnsruby are necessarily after a fully-featured nameserver : “I would like to get some kind of simple ruby dns server running, authoritative-only, for development purposes as it can interact nicely with existing ruby infrastructure and i have no need for high volume”

Of course, you could ask why you’d need to use Dnsruby at all - after all, if only an A record is to be requested, how much DNS functionality is required to respond? I guess there are two reasons :

1) Code reuse. Even if only a few lines of the parser in Dnsruby are used, it still beats re-implementing (and debugging) them.

2) Future extensions. You might not think you’ll want that fancy new feature, but who knows how things will change in the future? The Dnsruby library already provides support for may DNS features (most notably those related to security).

One issue is that I wrote Dnsruby as a client-side library. So, although it supports zone transfers, it only supports initiating and receiving them - not serving them. Currently, support is only there for verifying DNSSEC signatures (although very little code would be needed in order to support signing packets). Similarly, there is currently only support for sending queries; there is no listening server. And there is certainly no data store (or cache).

Anyway, the question was : “If I could have dnsruby sit on a machine (using EM - awesome!) and receive dns requests, and I could put some hook code in there somewhere to look it up in a DB or YAML file or whatever, it sounds like I could do that, but I just can’t find where to begin.”

I think most of the answer is in the question! The main tasks would be to write a server to listen for incoming queries, and some kind of storage to hold the data you want to serve (be that in memory or on disk).

I’d use EventMachine to sit on (presumably) port 53 listening for incoming UDP queries. You should really also listen for TCP queries (especially if you want to support zone transfer). When you receive something, call Dnsruby to decode it, and look up the response in your data store.

As for the storage, you’d need to decide if it should support updates, or whether it can be read-only. You could use the Dnsruby::RRSet class to hold resource records which have the same label, class and type. When your server receives a DNS request, do something like :

def NameServer.createResponseToIncoming(packet)
    message = Dnsruby::Message.decode(packet)
    question = message.question()[0]

    # If the question is a Dnsruby::Update or an AXFR/IXFR transfer request then deal with it separately
    # if (message.header.opcode != Dnsruby::OpCode.Query) ....

    # Find the records which were requested
    rrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, question.qtype)
    rrset.each {|rr| message.add_answer(rr)}

    # And get the NS records for the authority section
    nsrrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, Dnsruby::Types.NS)
    rrset.each {|rr| message.add_authority(rr)}

    # We also want the NS A records for the additional section
    nsrrset.each do |ns|
         rr = DataStore.findArecord ForNS(ns)
         message.add_additional(rr)
    end

    # Make sure that we are authoritative for the response! Otherwise, return REFUSED
    message.header.rcode = Dnsruby::RCode.NoError # Or Refused

    # Now encode the packet and return it for the server to respond with
    response = Dnsruby::Message.encode(message)
    return response
 end

[As you can see, Dnsruby is not doing a great deal for you here. Indeed, it would be possible to do away with most of the requirement for understanding DNS : you could store the data in wire format, thus removing the need to decode/encode the message. However, you’d still need to construct the encoded data store, and you’d have problems if you decided to secure your zone with DNSSEC with this approach]

If anybody does end up implementing a namserver with Dnsruby, I’d be happy to consider adding the code to the Dnsruby library! ;0)

Hopefully by now many of you will have read of the new and serious vulnerability in DNS servers that allows them to be spoofed easily and bypassing detection systems.  We at Nominet are privy to the details of this vulnerability and can assure you that it is every bit as serious a threat as being portrayed in the various advisories being released by CERTs around the world.

Whilst it is based around a known vulnerability in DNS (the 16 bit ID field) it enables spoofing of a caching resolver with a very small number of packets, far less than might trigger any normal detection system.  It can also be triggered remotely by various techniques, so an attacker does not need query access to your resolver to exploit the vulnerability.

The full details of the vulnerability will be released at BlackHat on August 6th, which gives a four week window for the upgrade of _all_ caching resolvers.   This is something that we strongly urge you to do and a process that we have already begun.  To be clear, this only applies to caching resolvers, not authoritative servers.

The main CERT vulnerability note is here (with links in it for every product)

If you use BIND then details of the patched versions are here.

If you use Microsoft DNS server then details are here.

If you wish to use a caching resolver that is built from the ground up for security, and already implements the mitigation technique, then please consider Unbound.  If you use djbdns then that too already implements the interim mitigation technique of source port randomisation.

It should be clear to all those who work with DNS on a daily basis that the only true mechanism to prevent DNS spoofing techniques is DNSSEC and we all need to begin taking that seriously.

So please use this short window to upgrade your caching resolvers and iron out any issues that might arise before exploits of this vulberability are seen in the wild

Background
In many ways, name servers are standardised: the format of queries and responses are defined by standards, as are the ways of transferring zone information into and out of them (zone files, dynamic updates, AXFR, IXFR). This is not the case of the commands and files used to configure and control them, which are specific to each type of server.

Having a common means of interacting with servers would stimulate the development of a common management client, so simplifying operations. Although particularly benefiting users of multiple name servers, good management software should make it easier for occasional users to securely configure and manage their systems.

We have been investigating the idea of a common management interface, as have the IETF, who set up the DCOMA (DNS Configuration Management) committee to consider the problem. We have contributed to the DCOMA discussions, the result of which has been the publication of an Internet Draft containing the requirements for the system.

Implementation
Our approach has to define a protocol (NSCP - Name Server Control Protocol) layered on top of NETCONF, an XML-based protocol for the configuration and control of network devices. In NETCONF, a data model is defined for a network device, with configuration commands being framed in terms of it. NSCP defines a generic object model for a name server, and extends the NETCONF command set with name server-specific ones.

Although the long-term aim is to get NSCP understood by name server implementations, a more pragmatic approach is to put the control into server-specific wrappers, so avoiding the need for changes to the server software. The wrapper accepts NSCP commands and, on the basis of them, modifies the configuration file (and zone files), and causes the server to reload the data. The way this operates is shown in the figure below:

The first step is to create an XML version of the server configuration file. Name server configuration files tend to map into XML quite well as they usually have a hierarchical structure. In our tests, this was accomplished for BIND and NSD by modifying the parser module to emit XML as the configuration file was processed. Although this required a modification to the server software, the modification is a small, localised, change and does not otherwise affect its operation.

Next, an XSL transformation is used to convert the server-specific XML into NSCP, a process that involves mapping server-specific objects and attributes into NSCP objects and attributes. A two-step approach is used to keep changes to the name server software to a minimum: the parser module only needs to create XML that is isomorphic to the configuration file, something that is relatively simple to do. The intelligence needed to convert the representation into an NSCP is held in an (external) XSLT.

Once the NSCP representation of the configuration file is obtained, it can be manipulated with NETCONF commands. “Listing” commands (such as “get-config”) extract the relevant part of the configuration from the representation and send it back to the client. “Modification” commands (e.g. “edit-config”) are applied to the NSCP representation of the configuration file (again using an XSLT) to obtain NSCP describing the updated configuration: this is converted back into the configuration file format. As before, the conversion is a two-stage process to separate out the logic of the object model mapping from the mechanical process of creating the configuration file.

Once the configuration file has been updated, the wrapper forces the server to reload it to apply the changes.

Current Work
A small proof of concept project has shown that this approach is both feasible and practical. Effort is now being put into a pilot implementation.

There has been recent instability in one of the LANs that is run by the London Internet Exchange (LINX). Thanks to their hard work the network is now stable again, but this RIPE NCC DNSMON plot shows that some of our nameservers were affected.

The nameservers worst affected were all within the UK. ns6.nic.uk is in Amsterdam, ns[a-d].nic.uk are anycasted by Neustar from locations around the world.

ns1.nic.uk is hosted by LINX themselves, so it is unsurprising that this box was worst affected. The other servers show how much connectivity relies on peering agreements, to a greater or lesser degree. ns3.nic.uk is hosted by Telecity, and was much less affected. It is interesting that ns7.nic.uk, which is in Manchester, suffered quite badly. I guess this because most traffic heading for Manchester is going through London.

An instability within the Dutch internet exchange, AMS-IX is also visible in a DNSMON plot. Here, the Amsterdam node was worst affected.

No name resolution problems were reported to us during either of these outages. I would like to claim credit for running a resilient and diverse nameserver network, but DNS itself can cope with these network issues.

There comes a point in a zone’s life when it gets too big to be held in memory. For TLDs we really only have .com that has reached this scale, but ENUM zones with this problem are numerous.

So if you want to run an authoritative nameserver for these zones you basically have these options:

1. Use a DB plugin to an existing nameserver like BIND. However the performance from this is so poor that this is not a reasonable option in most scenarios.
2. Buy an off-the-shelf nameserver like ANS from Nominum. However this is only a real option for the fantastically rich or those with such a controlled network they can make do with just two nameserver instances.
3. Use a service provider that runs the zones for you and has their own database back end technology. We use UltraDNS who have this functionality.
4. Write your own. Of course this is all Nominum and UltraDNS have done and more recently CommunityDNS, so how hard can it be.

Common misconceptions about a database back end

So how exactly do you go about designing the database back end for a nameserver? Well, in my opinion most people start back to front and continue that way.

What gets most people excited about using databases as a back end is three things. Each of which is quite wrong and can be dismissed in turn.

Use the main registration database

If you start using databases then why not have the nameserver run off the main registration database. That is already a fault tolerant cluster and it would mean that the nameserver was always instantly up to date.

This is madness for several reasons. First, no database has 100% uptime but a nameserver cluster should. To be clear, that does not mean every nameserver available 100% of the time, but at least one nameserver can be accessed 100% of the time.

Then there is the issue of zone file serial numbers. Are you going to update the serial number for every single update of the database (yes you have to)? What happens when things go wrong and need to be unwound? What happens if you need to restore from a backup?

There are also the performance issues from the way the data is stored. It is likely to be optimised for the registration system, not the nameserver system that pulls it off.

Finally this limits you to nameservers that are connected to the database by a reliable, fast and secure channel. Maybe possible in a single enterprise but not to be tried across the public Internet.

In essence running it this way is just too brittle and should not be considered. To be honest I don’t think any experienced people would think of that, but I wanted to make sure I covered all the options.

Multi-headed nameserver

The second cause for excitement is the possibility of a multi-headed nameserver. In other words, lots of front ends all dealing with the same database back end (which we assume is separate from the source database). The reasoning for this is that databases are large, expensive beasts, optimised for handling data requests from many clients. Whereas the front ends are much lighter, simpler beasts that are optimised around network processing. So fitting the two together seems to be a natural fit.

It also means that updates are only processed by one machine, the database, not directly impacting the others. Fewer updates need to be sent and there is less chance of inconsistency of the data.

In order to see why this is wrong (in most cases) we need to think about what kind of database do we really need for a nameserver. It turns out that we don’t actually need most of the features found in a modern RDBMS. For example we don’t need views, stored procedures, pluggable indexes etc. All we need is a simple, fast and efficient database, which doesn’t require hardware of a different nature to the front end.

The next point is to examine where the bottlenecks of performance are in that setup. The most obvious one is whenever the front end needs to ask the database for the data rather than use its cache. It starts with the network transfer then the contention for the lookup, it may have to go to disk, then the network transfer between the front end and back end. I would contend that with a multi-headed setup these bottlenecks are much worse than a simple 1-1 configuration of one front end and one database both on the same machine.

Note that one of the bottlenecks is not the processing of updates. The time spent processing updates is about one thousandth the time spent handling requests (or less) and so has no impact of the overall performance unless it is very badly implemented. Having more updates as a result of the 1-1 config is not going to outweigh the benefits.

Database replication

The third cause for excitement is the possibility of using database replication. It brings up a vision of nameservers magically being up to date across a whole cluster without anyone doing anything. After all databases are supposed to be good at replication.

This is the easiest point to rebut. Database replication is general purpose and proprietary. Whereas DNS already has an open standard, tried and tested and DNS specific database replication mechanism - AXFR and IXFR. Yes I hear you say, but using database replication means that you don’t need to serialise the data into DNS and back again and translate between DNS packet structure and the database structure.

Well for a start, the database structure should be very close to the DNS packet structure, not much point in having it any other way. Then of course even database replication has to do the serialisation at some point or another, so is there any thing actually gained by it?

The real killer though is the impact on caches. A sensible nameserver implementation will have multiple levels of data store, with a small, fast, pre-compiled cache first in line to check, then the database next which will aim for an in-memory hit first and finally a disk hit if all else fails. If you allow data to enter the nameserver directly at the database level then you need a mechanism for the database to signal the cache that it may have invalid data, and then have the cache refresh itself.

In my view this is simply too complex and will just slow down performance. Far easier to do away with it and have all updates come in the front door, where the nameserver knows they are and so can update its cache accordingly.

Ideal nameserver design

So putting this altogether gives me an ideal design for the database back end in a nameserver. As you’ve probably guessed, the big emphasis for me has been on performance, but that’s because adding a database is bound to be slower than an in-memory representation and so everything possible needs to be done to compensate for that.

In this design then the database is a lightweight embedded database in every single instance of the nameserver. It is stripped of all unnecessary features and the data tables and indexes are optimised solely for great DNS performance.

All access to this database is through the front end - either IXFR, AXFR, DDNS or a control process. No round-the-back access to the database is possible, it is hermetically sealed.

You may even be lucky and find a database used like this obviates the need for a separate cache.

Recently we have seen a marked increase in the number of .uk domain names being used for phishing purposes. One phishing syndicate seems to be particularly prolific. They register 40-60 domains at a time to run phishing sites.

Real individuals with genuine addresses were listed as the registrant and administrative contact. Based on past activity, it would appear that the registrant listed is the victim of identity theft.

Please make sure you have malware protection installed before you investigate one of these sites. These are highly sophisticated phishing sites and have been known to embed malware in them.

They are targeting more than just banking information. For example, we have seen them phish for accounts of a well known online auction service. They set up a site that checked in with this auction service to ensure the victim has entered valid username and password. If a valid user and password are entered, it will log the victim into the real auction service. The phishing victim would have no idea that they did not visit a legitimate log-in page for the online auction service.

They do not target domains with keywords that you may expect to see with phishing. Here is a sample list of domains that have been used:
loltech.co.uk
loltech.me.uk
loltech1.co.uk
loltech1.me.uk
loltech2.co.uk
loltech2.me.uk
loltech3.co.uk
loltech3.me.uk
modeisp.co.uk
modeisp.me.uk
modeisp.org.uk

The only pattern we have been able to identify to help you establish whether this group is attempting to register a domain with your company is the IP addresses behind the name servers. During registration, they will provide their own name server to host the domain on. The host name used will be random, but the host will resolve to one of the following four IP addresses:
81.16.131.40
88.16.131.40
200.72.139.67
202.44.71.149

We have been advised that these IP address will eventually change, but may be used for up to 3 months.

Recently I had a weird problem on my laptop. Web browsing was slow and certain lookups failed altogether. The failures were exclusively associated with nominet.org.uk domain names, specifically connections to our office network. When the problems first started my mail client failed to work unless the VPN was activated, which is not the way our security policy mandates it should work. After a while the mail client failed even after activating the VPN. I was effectively locked out of my email unless I visited the office!

The error messages from the mail client suggested that the DNS lookup for the mail server was timing out. Using wireshark on the wireless interface I noticed that there were no DNS lookups for nominet.org.uk, though other search domains were being appended. Everything in System Preferences looked fine, /etc/resolv.conf had no surprises. However, I did find a suspect file: /etc/resolver/vpn-resolver-662638-0 which contained:

domain nominet.org.uk
nameserver 213.248.199.17
timeout 3

This file dated from early 2006, which seemed odd as the laptop was only installed in March 2008! I presume it was copied over from my previous laptop by the Migration Assistant. Removal of this file fixed the problem, but where did it come from?

Our present VPN solution is based on OpenVPN and I use the Tunnelblick client to connect. My first thoughts were to blame this combination. There have been some stability problems with Tunnelblick on Leopard, apparently. But, before we used this system we used an SSL VPN solution terminating on a Netscreen firewall. I used the VPN Tracker client from Equinux to connect. I now believe the rogue resolver file was left behind by VPN Tracker after it was deleted. I am still at a loss to explain why it took so long to start affecting the laptop, or why it appeared to degrade in stages, rather than just fail.

Today, NLnet Labs released Unbound. A high performance, validating caching resolver implementation. It is fully DNSSEC-aware, up to the very latest standards such as NSEC3 and Opt-Out. Even if you don’t use DNSSEC, it is highly resilient to forgeries. It is the natural counterpart of NSD, NLnet Labs’ authoritative server.

The architectural concept behind unbound was developed in 2004. The modular design came from a wish to constrain complexity to very small modules. David Blacka implemented a prototype in Java, and Matt larson and others helped to make it fully featured. By the end of 2006, the prototype was done, and has been used in many standardization efforts in the IETF.

By that time, NLnet Labs became interested in building a validating caching resolver. Though inspired by early prototype, the C version is completely built from scratch. If you’ve ever used NSD, and appreciate the clarity, quality and robustness of the source code, then Unbound should not surprise you.

Unbound is very, very fast. It easily outperforms other DNS resolvers like PowerDNS or BIND. It runs on Linux, *BSD, Solaris, MacOSX. We expect Unbound to find its way to ISPs and other production level environments. NLnet Labs is committed to support Unbound. Any changes to that support will be notified two years in advance.

Last year, Duane Wessels published some maps of the IPv4 address space. These were static maps containing one dimensional IPv4 addresses plotted along a 12th order Hilbert curve, resulting in a two dimensional “heatmap”.

In these maps, consecutive networks are grouped nicely together. However, trying to orientate in the map is hard. This is due to the stringent locality preserving properties of hilbert curves. Since we only need to group networks together that share the same prefix, I looked for a curve that has better orientation at the cost of less locality preserving properties. A Morton curve, or Z-order curve does exactly that.

Every pixel represents a /24 network. To cover the entire IPv4 address space, the size of a map is 4096 x 4096 pixes. If this is plotted in three dimensions, the map would be 256 x 256 x 256 pixels.

I’ve created a proof of concept tool to view these three dimensional IPv4 maps. The input is a file with IPv4 addresses, one per line. There are many controls to orientate the cube, select address blocks, highlight problem parts, change focus, zooming, etc.

A few examples:

The 3d image above represents open resolvers on the internet. The more open resolvers per /24, the hotter the color.

The 3d image above shows a small detail of the internet. It shows the open resolvers in 64.0.0.0/8.

Though the data source for the proof of concept was a list of open resolvers, the uses for this vizualizing technique are numerous. The software is available here. Let me know if you’ve created any nice imagery with it.

I gave a presentation about this at the 70th IETF

Nominet host the authoritative Domain Name System (DNS) name servers for the .uk top-level domain. Put simply, a typical entry in our DNS consists of a domain name together with the addresses of any nameservers which are authoritative for that domain. However if one of the nameservers is actually contained within the domain it is authoritative for, then our DNS holds a glue record for that nameserver, which contains its IP address. Domain name registrars are continually sending us updates to the .uk DNS, registering new domains and amending the nameservers for existing domains.

Recently we discovered that for many months there has been an error in our software which updates the .uk DNS. In a rare set of circumstances we would not add a glue record to the DNS, even though it was required and had been provided by the registrar. This was brought to our attention by one registrar and corrected quickly.

I subsequently analysed our database to find how many DNS updates had been affected by this problem, so that I could then correct the DNS by applying the missing glue records. I found, out of the millions of DNS updates we had received between June 2007 and February 2008 (the duration of the problem), 120 had been affected by this problem.

In most cases these updates had been corrected by the registrars themselves within a couple of hours of the problem occurring, by sending in further updates, either containing the same nameserver details again, including the glue record details, or more often, containing different nameservers outside of the particular domain, and thus not requiring glue.

I actually only needed to apply the missing glue records to 17 domains, most of which had at least one nameserver outside the domain, and so had probably not noticed the missing glue record.

I found the reaction of most domain name registrars affected by this problem interesting. When they saw that the glue record had not made it into our DNS, rather than raise the problem with us they moved their domain to a nameserver not requiring glue. As soon as a registrar raised the problem with us we resolved it.