techblog 

A recent query from a registrar has prompted the Registrar Systems Support team to take a close look at how our EPP system works with Nominet’s first come, first served approach. The nature of our EPP service makes this challenging to apply and it may not be applied in the way you expect.

If we look at Nominet’s EPP service, the “EPP server” itself is actually multiple load balanced servers, operating multithreaded processes. These communicate with xml translation hardware also through load balancers, and also communicate with a database. A combination of factors can cause a difference in the sequence that transactions are acted upon. These include:

1. which EPP server the request goes to
2. thread scheduling (at the operating system level) in the specific EPP server
3. which piece of hardware the xml load balancers select
4. scheduling of translation requests within the xml hardware
5. internal scheduling within the database

When you look at how EPP handles requests over “large” periods of time, EPP is clearly a first come, first served system. However because of the nature of multithreaded systems, it is not feasible (or desirable) to apply that principle when the period of time is a handful of milliseconds. Most of the advantages that EPP has over the Automaton stem from the fact that it is multithreaded and acts on requests in parallel.

The principle behind first come first served is that no party is given preferential treatment when registering a domain name. We do not shape our traffic and no registrar is given any sort of priority when our EPP system processes a request. We apply first come first served based on when the first valid request gets committed to the database. We must do this as we operate three different registration systems.

For registrars who work in an environment where milliseconds can mean the difference between successfully registering a domain name or not, this may be significant when deciding how your EPP client communicates with Nominet.

A cloud refers to “the provision of dynamically scalable and often virtualized resources as a service over the Internet” (from Wikipedia). In practice, a user that logs in a cloud service (the bottom of this page lists some of them), for a reasonable price, can rent “resources” such as disk space or virtual machines to run his own code.

Recently, I have been monitoring the queries coming to our WHOIS service  and have noticed that several requests were originated by machines belonging to the IP space of a well-known commercial cloud. Since the WHOIS is a free service and can be run from any machine, I strongly suspect this technique has been used to avoid hitting the limit of 1000 queries/day set by Nominet’s Acceptable Use Policy on a per user basis (and not per IP).

The impact of this episode, as far as I can see, is limited and, maybe, not worth too much attention. What is interesting, however, is the way the cloud has been used to circumvent Nominet’s rules. This rises questions about how easy it would be for a malicious user to exploit a cloud computing environment for illegal activities and how long shall we wait before the first large-scale attack based on this technology is reported.

If we consider how the cloud environment works, we realise that:

• A cloud gives a malicious user access to a virtually unlimited pool of resources and computing power
• It is difficult to enforce limits on the amount of resources a single user is allowed to control, because this would harm legimitate users, without preventing malicious ones to open multiple accounts
• Monitoring all processes and activities that run on the cloud is quite complex, maybe impractical. Besides, I don’t think legitimate users would be happy with service providers inspecting their data. They will be forced to use cryptography, which will make things even worse
• Assuming that a service provider could offer some level of protection from misuses of their service, malicious users could spread their activities across different cloud providers, making the task of early detection very complex.
• Finally, accessing cloud services is cheap and prices are expected to drop with the technology behind big data centres becoming more accessible.

The security issues associated to cloud computing are not unknown (recently, for example, botnet controllers have been discovered in the Google cloud), the problem is that this kind of attacks and  the threat associated to them are likely to increase in the coming years.

Defending from a cloud-based attack might not be easy and will need to rely on the “good will” of  the cloud service providers, which will be expected to monitor their users activities. And, to cite Joze Nazario, from Arbor Networks in a recent interview to The Register, “going to a company as big as Google and saying ‘Can we get an image of that server,’ that’s a pretty high barrier”. Especially for small-medium organisations affected by a small/medium -sized attacks.

Mutation testing is a technique for checking how good your unit tests are.  It mutates a class by for example swapping a subtraction for an addition or by negating an if statement.  It then runs the unit tests for that class.  If none of them fail, then maybe your tests are not good enough.

We took a look at Java mutation testing tools some time ago as a possible addition to our continuous integration system.  But at that time the tools fell short.  Jester works by mutating the source files.  We found that this was just too slow with all the compilation it needs to do.  Jumble is a bit smarter in that it mutates the bytecode of the compiled class files.  However, when we evaluated it only JUnit 3 tests were supported and we were already well on the way to transitioning most of our tests to JUnit 4.

Recently though, Jumble was modified to work with JUnit 4 tests, so I thought it was time to take another look at it.  I found it quite tricky to get it working as there is no ant integration and it only runs as a command line application.  However, with a bit of persuasion I managed to get it running over our codebase using ant.  The details of how I did this are given below, but I think that a better solution would definitely be a fully fledged custom ant task.

So how did I get on?  Initially I had some niggly classloader problems. It seems that you need to tell Jumble’s mutating classloader to defer the loading of various sets of classes to the default classloader.  I needed to do this for the JMX classes found under javax.management by specifying the command line flag --defer-class=javax.management. Once I’d done this I had it working and did indeed find some interesting things.  I found tests that had been cut-and-pasted and not changed to test what they claimed to test. I found some test data that wasn’t up to the job and I found an actual bug in the code.

However, I hit a roadblock once the code under test used a database.  For some reason Oracle’s JDBC driver would not behave. Even before any mutations were applied it would insist on trying to connect with a null password.  This meant that it tried a number of times before locking itself out of the database.  I assume that this is some kind of classloader thing, but it seems strange that it manages to successfully contact the database only to completely mess up the credentials.  I’ve contacted the developers of Jumble (who are based in New Zealand incidentally), but no solution has been forthcoming as yet.  Until this problem is fixed we won’t be able to add this to our continuous integration system, which is a shame, as I think it could be a useful tool.

So, how did I integrate Jumble into ant?  I used the follow macrodef to run Jumble:

<macrodef name="do-mutation">
    <attribute name="class-to-mutate"/>
    <sequential>

        <!-- Use this trick to convert a reference to a classpath to classpath as a string -->
        <property name="the-classpath-as-a-string" refid="execute-test-classpath"/>        

        <java dir="${base.directory}" classname="com.reeltwo.jumble.Jumble" fork="true">

            <!-- all we really need in this classpath is the jumble library -->
            <classpath refid="ants-own-classpath"/>

            <!-- but then it needs everything in the JVM it forks: -->
            <arg value="--classpath=${the-classpath-as-a-string}"/>
            <arg value="--exclude=equals,hashCode,toString"/>
            <arg value="--defer-class=javax.management."/>
            <arg value="@{class-to-mutate}" />
        </java>

    </sequential>

</macrodef>

To get this to work you will need the classpath used to run the tests set up with the id execute-test-classpath and the classpath used by ant to find custom tasks set up with the id ants-own-classpath. The latter will need to include the jumble jar. As you can see I have excluded the equals(), hashcode() and toString() methods from being mutated as the first two are often generated by the IDE anyway. In the case of toString, this rarely contains important logic.  As mentioned before, the JMX classes are deferred to the default classloader. You may need to add some other packages to get it to work in your environment.

The above macrodef will mutate a single class. But we want to run this across our whole code base. To do this I had to resort to using some further ant tricks in the shape of the ant-contrib library which adds additional functionality to ant. As I said before, a better solution would be to write a proper ant task. Here is how the macrodef is called to run Jumble over a whole project:

<target name="mutation.test">    

    <!-- The ant contrib jar contains the "for" task -->
    <taskdef resource="net/sf/antcontrib/antlib.xml" classpathref="ants-own-classpath"/>

    <!-- Strip off the leading directory and the .class. Then replace the slashes
         with dots and separate each one with a comma.
         Results go into the classlist property -->

    <pathconvert dirsep="." pathsep="," property="classlist">
        <mapper type="glob" from="${build.dir}/*.class" to="*"/>
        <fileset refid="classes-to-mutate"/>
    </pathconvert>

    <!-- Iterate through the comma separated list and call jumble -->
    <for list="${classlist}" param="class">
        <sequential>
            <do-mutation class-to-mutate="@{class}"/>
        </sequential>
    </for>

</target>

This takes a fileset with id classes-to-mutate which consists of classes under the directory ${build.dir}. It turns the path into a package and removes the .class from the end to get a list of classes to mutate. (NB This was written to work with Unix style paths, it may need alteration to work under Windows). Then the macrodef given previously is called for each. Note that we refer to the ants-own-classpath classpath again which must contain the ant contrib jar this time.  The code given above was put in a standard ant build file included by other projects. The fileset to mutate could then be defined in each like this:

<fileset id="classes-to-mutate" dir="${build.dir}" includes="**/*.class">
        <exclude name="**/WeWantToLeaveThisOut.class"/>
</fileset>

UPDATE: The story gets even weirder. One of the Jumble developers got back to me with a suggestion for how to fix the Oracle problem, which was to add the JDBC driver to the list of classes deferred to the parent classloader. This didn’t help, but I then discovered bizarrely that the JDBC problem goes away if you are connecting to an 11g database as opposed to a 10g one. So it means that somehow the 10g JDBC driver fails to connect to 10g when run by Jumble, but succeeds against a later version. Curiouser and curiouser…

I’ve recently written and released source code for “evldns”.

evldns is a software mashup - it takes libevent’s fast event processing code and combines it with ldns’s DNS packet handling.  It’s derived from the server-side half of libevent’s “evdns” component.

The resulting framework is particularly intended for writing servers which generate custom responses. Examples included are:

• an AS112 server which has been benchmarked at over 60,000 queries per second on an HP DL385 server.
• a server which responds with the IP address of the client which sent the query - this can be useful for network discovery

The framework could also be used to write a “fuzzing” DNS server - one that deliberately returns malformed responses so as to trigger and test for bugs in DNS clients.

Here’s an extract from the package’s README:

evldns works using callback functions. A list of packet matching patterns
may be registered, along with a pointer to the function that will be
invoked when each pattern is matched.

The packet match works on the usual DNS triple of (QNAME, QCLASS, QTYPE)
where QNAME may be an exact match or a wildcard, and QCLASS or QTYPE may
be “ANY”.

The callback function is passed two parameters:

void callback(struct evldns_server_request *req, void *data)

The “req” parameter contains the complete received DNS request as an
“ldns_pkt”. The callback should create a response packet and populate
“req” with that response, which may either be in raw wire format
(req->wire_response and req->wire_len) or in ldns format (req->response).

If the callback function fails to populate either of the response fields
then the evldns system will pass the received packet onto the next
matching callback.

Should no callback match then evldns will automatically generate and
return a packet with RCODE = 5 (Refused).

The “data” parameter is used to pass an additional parameter supplied when
the callback function was registered. See “mod_txtrec.c” for an example
of how “data” may be used to pass expected response data into a callback.

A complete evldns application requires just a few lines of code:

event_init(); /* initialise libevent */
evldns_init(); /* initialise evldns */

/* create an evldns server context */
struct evldns_server *server = evldns_add_server();

/* register a UDP socket with evldns */
evldns_add_server_port(server, bind_to_udp4_port(53));

/* register callbacks here */
evldns_add_callback(server, qname, qclass, qtype, callback, data);
...

/* and set libevent running */
event_dispatch();

Please see the project home page for more information. There is also a Google hosted discussion group.

Ray Bellis, Advanced Projects Team

The day following the death of Michael Jackson, Google published a graph showing that their system were heavily hit by queries related to this news. Details can be found on the Google Official Blog.

Our experiments suggest that Nominet systems experienced an analogous, although orders of magnitude smaller, phenomenon. The following figures show the number of new registrations per hour of domain names that contain the name of Michael Jackson (or part of it) and the number of WHOIS queries that Nominet systems received in the same period.

The two graphs are highly correlated because it is common practice for domain name owners to make WHOIS lookups around the period of time they register new domains. The peak around the 27 of June in the second graph is probably related to news stories concerning suspicions about Michael’s death.  Apparently, it did not lead to an immediate rise in the number of domain name registrations.

We have conducted an informal analysis of the domain names that were registered in the last week. The majority of them belong to three categories: parking pages, commercial pages and commemorative sites such as blogs and forums. At the moment, we have no evidence of domain names used for scam or phishing.

In general, this episode confirms (again) that the dynamics of the Domain Name System follow those of the “real world”. A question that is still partially unanswered is at which degree these dynamics are followed by Internet users, i.e. how much their navigation behaviour depends on news stories. In the following months we plan to study the correlation between DNS data and other public events. Google has done something similar in the past, by correlating Google searches for flu-related terms with the spread of flu in North America. The results are very interesting and definitely merit extension to other data sources such as DNS traffic.

Typo-squatting is the practice of registering a domain name with the intent to confuse it with the name of a trademark or a famous other domain name

In March, I presented the paper Typo-Squatting: The “Curse” of Popularity in the poster session of the first International Conference on Web Science in Athens. The paper, written together with co-authors David Duce and Faye Mitchell (Oxford Brookes University) and Stephen Morris (Nominet) can be downloaded here.

In the paper we study typo-squatting from a statistical point of view. The distribution of names in the co.uk registry is analysed using the concepts of syntactic and visual neighbourhoods of a domain name (the sets of all other domain names which are syntactically or visually similar to to it).  Our preliminary results show a strong correlation between the popularity of a domain name and the size of its syntactical and visual neighbourhoods although, counter-intuitively, the neighbourhood size does not depend on length.  This suggests anomalous activity “around” very popular domain names, as well as indicating that the size of the neighbourhood can be used as a reliable indicator for the likelihood of being typo-squatted.

I’m pleased to announce the release of enumdroid.

This application adds ENUM (E.164 Number Mapping) support to your Android phone.

Each time you dial a full international number (i.e. starting with a ‘+’) your phone will check the DNS for additional routing information and offer you a list of alternate contact methods.

The application is open source (under the Apache License) and the code is available for download from Google Code.  The application can be downloaded from the Google Market under Applications -> Communication

Here are some screenshots, which show in turn:

1. Nominet’s switchboard number being dialled
2. ENUM results being returned
3. A call being placed over the PSTN to a tel: URI
4. The ENUM application’s settings page

Last week I attended the user experience conference UXLondon organised by Clearleft, with a solid day of keynote talks, followed by two days of half-day workshops.

In brief summary the conference highlighted several key points:

• How we should work with the customer at all stages to ensure both designer and customer are working towards the same goals.
• Developing software that is intuitive, aligned with user behaviour, can really make a software product stand out from it’s competitors.
• How using prototypes before starting development can really help iron out usability bugs before investing too much time and expense.
• Designing good interfaces for complex systems is hard!

My own detailed notes can be found in “UXLondon – Notes on User Experience and Design” (with one of the presentations embedded), where I talk more about what I personally learnt and what I thought of the individual sessions attended.

I got bitten by a silly bug in my dnsruby code recently - I thought I’d share it here in case anyone else starts pulling their hair out over this in future!

DNSSEC RRSIG records contain the signatures required to prove that a DNS zone has been correctly signed by an entity which possesses the correct keys for the zone. DNS clients can obtain the correct keys for the zone, then use the RRSIG records to prove that the zone (or record of interest) is correct. Of course, these records shouldn’t last forever - the data records are periodically re-signed (possibly with different keys), and the RRSIGs updated. The RRSIG includes an inception time, and an expiration time, to show the period over which it is valid. To verify a set of records, the DNS client must first produce an array of bytes, the digest of which is taken and used as the signature for the records - the salient data of of the RRSIG record (including the inception and expiration times) is included in this set of bytes.

The code I had written to do this was working fine - I had coded in the examples from the RFCs, and done a lot of work with actual signed zones (reading the data from the authoritative servers, and proving that it was correct). However, when I started to try to do this with real-world zone files, I started noticing that the signatures weren’t verifying. At least, they *sometimes* weren’t verifying. Very odd. Of course, with this kind of work, all you get is a Pass/Fail - no clue as to what is going wrong. I could see that the records were being translated to and from text format correctly - all the data in the RRs was showing as fine. However, when I compared the byte sequences prepared by my DNS client and Net::DNS, I noticed that four bytes were different. It turned out that, of the four byte sequences written to for the RRSIG inception and expiration time, there was a difference of 0xE100 - this works out to 16 hours worth of 3600 seconds. At last, I was onto something!

There are several ways to express time in the presentation format of the RRSIG record - “1234567890″ (seconds since 01/01/1970), or “YYYYMMDDHHMMSS” (e.g. “20090608123435″). Dnsruby worked fine with the first, and even translated the second from presentation format and back to presentation format correctly. However, when read using the following line :

return Time.mktime(year, mon, day, hour, min, sec).to_i

I got a 16 hour offset from the correct time! [When converting this time to/from the text format, the translation worked perfectly - it was only when inspecting the internal epoch time that the difference could be noticed]

I changed this to :

return Time.gm(year, mon, day, hour, min, sec).to_i

And suddenly everything worked just fine.

I’m sure that seasoned Rubyists will sneer at me for my stupidity - it did take me some time to track this one down! So, if you start noticing strange 16 hours offsets in your Ruby code, it’s worth checking your usage of the Time class…

In this post, I’d like to look at how to use Dnsruby with DNSSEC. As before, I’ll run these examples in irb, and assume that you’ve included Dnsruby there.

Dnsruby has DNSSEC support switched on by default. This means it will attempt to validate any DNS responses against its trust anchors. However, by default, no trust anchors are configured - to get dnsruby to validate responses, you must first configure a trust anchor (or DLV repository).

Trust Anchors

DNSSEC works by following a chain of trust from parent zone to child zone. This chain of trust must start somewhere - the “trust anchor”. In a world with a signed root, the root would be the anchor. Delegations to children zones would be signed, all the way down to the domain that is being queried. The querier can then be sure that the signed response is genuine.

Unfortunately, the root is not yet signed - we have many “islands of security”. Each island is signed, but has no chain to it from the root. It is possible to configure dnsruby with the keys for these zones using Dnsruby::Dnssec#add_trust_anchor() - it’s also possible to define an expiration time for each anchor. Dnsruby will then follow the chain of trust from the anchor down to the queried domain in the signed zone.

Managing these trust anchors quickly becomes a headache. You need to have secure means of obtaining and verifying them, and rolling over to new keys as time goes on. Fortunately, there are two mechanisms to help with this : IANA’s TAR and ISC’s DLV repository.

IANA (who manage the root zone) have created a Trust Anchor Repository (ITAR) which can be used until the root is signed. This holds delegation records for the DNSSEC-signed TLDs. It is possible to download this repository and configure dnsruby with the anchors. A method to do this is defined in Dnsruby::Dnssec#load_itar, but it is not currently secure. If you need to use the ITAR securely, you are currently advised to add the trust anchors from the ITAR directly into dnsruby. A secure method will be provided in future releases.

DLV

Even if the root was signed, there will still be some domains in unsigned zones, which wish to benefit from DNSSEC security. For example, signed-zone.unsigned-zone.example.org - there can be no chain of trust from the root to signed-zone. A solution exists for signed-zone : DNSSEC Lookaside Validation (DLV). Here, a DLV repository holds secure delegation records for zones like signed-zone. Instead of following the chain of trust from the root, a validator follows the chain of trust from the closest parent zone known to the DLV repository. Of course, this method involves more validation queries for each application query.

As an example, considering querying for random.example.com - first, the query itself must be made. Then, if unsuccessful, a DLV query for random.example.com.dlv.isc.org must be made, followed by a query for example.com.dlv.isc.org, followed by a query for com.dlv.isc.org. If none of these succeed, then the message cannot be validated. Imagine that a response was received for the com.dlv.isc.org zone : then, the chain of trust could be followed through example.com down to random.example.com. Keys discovered from the DLV repository are cached.

Configuring Trust Anchors

To configure a trust anchor (in this case for the uk-dnssec.nic.uk DNSSEC test zone) :

    trusted_key = Dnsruby::RR.create({:name => "uk-dnssec.nic.uk.",
        :type => Dnsruby::Types.DNSKEY,
        :flags => 257,
        :protocol => 3,
        :algorithm => 5,
        :key=> "AQPJO6LjrCHhzSF9PIVV7YoQ8iE31FXvghx+14E+jsv4uWJR9jLrxMYm sFOGAKWhiis832ISbPTYtF8sxbNVEotgf9eePruAFPIg6ZixG4yMO9XG LXmcKTQ/cVudqkU00V7M0cUzsYrhc4gPH/NKfQJBC5dbBkbIXJkksPLv Fe8lReKYqocYP6Bng1eBTtkA+N+6mSXzCwSApbNysFnm6yfQwtKlr75p m+pd0/Um+uBkR4nJQGYNt0mPuw4QVBu1TfF5mQYIFoDYASLiDQpvNRN3 US0U5DEG9mARulKSSw448urHvOBwT9Gx5qF2NE4H9ySjOdftjpj62kjb Lmc8/v+z"
      })
    Dnsruby::Dnssec.add_trust_anchor(trusted_key)

Dnsruby will now attempt to validate any responses from the uk-dnssec.nic.uk zone (or its children).

To configure dnsruby to use ISC’s DLV repository, you must first obtain the key (from here). You can then configure dnsruby :

    dlv_key = RR.create("DLV_KEY_STRING_FROM_ISC")
    Dnssec.add_dlv_key(dlv_key)

This method queries the DLV registry to get the ZSK (zone signing key) from the above KSK (key signing key). Dnsruby will now attempt to validate all responses against the DLV repository, if it can’t validate from any trust anchors.

Configuring Validation Policy

It is possible to configure the validation policy to vary the precedence of search order - from the root only, or local anchors only, or either first. Separate key caches are maintained by each validator, making it possible to configure them dynamically. DLV validation is only performed once the DLV key has been added. Here is an example of changing the validation policy :

    Dnsruby::Dnssec.validation_policy = Dnsruby::Dnssec::ValidationPolicy::ROOT_THEN_LOCAL_ANCHORS

It is possible to clear all trusted keys (which will also stop DLV validation) by calling :

    Dnsruby::Dnssec.clear_trusted_keys()

You can remove just the trust anchors (leaving DLV keys and validation from the root, and all keys generated from them), and the keys generated from them, by calling :

    Dnsruby::Dnssec.clear_trust_anchors()

Configuring Validation Resolver

When a response is validated, it may be necessary to make several more queries in order to follow the chain of trust. As more queries are made, more chains are followed. Trusted keys are cached as they are discovered (for the length of time they are indicated to be good for) - this means that future queries for domains in those zones will not require so many validation queries to be performed.

It’s possible to configure dnsruby to use different methods for performing the validation queries. They can either be directed to recursive nameservers (which can be the system defaults, or a client-supplied set of addresses), or they can be performed recursively. I have found that many resolvers do not yet speak a perfect dialect of DNSSEC - performing validation queries recursively ensures that the correct DNSSEC-signed responses are received. The default is to perform validation recursively. Of course, while the caches are being built up when dnsruby starts, more queries will be performed than if the queries were directed to recursive nameservers.

To ask dnsruby to use query a recursive nameserver, call :

    Dnsruby::Dnssec.do_validation_with_recursor = false

Dnsruby will now use the system default configured nameservers for validation.

To use a specific set of servers to perform validation :

    res = Dnsruby::Resolver.new({:nameserver => ['192.168.1.1', '192.168.2.1']})
    Dnsruby::Dnssec.default_resolver = res

Validating Responses

Once dnsruby has been configured with a trust anchor, it will attempt to validate any responses for domains within that zone (or its subzones). If it detects that validation is necessary, then it will fire up a new thread to handle that validation. Since many queries may need to be performed in order to validate the reponse, this can take some time longer than the original query would have done alone. This means that the query timing settings in the Resolver class apply only to each query - *not* to the whole validation process.

For example, a query may have a Resolver#query_timeout of 5 seconds. As long as the answer for that query is returned in 5 seconds, then no timeout will occur - even if it then takes another 6 seconds to validate that response. Future versions of dnsruby will include the ability for client applications to receive events detailing the progress of each asynchronous query (e.g. RECEIVED, VALIDATED).

It is possible to disable validation on a Message basis. Simply set :

  msg.do_validation = false

before sending the Message - dnsruby will not validate the response to that query.

Message Security Levels

Messages can have one of four security levels (defined in Dnsruby::Message::SecurityLevel) : BOGUS, UNCHECKED, INSECURE and SECURE. Dnsruby will only raise an error if it detects that a response is BOGUS - this means that the message does not contain the correct set of signatures. INSECURE means that the response has been verified to have come from a non-secured zone. SECURE means that the chain of trust has been correctly followed from a configured trust anchor to the response, and that all signatures check OK. Note that an NXDOMAIN response can still be SECURE - this means that the NSEC(3) records have been verified to prove non-existence.

To check the security level of a Message, use Message#security_level :

  if (msg.security_level == Dnsruby::Message::SecurityLevel::SECURE)
      print "Response was validated OKn"
  end

Examples of Use

DNSSEC examples can be found in the EXAMPLES file in the dnsruby distribution.

Limitations

Dnsruby does not yet perform NSEC3 validation (although NSEC3/NSEC3PARAM records can be read from the wire, or presentation format). This will be added to a future release.