A third person has now asked me about using Dnsruby to write an authoritative nameserver. It seemed worth jotting some notes here.

A nameserver at its most basic simply provides a mapping service between names and addresses. Of course, the RFCs that specify DNS have more than that to say on the subject, even at the start. And, in the 25 years since the first DNS specifications, there’s been a whole load more! For example, an authoritative nameserver might support views, ACLs, different flavours of zone transfers, dynamic updates, etc. And then there’s the cryptographic extensions : TSIG, DNSSEC and so on.

If you want the latest, fully RFC-compliant authoritative nameserver, then your best bet is definitely just to download BIND or NSD. It will run a lot faster than anything you can do with Ruby, and will support most of the currently fashionable DNS features. You won’t even need to debug it! ;0)

However, I don’t think that the people asking about Dnsruby are necessarily after a fully-featured nameserver : “I would like to get some kind of simple ruby dns server running, authoritative-only, for development purposes as it can interact nicely with existing ruby infrastructure and i have no need for high volume”

Of course, you could ask why you’d need to use Dnsruby at all - after all, if only an A record is to be requested, how much DNS functionality is required to respond? I guess there are two reasons :

1) Code reuse. Even if only a few lines of the parser in Dnsruby are used, it still beats re-implementing (and debugging) them.

2) Future extensions. You might not think you’ll want that fancy new feature, but who knows how things will change in the future? The Dnsruby library already provides support for may DNS features (most notably those related to security).

One issue is that I wrote Dnsruby as a client-side library. So, although it supports zone transfers, it only supports initiating and receiving them - not serving them. Currently, support is only there for verifying DNSSEC signatures (although very little code would be needed in order to support signing packets). Similarly, there is currently only support for sending queries; there is no listening server. And there is certainly no data store (or cache).

Anyway, the question was : “If I could have dnsruby sit on a machine (using EM - awesome!) and receive dns requests, and I could put some hook code in there somewhere to look it up in a DB or YAML file or whatever, it sounds like I could do that, but I just can’t find where to begin.”

I think most of the answer is in the question! The main tasks would be to write a server to listen for incoming queries, and some kind of storage to hold the data you want to serve (be that in memory or on disk).

I’d use EventMachine to sit on (presumably) port 53 listening for incoming UDP queries. You should really also listen for TCP queries (especially if you want to support zone transfer). When you receive something, call Dnsruby to decode it, and look up the response in your data store.

As for the storage, you’d need to decide if it should support updates, or whether it can be read-only. You could use the Dnsruby::RRSet class to hold resource records which have the same label, class and type. When your server receives a DNS request, do something like :

def NameServer.createResponseToIncoming(packet)
    message = Dnsruby::Message.decode(packet)
    question = message.question()[0]

    # If the question is a Dnsruby::Update or an AXFR/IXFR transfer request then deal with it separately
    # if (message.header.opcode != Dnsruby::OpCode.Query) ....

    # Find the records which were requested
    rrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, question.qtype)
    rrset.each {|rr| message.add_answer(rr)}

    # And get the NS records for the authority section
    nsrrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, Dnsruby::Types.NS)
    rrset.each {|rr| message.add_authority(rr)}

    # We also want the NS A records for the additional section
    nsrrset.each do |ns|
         rr = DataStore.findArecord ForNS(ns)
         message.add_additional(rr)
    end

    # Make sure that we are authoritative for the response! Otherwise, return REFUSED
    message.header.rcode = Dnsruby::RCode.NoError # Or Refused

    # Now encode the packet and return it for the server to respond with
    response = Dnsruby::Message.encode(message)
    return response
 end

[As you can see, Dnsruby is not doing a great deal for you here. Indeed, it would be possible to do away with most of the requirement for understanding DNS : you could store the data in wire format, thus removing the need to decode/encode the message. However, you’d still need to construct the encoded data store, and you’d have problems if you decided to secure your zone with DNSSEC with this approach]

If anybody does end up implementing a namserver with Dnsruby, I’d be happy to consider adding the code to the Dnsruby library! ;0)

Recently I’ve been running puppet a *lot* on Solaris 10 (Sparc). I would say consistently around 50% of the time the following happens:

bash-3.00# /usr/local/bin/puppetd -t
notice: Ignoring cache
/opt/ruby/lib/ruby/1.8/timeout.rb:52: [BUG] Segmentation fault
ruby 1.8.6 (2007-09-24) [sparc-solaris2.10]
Abort (core dumped)

Though this has been causing me to get very frustrated with puppet, and indeed it has made the adoption of puppet within the organisation more problematic than it should have been, I realise this is unfair on puppet as the problem seems to be with the ruby jvm. We have not seen this type of behaviour with puppet on Linux or indeed with the Solaris x86 port. Maybe ruby can’t handle the 32 cores of our T2000.

I’m hoping with the release of a later version, either 1.8.7 or the developer release of 1.8.9 that these problems disappear. So a question:

Has anyone else experienced this type of segmentation fault with ruby?

I’ve been making packages for Solaris i386 architecture recently and have been compiling ruby as it’s required by puppet - our excellent configuration management software.

I had no trouble compiling it and making a Solaris package but when I came to use it to install puppet I ran into this

-bash-3.00$ ruby install.rb DESTDIR=${DESTDIR}
Could not load openssl; cannot install

It seems that ruby hadn’t included the openssl files into it’s build.

I found a post that said that openssl wasn’t included in ubuntu by default and it could be installed manually, so I tried it and found out that it could not find a header file:

-bash-3.00$ cd ruby-1.8.6-p114/ext/openssl/
-bash-3.00$ ruby extconf.rb
=== OpenSSL for Ruby configurator ===
=== Checking for system dependent stuff... ===
checking for t_open() in -lnsl... yes
checking for socket() in -lsocket... yes
checking for assert.h... yes
=== Checking for required stuff... ===
checking for openssl/ssl.h... no
=== Checking for required stuff failed. ===
Makefile wasn't created. Fix the errors above.

A quick modify of the CFLAGS variable to add “-I” and a re-run of configure/make and ruby had openssl support built in.

I’ve just released version 1.1 of Dnsruby.

This release adds DNSSEC support. More precisely, this version implements a non-validating stub resolver : the DO bit (in the EDNS header) is set to indicate that DNSSEC is understood. However, the checking is expected to be performed on the remote resolver - therefore, this should be a trusted resolver on a secure link.

If an untrusted resolver or an insecure link is used, then Dnsruby can still be used to verify responses for zones for which the client application knows a trusted key. A cache of trusted keys is maintained by Dnsruby, with new keys (from signed DNSKEY or DS RRSets) added to the cache. However, it is up to the client application to ensure that appropriate queries have been issued to follow the chain of trust from the known trusted key to the key required to verify particular records in the zone. For more details, see the DNSSEC file in the Dnsruby distribution.

Of course, once we have a signed root, then Dnsruby should be changed to automatically follow the chain of trust from the root down to the zone of interest. At that point, I’ll also need to work out a better API that conveys more sensible information about the resolution. The current system of “success/failure” doesn’t handle the many different types of “failure” that can occur in a DNSSEC world.

A concern with EventMachine has been potential blocking in the EventMachine::connect call, e.g. :

EventMachine::connect("dns-that-takes-ages.com", port, MyClass)

This call can block for a very long time, which is against the spirit of EventMachine. Fortunately, this problem can be solved using Dnsruby. A certain amount of boilerplate code is required here, which could be wrapped up in a new "EventMachine::connect_nonblock" call :

#boilerplate code
require 'Dnsruby'
require 'eventmachine'
res = Dnsruby::Resolver.new # use system defaults
Dnsruby::Resolver.use_eventmachine
Dnsruby::Resolver.start_eventmachine_loop(false)

#interesting code
#assumes EventMachine::run has been called
name = "dns_that_takes_ages.com"
df = res.send_async(Message.new(name))
df.callback {|msg|
  EM.connect(msg.answer[0], 8289, MyClass) {}
}
df.errback {|msg, err|
  print "Sorry - can't resolve #{name}. Error = #{err}n"
}

Now the call is non-blocking. If the DNS lookup succeeds, then the EM::connect call will continue as before. If the name can’t be resolved, then a different action is taken.

I’ve just released version 1.0 of dnsruby. You can find it here or install it by running :

gem install Dnsruby –test

Dnsruby now does pretty much everything you would expect from a stub resolver library - with the exception of DNSSEC (which will come with dnsruby version 2.0). However, you can still do signed zone tranfers and signed dynamic updates with this release. EDNS(0) is also fully supported. It can use EventMachine for its I/O, or an inbuilt pure Ruby event loop.

For more information, check out the RDoc and have a rummage through the demo folder in the distribution.

Enjoy!

Dnsruby can use either its inbuilt (pure Ruby) event loop, or
EventMachine (a native extension to Ruby which must be installed
on the local platform).

Configuring Dnsruby to use EventMachine

I left a couple of switches in Dnsruby::Resolver :

Dnsruby::Resolver.use_eventmachine(on=true)

Dnsruby::Resolver.start_eventmachine_loop(on=true)

The first of these tells Dnsruby to use EventMachine, rather
than its own event loop.

The second tells Dnsruby whether to start the EventMachine loop
or not.

If standard Dnsruby client code is used, then Dnsruby needs to
call EventMachine::run{} in order to start the EventMachine loop.
However, if more than one EventMachine loop is started in a Ruby
process, then the process terminates.

So, if client code is written in an EventMachine style, contained
in an EventMachine::run{} call, then it will need to tell Dnsruby
NOT to start the EventMachine loop (on pain of sudden death!).

Example code

Here is an example of using the code in an EventMachine style :

require 'Dnsruby'
require 'eventmachine'
res = Dnsruby::Resolver.new
Dnsruby::Resolver.use_eventmachine
Dnsruby::Resolver.start_eventmachine_loop(false)
EventMachine::run {
  df = res.send_async(Dnsruby::Message.new("example.com"))
  df.callback {|msg|
     puts "Response : #{msg}"
     EM.stop}
  df.errback {|msg, err|
     puts "Response : #{msg}"
     puts "Error: #{err}"
     EM.stop}
}

And an example in a normal Dnsruby style :

require 'Dnsruby'
res = Dnsruby::Resolver.new
Dnsruby::Resolver.use_eventmachine
Dnsruby::Resolver.start_eventmachine_loop(true) # default
q = Queue.new
id = res.send_async(Dnsruby::Message.new("example.com"),q)
id, response, error = q.pop

I recently spent some time enabling my Ruby DNS library (Dnsruby) to use EventMachine.

Dnsruby comes with its own (pure Ruby) event loop. This allows all the I/O to be done in one thread, the event loop, while the client code runs in its own thread. However, given the limitations of Ruby’s threads and select, and the variety of behaviours that exist on different platforms, it’s hard to get a pure Ruby system to work perfectly everywhere.

EventMachine gets round this issue by coding the guts of the I/O in C++ (or Java). The Ruby client code calls out to the native extensions to actually do the I/O, and all the Ruby code lives in one single thread, thus eliminating any irritating shared data issues and avoiding the whole Ruby thread quagmire.

So, I thought it would be nice for Dnsruby to be able to use the EventMachine library, when it was installed on the local platform. It would also make it easy for EventMachine users to use Dnsruby to make DNS queries from within their EventMachine loop.

Here are a few notes about the development. Continue Reading »

Having got the basic querying functionality and configuration sorted in dnsruby, I’m now implementing the high level lookup. I had hoped there would be an RFC to guide me here, but the closest I can find is in RFC1034, section 5.3.3, which is really for nameservers rather than stub resolvers. It says :

1. See if the answer is in local information, and if so return it to the client.

2. Find the best servers to ask.

3. Send them queries until one returns a response.

4. Analyze the response, either:

a. if the response answers the question or contains a name error, cache the data as well as returning it back to the client.

b. if the response contains a better delegation to other servers, cache the delegation information, and go to step 2.

c. if the response shows a CNAME and that is not the answer itself, cache the CNAME, change the SNAME to the canonical name in the CNAME RR and go to step 1.

d. if the response shows a servers failure or other bizarre contents, delete the server from the SLIST and go back to step 3.

In order to figure out the exact algorithm for dnsruby, I thought I’d look at some current open source stub resolver implementations, in dnsjava and Net::DNS / pnet-dns. I also thought I’d check out Ruby’s native resolv.rb (incomplete) DNS implementation.

dnsjava

dnsjava has several layers which perform lookups. The SimpleResolver targets a single nameserver and sends queries to it. No retries are performed in the case of timeouts or other errors.

The ExtendedResolver class has a clutch of SingleResolvers which it uses to fire queries at multiple nameservers. The class implements the retry mechanism in the case of failure. The basic algorithm is as follows :

• Send the query to the first nameserver (this could be first in a list, or a randomly selected nameserver from the list)
• If the response is good, or an NXDOMAIN is returned, then return it to the client.
• If it is a timeout, then retry that server, but also, if it is the first timeout for that server, send the first query to the next server on the list. If it’s not the first timeout, then simply try that nameserver again up to the maximum number of retries
• If an error is returned as the last response from the last nameserver, then send that to the client

The Lookup class adds another layer of complexity. This class implements a results cache, and performs lookups on names with the search list applied. When a query is made, the cache is checked first. If no records are found, then the ExtendedResolver which belongs to the Lookup is used to query all configured nameservers for the name (with the previous algorithm). The results are then added to the cache. CNAMEs are followed by the Lookup class.

Dnsjava can therefore be seen to follow what specification there is in the RFC.

In the case of totally unresponsive servers, this algorithm may take a total time of (retries*timeout + num_nameservers * timeout). However, if using the dnsjnio asynchronous extension, multiple retrying queries can be run concurrently in a single thread (plus a worker thread to do the I/O).

Net::DNS

The Net::DNS project has a different model. The Resolver class implements both the SingleResolver and ExtendedResolver functionality of dnsjava. It does not implement any form of result cache, and query forms exist to either append or not append the search list to query candidates.

The algorithm, the implementation of which is horrendously long and complicated, is as follows :

• If the search list is to be applied, then for each element in the list, that element is appended to the query name, and the remaining steps are run for each result. Otherwise, the remaining steps are only run for the queried name
• retry Retries of the following steps are attempted, and for each round of retries, retrans is doubled
• If udp_timeout is more than 0, and udp_timeout seconds have elapsed since the query was started, then the query times out to the client
• The first nameserver is queried, and the response awaited for retrans/num_nameservers seconds.
• If NXDomain or the result is returned, then that is passed back to the caller
• If the server returns any other error, then it is removed from the list of nameservers for this query
• If the query timed out, then the next nameserver is tried.
• Once all nameservers have been tried once, and all timed out, then this process is repeated up to retry times (see above)
• If still no response after all of this, then the query times out to the client

If udp_timeout is set, then this is the maximum time that the query can take.

If udp_timeout is set to 0, then the retrans and retry parameters control the query time. In this case, the query could take up to ((2^retry) * retrans) seconds. [It could take an additional num_nameservers * retrans seconds if the nameserver is totally duff]

Ruby resolv.rb

The original Ruby DNS implemenation, resolv.rb, has a slightly different algorithm. This implementation only performs retried lookups, and will apply the searchlist. The algorithm is as follows :

• All nameservers are tried sequentially. If a valid response is returned then that it is returned to the client. If an error is returned, it is returned to the user and the search stopped
• Once all nameservers have been tried, and all timed out, then they are all tried again with double the timeout value. This is tried three times (so all nameservers can be tried up to 4 times with increasing timeouts)
• If this is not successful, then the whole process is repeated with the next “candidate name”, which is formed by appending the next element of the search list to the domain

There is no way to set a per-client-query timeout.
This algorithm is probably the worst of the three - at least with dnsjava, the retries are concurrent once the first round of queries have timed out. However, Net::DNS has the nicest algorithm; splitting the timeout per retry between the number of nameservers limits the worst case scenario, and gives all nameservers a chance to respond. However, if the initial retry interval is set too short by the client, then more queries could be sent than are actually required (especially if several nameservers are targetted).

I think I’m going to go with a combination of the Net::DNS and dnsjava approach (as with so much in this project!), and have the option of :

• A total timeout for the query
• A retransmission system that targets the namervers concurrently once the first query round is complete, but in which the total time per query round is split between the number of nameservers targetted for the first round. and total time for query round is doubled for each query round

This system should limit the total number of queries sent in the best case, but allow good response from a poor or worst case.

Of course, unlike Net::DNS, all these queries will be done in the background by a worker thread.

Any comments?

I’ve been having some issues with Ruby IO#select() recently, which I think I now have my head around.

My Java non-blocking DNS project, dnsjnio, has a select loop in one thread, which processes all of the IO for all DNS queries. The client thread can wait on a blocking queue for more responses from the select thread. I had hoped to follow this model for Dnsruby. However, there are some differences between the select implementation in Ruby and Java which make things a little more tricky.

In Java, there is a Selector.wakeup() method which a client thread can call to rouse the select thread from its slumber. This means that you can send something (or ask the select thread to send something), and then wake the select thread up to deal with it. Otherwise, nothing would happen until the select() call timed out, or something happened on one of the descriptors it was monitoring - which could be a long time on a quiet system.

Ruby doesn’t have this facililty. So once a thread calls select(), it’s not going to do anything until either the timeout or a descriptor event occurs.

I read a comment by Dave Thomas in which he suggested waking the select thread up with an Exception. Well, I tried that, and it does work, but it doesn’t seem happy under JRuby. OK, it is possible to set up simple test code to get this working in JRuby. However, when you start hitting it hard it just doesn’t seem to work right, and I haven’t had the time to dig around in JRuby to figure out why.

UPDATE - I think this probably has something to do with it. Sounds like JRuby is having problems with Thread.critical=, which was getting used heavily in my multi-threaded stress testing code!

It would be nice to try to get the exception signalling working again in the future - the code is all still there. However, it is necessary to use excessive synchronisation between the threads in order to ensure that the exception is only fired at the time the select call is blocking. Basically, all the processing of the results of select has to be done in a synchronized block, guarded by the same Mutex that guards the exception raising.

Another possibility is to use pipes to signal the select that it is time to wakeup. Whilst this is a hack, it was still worth trying. The problem with it is that (once again) the behaviour or Ruby varies across platforms. Pipes aren’t implemented in Windows MRI, so we can’t use them.

This leaves the only option being to run select with a short timeout. Of course, this leaves us with a thread effectively in polling mode, which is not good. So, once the thread has not serviced any queries for a defined period of time (currently 1 second), it shuts itself down, to be restarted the next time a client query is sent. This leaves us with a worst case not much worse than the standard resolv.rb, and a best case significantly better!

It would be nice if there a non-hacky way to do this. If anyone knows of one, I’d love to hear it!!