techblog 

Having got the basic querying functionality and configuration sorted in dnsruby, I’m now implementing the high level lookup. I had hoped there would be an RFC to guide me here, but the closest I can find is in RFC1034, section 5.3.3, which is really for nameservers rather than stub resolvers. It says :

1. See if the answer is in local information, and if so return it to the client.

2. Find the best servers to ask.

3. Send them queries until one returns a response.

4. Analyze the response, either:

a. if the response answers the question or contains a name error, cache the data as well as returning it back to the client.

b. if the response contains a better delegation to other servers, cache the delegation information, and go to step 2.

c. if the response shows a CNAME and that is not the answer itself, cache the CNAME, change the SNAME to the canonical name in the CNAME RR and go to step 1.

d. if the response shows a servers failure or other bizarre contents, delete the server from the SLIST and go back to step 3.

In order to figure out the exact algorithm for dnsruby, I thought I’d look at some current open source stub resolver implementations, in dnsjava and Net::DNS / pnet-dns. I also thought I’d check out Ruby’s native resolv.rb (incomplete) DNS implementation.

dnsjava

dnsjava has several layers which perform lookups. The SimpleResolver targets a single nameserver and sends queries to it. No retries are performed in the case of timeouts or other errors.

The ExtendedResolver class has a clutch of SingleResolvers which it uses to fire queries at multiple nameservers. The class implements the retry mechanism in the case of failure. The basic algorithm is as follows :

• Send the query to the first nameserver (this could be first in a list, or a randomly selected nameserver from the list)
• If the response is good, or an NXDOMAIN is returned, then return it to the client.
• If it is a timeout, then retry that server, but also, if it is the first timeout for that server, send the first query to the next server on the list. If it’s not the first timeout, then simply try that nameserver again up to the maximum number of retries
• If an error is returned as the last response from the last nameserver, then send that to the client

The Lookup class adds another layer of complexity. This class implements a results cache, and performs lookups on names with the search list applied. When a query is made, the cache is checked first. If no records are found, then the ExtendedResolver which belongs to the Lookup is used to query all configured nameservers for the name (with the previous algorithm). The results are then added to the cache. CNAMEs are followed by the Lookup class.

Dnsjava can therefore be seen to follow what specification there is in the RFC.

In the case of totally unresponsive servers, this algorithm may take a total time of (retries*timeout + num_nameservers * timeout). However, if using the dnsjnio asynchronous extension, multiple retrying queries can be run concurrently in a single thread (plus a worker thread to do the I/O).

Net::DNS

The Net::DNS project has a different model. The Resolver class implements both the SingleResolver and ExtendedResolver functionality of dnsjava. It does not implement any form of result cache, and query forms exist to either append or not append the search list to query candidates.

The algorithm, the implementation of which is horrendously long and complicated, is as follows :

• If the search list is to be applied, then for each element in the list, that element is appended to the query name, and the remaining steps are run for each result. Otherwise, the remaining steps are only run for the queried name
• retry Retries of the following steps are attempted, and for each round of retries, retrans is doubled
• If udp_timeout is more than 0, and udp_timeout seconds have elapsed since the query was started, then the query times out to the client
• The first nameserver is queried, and the response awaited for retrans/num_nameservers seconds.
• If NXDomain or the result is returned, then that is passed back to the caller
• If the server returns any other error, then it is removed from the list of nameservers for this query
• If the query timed out, then the next nameserver is tried.
• Once all nameservers have been tried once, and all timed out, then this process is repeated up to retry times (see above)
• If still no response after all of this, then the query times out to the client

If udp_timeout is set, then this is the maximum time that the query can take.

If udp_timeout is set to 0, then the retrans and retry parameters control the query time. In this case, the query could take up to ((2^retry) * retrans) seconds. [It could take an additional num_nameservers * retrans seconds if the nameserver is totally duff]

Ruby resolv.rb

The original Ruby DNS implemenation, resolv.rb, has a slightly different algorithm. This implementation only performs retried lookups, and will apply the searchlist. The algorithm is as follows :

• All nameservers are tried sequentially. If a valid response is returned then that it is returned to the client. If an error is returned, it is returned to the user and the search stopped
• Once all nameservers have been tried, and all timed out, then they are all tried again with double the timeout value. This is tried three times (so all nameservers can be tried up to 4 times with increasing timeouts)
• If this is not successful, then the whole process is repeated with the next “candidate name”, which is formed by appending the next element of the search list to the domain

There is no way to set a per-client-query timeout.
This algorithm is probably the worst of the three - at least with dnsjava, the retries are concurrent once the first round of queries have timed out. However, Net::DNS has the nicest algorithm; splitting the timeout per retry between the number of nameservers limits the worst case scenario, and gives all nameservers a chance to respond. However, if the initial retry interval is set too short by the client, then more queries could be sent than are actually required (especially if several nameservers are targetted).

I think I’m going to go with a combination of the Net::DNS and dnsjava approach (as with so much in this project!), and have the option of :

• A total timeout for the query
• A retransmission system that targets the namervers concurrently once the first query round is complete, but in which the total time per query round is split between the number of nameservers targetted for the first round. and total time for query round is doubled for each query round

This system should limit the total number of queries sent in the best case, but allow good response from a poor or worst case.

Of course, unlike Net::DNS, all these queries will be done in the background by a worker thread.

Any comments?

I’ve been having some issues with Ruby IO#select() recently, which I think I now have my head around.

My Java non-blocking DNS project, dnsjnio, has a select loop in one thread, which processes all of the IO for all DNS queries. The client thread can wait on a blocking queue for more responses from the select thread. I had hoped to follow this model for Dnsruby. However, there are some differences between the select implementation in Ruby and Java which make things a little more tricky.

In Java, there is a Selector.wakeup() method which a client thread can call to rouse the select thread from its slumber. This means that you can send something (or ask the select thread to send something), and then wake the select thread up to deal with it. Otherwise, nothing would happen until the select() call timed out, or something happened on one of the descriptors it was monitoring - which could be a long time on a quiet system.

Ruby doesn’t have this facililty. So once a thread calls select(), it’s not going to do anything until either the timeout or a descriptor event occurs.

I read a comment by Dave Thomas in which he suggested waking the select thread up with an Exception. Well, I tried that, and it does work, but it doesn’t seem happy under JRuby. OK, it is possible to set up simple test code to get this working in JRuby. However, when you start hitting it hard it just doesn’t seem to work right, and I haven’t had the time to dig around in JRuby to figure out why.

UPDATE - I think this probably has something to do with it. Sounds like JRuby is having problems with Thread.critical=, which was getting used heavily in my multi-threaded stress testing code!

It would be nice to try to get the exception signalling working again in the future - the code is all still there. However, it is necessary to use excessive synchronisation between the threads in order to ensure that the exception is only fired at the time the select call is blocking. Basically, all the processing of the results of select has to be done in a synchronized block, guarded by the same Mutex that guards the exception raising.

Another possibility is to use pipes to signal the select that it is time to wakeup. Whilst this is a hack, it was still worth trying. The problem with it is that (once again) the behaviour or Ruby varies across platforms. Pipes aren’t implemented in Windows MRI, so we can’t use them.

This leaves the only option being to run select with a short timeout. Of course, this leaves us with a thread effectively in polling mode, which is not good. So, once the thread has not serviced any queries for a defined period of time (currently 1 second), it shuts itself down, to be restarted the next time a client query is sent. This leaves us with a worst case not much worse than the standard resolv.rb, and a best case significantly better!

It would be nice if there a non-hacky way to do this. If anyone knows of one, I’d love to hear it!!

For those who care, I’ve just released version 1.0.1 of pnet-dns, the Ruby direct port of perl’s Net::DNS.
This release adds JRuby support - for everything other than TCP sockets, that is. Hopefully JRuby will fix this soon; if so, then this release of pnet-dns should fully support JRuby.
This release also fixes a bug to do with OPT record processing. OPT records should now be handled correctly.
This release also conforms to a change in the Digest::MD5 API between Ruby 1.8.5 and 1.8.6.
This release has been tested on MRI 1.8.6 for Windows, Linux and Solaris.
Give it a go - feedback is appreciated!

I’ve been trying (again) to get my Ruby DNS library working with JRuby. I’ve been having some difficulties - not all of which are JRuby’s fault!

I need to be able to send data from a TCP port and receive a response. Many connections may be running from the same socket, and I need to be able to tell where a response has come from. The easy way to do this would be use TCPSocket#recvfrom - unfortunately, that raises an exception on Windows for Ruby 1.8.5 and 1.8.6 in MRI.

That leaves me with the option of creating a Socket directly, which is what pnet-dns does. Unfortunately, this API is still not fully supported by JRuby, so I’m stuck with TCPSocket. However, I thought I could still do something like :

          if (/java/=~RUBY_PLATFORM)
            sock = TCPSocket.new(ns, dstport, srcaddr, srcport)
          else
            sock = Socket.new( Socket::AF_INET, Socket::SOCK_STREAM, 0 )
            sockaddr = Socket.pack_sockaddr_in( srcport, srcaddr )
            sock.bind( sockaddr )
            sockaddr = Socket.pack_sockaddr_in( dstport, ns )
            sock.connect(sockaddr)
          end

Unfortunately, JRuby doesn’t support TCPSocket.new with 4 parameters. *sigh*

Recently I was looking at some of our JUnit tests to work out why they were taking quite so long to run. After going down a few dead ends, I found one big reason. The <junit> task in ant had its fork attribute set to “true”. I had naively assumed that this would fork a new JVM in which to run the tests. In fact, the documentation says:

fork:  	Run the tests in a separate VM.

but you need to look at the next line which tells you that the forkmode attribute controls how this forking occurs. The default is per test! So each test incurs a JVM startup overhead. No wonder the tests were taking a long time!

Having the tests run each in its own JVM also covers up problems. You can create whatever sort of mess you like and it’ll all be swept away before the next test runs. While going through a setting the forkmode to “once” I found a database connection leak in some test code. This sort of problem would be much more visible if all the test code was running in a single JVM and reminded me of Martin Fowler’s advice on testing resource pools.

It’s hard to figure out the best way to design a new Ruby networking application to work on all Ruby platforms.

Quite apart from the issue of UDPSocket not existing in JRuby 1.0, which puts a bit of a damper on JRuby DNS implementations, the threading issue is a bit of a nightmare.

One might think to write a single-threaded application using continuations. This would avoid all the difference between green/native-threaded Ruby implementations, and could provide a truly Ruby-esque application. But then, continuations don’t exist in JRuby either, and will be removed from Ruby 2.0. Oh, well…

[I’m also not convinced by the maintainability of code which makes heavy use of continuations - multi-threaded code is bad enough, without trying figure out control flow from a bunch of callcc calls!]

One could run a new thread per DNS query - although this presents few problems in the MRI Ruby 1.8.6 implementation, it would not scale well in the JRuby native-threaded world. One also has to worry about threading, I/O and timeouts in the Windows world. :0(

In the original Ruby resolv.rb implementation, the issue was solved by having all queries run over a single port. Then, there was a single thread calling the synchronous recvfrom() for all communications on the port. Each new client call would run under a timeout - making each client call synchronous, although users could run mutliple queries in multiple client threads if desired. Given that the latest RFC drafts specifically recommend against using a single port, we’d need to have a recvfrom thread *and* a timeout thread for *each* concurrent query - yuk!

In the end, it seems that the only sensible way to write this is a classic producer/consumer two-threaded solution : have a select thread running, which looks after all the socket input, sending responses back to the client thread through a queue.

Of course, it would have been nice to have been able to use continuations instead of a queue, but we’ve been there before! Also, if a native-threaded Ruby ever implemented continuations, it’s far from clear (to me!) what thread the continuation would actually run in. If it ran in the calling thread, you’d run the risk of stopping the select loop while a poorly behaved client spent an age processing a query response - or, even worse, made a synchronous send call!

The client thread makes an asynchronous send_async() call, which synchronously sends data over a socket, adds the socket to the array listened for in the select() call of the select thread, and returns, able to make more asynchronous calls while checking the status of the response Queue.

Now all I need is UDPSocket support in JRuby! [Or a pure Ruby EventMachine implementation!]