random technical thoughts from the Nominet technical team

Dnsruby and EventMachine

1 Star2 Stars3 Stars4 Stars5 Stars (5 votes, average: 4.6 out of 5)
Loading ... Loading ...
Posted by alexd on Oct 12th, 2007

I recently spent some time enabling my Ruby DNS library (Dnsruby) to use EventMachine.

Dnsruby comes with its own (pure Ruby) event loop. This allows all the I/O to be done in one thread, the event loop, while the client code runs in its own thread. However, given the limitations of Ruby’s threads and select, and the variety of behaviours that exist on different platforms, it’s hard to get a pure Ruby system to work perfectly everywhere.

EventMachine gets round this issue by coding the guts of the I/O in C++ (or Java). The Ruby client code calls out to the native extensions to actually do the I/O, and all the Ruby code lives in one single thread, thus eliminating any irritating shared data issues and avoiding the whole Ruby thread quagmire.

So, I thought it would be nice for Dnsruby to be able to use the EventMachine library, when it was installed on the local platform. It would also make it easy for EventMachine users to use Dnsruby to make DNS queries from within their EventMachine loop.

Here are a few notes about the development.

Deferrables

The EventMachine::Deferabble interface really appealed to me, once I had got my head round the idea. Basically, you set up a deferrable to do something (in this case, send a Dnsruby query) :

deferrable = resolver.send_async(Message.new("example.com"))

Then you set up a callback and error callback (errback) Proc to be called on completion :

deferrable.callback {|response| 
                     print "Response received : #{response}"}
deferrable.errback {|msg, errror| print "Error : #{error}"}

Code tells the Deferrable it’s complete by calling :

deferrable.set_deferred_status :succeeded, response

Once the Deferrable is complete, then EventMachine will call the appropriate callback Proc. Of course, this all happens in the single-threaded event loop, so there’s no worries about race hazards.

Dnsruby comes with its own asynchronous interface, which uses the Queue class. I thought it would be best to combine the two – so code can use Queue (and switch between I/O implementation) or the Deferrable interface (which is used internally in the Dnsruby EventMachine implementation).

Timers

I had some troubles with timers at first. EventMachine presents a few alternatives :

  • Connection#comm_inactivity_timeout
  • EventMachine::Timer
  • EventMachine::add_timer
  • I couldn’t get comm_inactivity_timeout to work.For a timeout of 2 seconds, I’d get 1.3 seconds for UDP, and 3.75 for TCP.

    EventMachine::Timer also has a couple of issues : firstly, the one second granularity, and secondly, the limit of 1000 timers.

    I could counter the second issue by maintaining an ordered list of timeouts in the Dnsruby code, and using a single heartbeat timer from EventMachine. I thought I’d be clever, and hook it up to the EventMachine Reactor core by scheduling the heartbeat with EventMachine#next_tick. However, this was called with such high frequency that it burned up all my CPU cycles! I guess there’s also a certain context switching penalty between the native extension and the Ruby code.

    EventMachine::add_timer turned out to be the call to use. I had assumed that it offered the same one second precision of EventMachine::Timer, but a closer read of the documentation revealed the ability to get up to about 10Hz by default.

    Windows (and Java) Troubles

    I started my development on Windows, and this proved to be a problem. I was using the EventMachine 0.8.1 gem, and found it difficult to control. It seemed fine for small numbers of packets, but didn’t like to handle many concurrently open connections. Once I switched to the Mac, things got a lot easier.

    I thought I must be doing something wrong, but the same code works perfectly on the Mac (admittedly using EventMachine 0.9.0), so I just don’t know.

    I was also quite excited to discover that there is a Java implementation of EventMachine in subversion. It looks like this must still be a work-in-progress, however, as I get a LoadError for em_reactor when trying to require ‘eventmachine’.

    EventMachine::reactor_running?

    EventMachine 0.9.0 has a new method – reactor_running? This method is well needed!

    It turns out that EventMachine doesn’t like having two loops started in one Ruby process – in fact, it will terminate the process immediately. I’m not entirely sure why this is, but I’m sure there’s a good reason…

    Dnsruby doesn’t know whether the client code is written in an EventMachine style, running in an outer EventMachine::run {} call, or if it is standard Dnsruby client code, which doesn’t know to start the EventMachine loop. The reactor_running? method allows us to check whether a loop is running and to start one if not. Unfortunately, this is not present in all EventMachine implementations. Some fairly unpleasant code has to live in Dsnruby to handle the EventMachine 0.8.1 version, which omits this method.

    Getting hold of the code

    The code is available on rubyforge, and is also available as a gem :

    gem install dnsruby

    7 Responses

    1. roger pack Says:

      dnsruby rox!

    2. Ruby EventMachine - The Speed Demon - igvita.com Says:

      [...] No threading, and yet we still finish 40 requests in ~4 seconds – the only limitation is the threaded server we built in the previous example, which maxes out at 20 concurrent requests. Magic, almost! This is the beauty of EventMachine: if you can structure your worker to defer, or block on a socket, the Reactor loop will continue processing other incoming requests. When the deferred worker is done, it generates a success message and reactor sends back the response. The sky is the limit here, no Ruby threads, no synchronous processing. For a great example of this pattern, take a close look at Dnsruby. [...]

    3. dima Says:

      How can i sent Dnsruby::Message to client with all resolved information ?

    4. Alex Says:

      Hi Dima –

      Sorry, I’m not quite sure what you mean by your question.

    5. Aman Gupta Says:

      EM::Timer is a very thin wrapper for EM.add_timer, so both can be used with floats for granular timeouts. Also, the max timers can be increased using EM.set_max_timers(1_000_000)

      Since 0.12.1, EM includes EM.reactor_running?, and EM.run calls can be nested without problems.

    6. Markus Jais on software development » 9 websites about Ruby/EventMachine Says:

      [...] Dnsruby and EventMachine. Enabling Dnsruby to use EventMachine. [...]

    7. Lâmôlabs » Delicious Bookmarks for April 3rd from 13:32 to 13:55 Says:

      [...] techblog » Blog Archive » Dnsruby and EventMachine – April 3rd ( tags: example ruby dns messaging eventmachine dnsruby ) [...]

    Leave a Comment

    Please note: Comment moderation is enabled and may delay your comment. There is no need to resubmit your comment.

    Recent Posts

    Highest Rated

    Categories

    Archives

    Meta: