Dnsruby threading and continuations
(1 votes, average: 4 out of 5)
Loading ...It’s hard to figure out the best way to design a new Ruby networking application to work on all Ruby platforms.
Quite apart from the issue of UDPSocket not existing in JRuby 1.0, which puts a bit of a damper on JRuby DNS implementations, the threading issue is a bit of a nightmare.
One might think to write a single-threaded application using continuations. This would avoid all the difference between green/native-threaded Ruby implementations, and could provide a truly Ruby-esque application. But then, continuations don’t exist in JRuby either, and will be removed from Ruby 2.0. Oh, well…
[I’m also not convinced by the maintainability of code which makes heavy use of continuations - multi-threaded code is bad enough, without trying figure out control flow from a bunch of callcc calls!]
One could run a new thread per DNS query - although this presents few problems in the MRI Ruby 1.8.6 implementation, it would not scale well in the JRuby native-threaded world. One also has to worry about threading, I/O and timeouts in the Windows world. :0(
In the original Ruby resolv.rb implementation, the issue was solved by having all queries run over a single port. Then, there was a single thread calling the synchronous recvfrom() for all communications on the port. Each new client call would run under a timeout - making each client call synchronous, although users could run mutliple queries in multiple client threads if desired. Given that the latest RFC drafts specifically recommend against using a single port, we’d need to have a recvfrom thread *and* a timeout thread for *each* concurrent query - yuk!
In the end, it seems that the only sensible way to write this is a classic producer/consumer two-threaded solution : have a select thread running, which looks after all the socket input, sending responses back to the client thread through a queue.
Of course, it would have been nice to have been able to use continuations instead of a queue, but we’ve been there before! Also, if a native-threaded Ruby ever implemented continuations, it’s far from clear (to me!) what thread the continuation would actually run in. If it ran in the calling thread, you’d run the risk of stopping the select loop while a poorly behaved client spent an age processing a query response - or, even worse, made a synchronous send call!
The client thread makes an asynchronous send_async() call, which synchronously sends data over a socket, adds the socket to the array listened for in the select() call of the select thread, and returns, able to make more asynchronous calls while checking the status of the response Queue.
Now all I need is UDPSocket support in JRuby! [Or a pure Ruby EventMachine implementation!]
July 6th, 2007 at 4:28 am
Ask and you shall receive! A JRuby community member just submitted a UDPSocket implementation that was committed today. Check out trunk to give it a shot and help us work out any bugs.
July 6th, 2007 at 11:11 am
Wow - that was pretty neat timing!
I’ve used the JRuby trunk to test pnet-dns, and the UDP support is working just fine. I had to change a few Socket calls in my code, but nothing too serious.
However, I still can’t run pnet-dns in TCP mode under JRuby (there’s another post about that). It would be nice if TCPSocket supported the full MRI interface, even if full Socket support isn’t there.
Thanks very much!
July 28th, 2007 at 4:37 am
Just FYI, but there is now a working Java implementation of EventMachine that is in the SVN on rubyforge. It is still in development, but is functional at this point.
I’d love to see a resolver implemented as an EventMachine protocol, too. That would be useful.
August 6th, 2007 at 9:00 am
Hi Kirk -
Thanks for the info - I’ll check it out soon.
I do plan to make an EventMachine DNS implementation too; just as soon as I’ve finished a pure Ruby version. A first release should be up on Rubyforge pretty soon now.
Thanks,
Alex.
October 11th, 2007 at 8:17 am
Just FYI, there is now an EventMachine implementation of Dnsruby available on the trunk of Dnsruby at Rubyforge. I plan to make a gem release soon, but would appreciate any comments from anyone interested.
I’ll be blogging about it soon, but this version basically returns an EventMachine::Deferrable for any asynchronous queries.
To use :
require ‘Dnsruby’
res = Dnsruby::Resolver.new
Dnsruby::Resolver.use_eventmachine
Dnsruby::Resolver.start_eventmachine_loop(false)
df = res.send_async(Dnsruby::Message.new(”example.com”))
df.callback {|msg| puts “Response : #{msg}”}
df.errback {|msg, err|
puts “Response : #{msg}”
puts “Error: #{err}”
}
October 11th, 2007 at 9:04 am
Sorry - that last code sample was half-baked. It should of course be :
require ‘Dnsruby’
require ‘eventmachine’
res = Dnsruby::Resolver.new
Dnsruby::Resolver.use_eventmachine
Dnsruby::Resolver.start_eventmachine_loop(false)
EventMachine::run {
df = res.send_async(Dnsruby::Message.new(”example.com”))
df.callback {|msg|
puts “Response : #{msg}”
EM.stop}
df.errback {|msg, err|
puts “Response : #{msg}”
puts “Error: #{err}”
EM.stop}
}
August 14th, 2008 at 10:05 am
[…] was a nasty memory leak here, and performance was not totally reliable (see tales of woe here and here). This has now been fixed, and the select thread seems to work just fine (with 100 threads […]