techblog 

I’ve been having some issues with Ruby IO#select() recently, which I think I now have my head around.

My Java non-blocking DNS project, dnsjnio, has a select loop in one thread, which processes all of the IO for all DNS queries. The client thread can wait on a blocking queue for more responses from the select thread. I had hoped to follow this model for Dnsruby. However, there are some differences between the select implementation in Ruby and Java which make things a little more tricky.

In Java, there is a Selector.wakeup() method which a client thread can call to rouse the select thread from its slumber. This means that you can send something (or ask the select thread to send something), and then wake the select thread up to deal with it. Otherwise, nothing would happen until the select() call timed out, or something happened on one of the descriptors it was monitoring - which could be a long time on a quiet system.

Ruby doesn’t have this facililty. So once a thread calls select(), it’s not going to do anything until either the timeout or a descriptor event occurs.

I read a comment by Dave Thomas in which he suggested waking the select thread up with an Exception. Well, I tried that, and it does work, but it doesn’t seem happy under JRuby. OK, it is possible to set up simple test code to get this working in JRuby. However, when you start hitting it hard it just doesn’t seem to work right, and I haven’t had the time to dig around in JRuby to figure out why.

UPDATE - I think this probably has something to do with it. Sounds like JRuby is having problems with Thread.critical=, which was getting used heavily in my multi-threaded stress testing code!

It would be nice to try to get the exception signalling working again in the future - the code is all still there. However, it is necessary to use excessive synchronisation between the threads in order to ensure that the exception is only fired at the time the select call is blocking. Basically, all the processing of the results of select has to be done in a synchronized block, guarded by the same Mutex that guards the exception raising.

Another possibility is to use pipes to signal the select that it is time to wakeup. Whilst this is a hack, it was still worth trying. The problem with it is that (once again) the behaviour or Ruby varies across platforms. Pipes aren’t implemented in Windows MRI, so we can’t use them.

This leaves the only option being to run select with a short timeout. Of course, this leaves us with a thread effectively in polling mode, which is not good. So, once the thread has not serviced any queries for a defined period of time (currently 1 second), it shuts itself down, to be restarted the next time a client query is sent. This leaves us with a worst case not much worse than the standard resolv.rb, and a best case significantly better!

It would be nice if there a non-hacky way to do this. If anyone knows of one, I’d love to hear it!!

For those who care, I’ve just released version 1.0.1 of pnet-dns, the Ruby direct port of perl’s Net::DNS.
This release adds JRuby support - for everything other than TCP sockets, that is. Hopefully JRuby will fix this soon; if so, then this release of pnet-dns should fully support JRuby.
This release also fixes a bug to do with OPT record processing. OPT records should now be handled correctly.
This release also conforms to a change in the Digest::MD5 API between Ruby 1.8.5 and 1.8.6.
This release has been tested on MRI 1.8.6 for Windows, Linux and Solaris.
Give it a go - feedback is appreciated!

I’ve been trying (again) to get my Ruby DNS library working with JRuby. I’ve been having some difficulties - not all of which are JRuby’s fault!

I need to be able to send data from a TCP port and receive a response. Many connections may be running from the same socket, and I need to be able to tell where a response has come from. The easy way to do this would be use TCPSocket#recvfrom - unfortunately, that raises an exception on Windows for Ruby 1.8.5 and 1.8.6 in MRI.

That leaves me with the option of creating a Socket directly, which is what pnet-dns does. Unfortunately, this API is still not fully supported by JRuby, so I’m stuck with TCPSocket. However, I thought I could still do something like :

          if (/java/=~RUBY_PLATFORM)
            sock = TCPSocket.new(ns, dstport, srcaddr, srcport)
          else
            sock = Socket.new( Socket::AF_INET, Socket::SOCK_STREAM, 0 )
            sockaddr = Socket.pack_sockaddr_in( srcport, srcaddr )
            sock.bind( sockaddr )
            sockaddr = Socket.pack_sockaddr_in( dstport, ns )
            sock.connect(sockaddr)
          end

Unfortunately, JRuby doesn’t support TCPSocket.new with 4 parameters. *sigh*

It’s hard to figure out the best way to design a new Ruby networking application to work on all Ruby platforms.

Quite apart from the issue of UDPSocket not existing in JRuby 1.0, which puts a bit of a damper on JRuby DNS implementations, the threading issue is a bit of a nightmare.

One might think to write a single-threaded application using continuations. This would avoid all the difference between green/native-threaded Ruby implementations, and could provide a truly Ruby-esque application. But then, continuations don’t exist in JRuby either, and will be removed from Ruby 2.0. Oh, well…

[I’m also not convinced by the maintainability of code which makes heavy use of continuations - multi-threaded code is bad enough, without trying figure out control flow from a bunch of callcc calls!]

One could run a new thread per DNS query - although this presents few problems in the MRI Ruby 1.8.6 implementation, it would not scale well in the JRuby native-threaded world. One also has to worry about threading, I/O and timeouts in the Windows world. :0(

In the original Ruby resolv.rb implementation, the issue was solved by having all queries run over a single port. Then, there was a single thread calling the synchronous recvfrom() for all communications on the port. Each new client call would run under a timeout - making each client call synchronous, although users could run mutliple queries in multiple client threads if desired. Given that the latest RFC drafts specifically recommend against using a single port, we’d need to have a recvfrom thread *and* a timeout thread for *each* concurrent query - yuk!

In the end, it seems that the only sensible way to write this is a classic producer/consumer two-threaded solution : have a select thread running, which looks after all the socket input, sending responses back to the client thread through a queue.

Of course, it would have been nice to have been able to use continuations instead of a queue, but we’ve been there before! Also, if a native-threaded Ruby ever implemented continuations, it’s far from clear (to me!) what thread the continuation would actually run in. If it ran in the calling thread, you’d run the risk of stopping the select loop while a poorly behaved client spent an age processing a query response - or, even worse, made a synchronous send call!

The client thread makes an asynchronous send_async() call, which synchronously sends data over a socket, adds the socket to the array listened for in the select() call of the select thread, and returns, able to make more asynchronous calls while checking the status of the response Queue.

Now all I need is UDPSocket support in JRuby! [Or a pure Ruby EventMachine implementation!]

I’ve been having a think about how the threading model should work in the dnsruby library (more on that in another post), and have been looking at those of other DNS libraries. I thought it might be useful to note them down here :

dnsjava

This library comes in layers. A SimpleResolver targets a single nameserver, performing no retries (other than normal TCP retries if appropriate). When a query is sent by a SimpleResolver, a new socket is opened and a new thread is fired up to handle the query (if asynchronous).

An ExtendedResolver sits on top of a collection of SimpleResolvers, with each one configured to communicate with a different nameserver. The ExtendedResolver then handles retries over different nameservers. Each query in each nameserver (and retried queries on the same nameserver) runs in a new thread on a new socket.

dnsjnio

dnsjnio is an extension to dnsjava that uses the java.nio library. It runs multiple queries in just 3 threads (one client, one select and one timer) over either a single or multiple ports. It allows heavy asynchronous use of the dnsjava library.

Net::DNS / pnet-dns

[pnet-dns is a Ruby port of perl’s Net::DNS and therefore has the same threading model]

These projects have a different approach to dnsjava. Instead of many resolver objects each speaking to a different nameserver in different threads, with one controlling thread handling retries, etc., the Net::DNS approach is to have everything related to a lookup run in one thread. However, with no asynchronous calls, you’d have to start a new query in a new thread if you wanted to run concurrent queries. You would also have to configure a Resolver specially if you wanted it to target a single nameserver with no retries (although this is possible).

Each time a new lookup query is issued, a new socket is opened by Net::DNS (unless a persistent socket is requested). This socket is then used in a select loop with timeouts. When a timeout occurs on a nameserver then a new query is sent, along with the next nameserver being tried, all on the same socket.

Ruby resolv.rb

This file is a pure Ruby DNS implementation. It can replace the OS calls made as standard by the language, thus bypassing the notorious “blocking lookup” problem. This problem occurs because all Ruby “threads” are in fact interpreter threads - so if one of them blocks on a system call, the whole of Ruby blocks.

This meant that native DNS lookup calls could grind the whole of Ruby to a halt. To avoid this, the resolv.rb library was written, which can be used by invoking :

require 'resolv-replace.rb'

The resolv.rb library is written in pure Ruby and allows the interpreter to continue whilst one thread is waiting for a response.

However, the library uses a single socket for each instance, which is not encouraged. I don’t think it could work if it were to use a different socket for each query, which will present a challenge for dnsruby.

It also provides no asynchronous calling mechanism, although different threads can be started to run different queries if desired.

Again, there is no mechanism to issue a single query packet (other a pathologically configured Resolv instance). The library will, by default, attempt retries over multiple nameservers (if configured).

While working in a rails project which involved several date and time fields, I hit some formating issues. After some research on the net and following a few forum threads related to dates and times in RoR, I come up with a partial solution to my problems in this post:

http://www.methods.co.nz/rails_date_kit/rails_date_kit.html

The article explains the problem for dates and I needed a solution for both, dates and times. I also wanted to simplify the solution so I dug on to the root of the problem and came up with a different solution.

I am describing in this post how I fixed the problem for dates and times when we want other than the default format in a rails application. I separated the issues into two groups Output and Input. Output is when rails gets a date or datetime object from database and converts it to string to show it on a view. Input is when rails reads the string representing a date or time from a text field and converts it to a date or time object. However both are interrelated and at some point both involve the same objects and method calls.

Output formating problem

If we have the date 20/06/2007 on a date field in your database and want to show it textually on a view it is presented in the format “2007-06-20″, similarly if you have a datetime field you get something like “Wed Jun 20 20:30:00 +0100 2007″

This may not be the format we want when displaying dates and times, so we look at ways to change the default format. The default format can be changed by setting the preferred date display format into ./config/environment.rb:

ActiveSupport::CoreExtensions::Date::Conversions::DATE_FORMATS.
    merge!(default => '%d/%m/%Y')

ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.
    merge!(default => '%d/%m/%Y %H:%M')

This will make Rails to show dates on the format you have chosen. However as Rails uses the default date format (which we just changed) to store dates to the DB, if your DB does not understand the format you set then null values will be stored.

The problem in this case is (at version 1.15.3 of activerecord) located in the file lib/active_record/connection_adapters/abstract/quoting.rb and can be fixed by dropping the [quoting_patch.rb] file into your application lib directory and add the following to your ./config/environment.rb file:

# Apply patch for date and date times quoting
ActiveRecord::ConnectionAdapters::Quoting.
    send(:include, QuotingPatch)

This patch corrects both, date and datetime conversion to string and quoting for the database.

Input formating problem

If we use a text field to input a date and enter “10/06/2007″ we get into the database “2007-10-06″. So the parser has interpreted the string as an American date format and set the month to be 10. If we write into the field “20/06/2007″ The parser returns null as 20 is not a good enough month and the database gets a null or fails if null are not permitted. A similar problem arises if we introduce times such as “10/06/2007 20:44″

Rails date handling when reading from a field calls ActiveRecord::ConnectionAdapters::Column.string_to_date(string) which in turn calls ParseDate.parsedate(string), but the parser does not have the current display format into account. The method tries to guess the date as best as it can but some formats are highly ambiguous. Analogously, when a time is being handled, a similar problem appears as rails calls ActiveRecord::ConnectionAdapters::Column.string_to_time(string).

To solve these issues we need to change the behavior of the methods

ActiveRecord::ConnectionAdapters::Column.string_to_date(string)

and

ActiveRecord::ConnectionAdapters::Column.string_to_time(string)

so they first try to match the date with the display format we have set as mentioned above. I do this by parsing the date using the default format with the DateTime.strptime method.

This solution alone, however, creates a new problem: The method string_to_date and string_to_time are called twice on the column class, one when the date is loaded from the database and another when it is loaded from a form field. Therefore the first call will carry a string in the form "yyyy-mm-dd" for a date field or "yyyy-mm-dd hh:mm:ss" in case of a datetime field which is the format the database returns. But in the second case the string will be on the default format mentioned above "dd/mm/yyyy" or "dd/mm/yyyy hh:mm" as the user is expected to input it on the field.

To avoid this problem, both patched methods fall back to the guessing procedure when the date or time fails to be matched to the default format so returning correct values in both cases.

To override the string_to_date method just drop the patch file [column_patch.rb] into your lib directory and require it from your ./config/environment.rb:

# Apply patch for date and date input
require 'column_patch'

This have been tested with "%d/%m/%Y" and "%d/%m/%Y %H:%M" formats, let me know the results of your experiences with other formats.

Very silly, I know, but it took me some time to change the project status on one of my Rubyforge projects. Just in case anyone else out there is suffering, here’s how I did it :

First off, log in to Rubyforge and navigate to your project’s Admin tab. You might have thought that there would be an obvious mechanism to change your project status here, but it’s actually hidden away in the “Trove categorization” settings. You need to edit the Trove category; the project development status is in the second box down.

We have recently released version 1.0 of a port of the perl Net::DNS library to Ruby. This port has been done line by line, and preserves the original perl API. This allows existing perl applications to be ported easily to Ruby, as well as giving new Ruby programs access to a complete DNS implementation (although it should be noted that the Net::DNS::SEC module has not been ported).
All resource record types are implemented and EDNS is supported, as are zone transfers and dynamic updates.
The project can be found on Rubyforge here.

A native Ruby implementation (dnsruby) will be released soon. This version presents a clean, full DNS implementation in Ruby with a well-designed API borrowing the best aspects of Net::DNS and dnsjava.

Whilst working through the examples in the “Ajax on Rails” book I had all sorts of problems with Prototype and script.aculo.us. For a start, there is a simple example of a link that toggles a div between visible and invisible, like this:

<%= link_to_function "Toggle DIV", "$('indicator').toggle()" %>

But all I got with this was the error message “toggle is not a function. I even tried replacing toggle() with show() and got another error. Eventually I upgraded from Prototype 1.4 to 1.5 and the problem went away.

Now all through this, script.aculo.us had been working fine. But then I tried another simple example:

Effect.toggle('indicator', 'blind')

Again I got the error message that ‘toggle is not a function’. So this time I upgraded script.aculo.us to 1.7 and it all seems to work fine.

Having said that all I’ve done is upgrade within my current project so now I need to work out how to upgrade the prototype and script.aculo.us built into Rails so that all new projects start with the new versions.

I’ve been playing around with JRuby recently, and thought it would be fun to get it working with our Java middleware. Once I got my IDE support up and running (which is a story in itself!), everything seemed to pretty much “just work”. However, I did have one issue which took me a minute or two to get to the bottom of…

Our middleware makes a lot of use of the Java Proxy class - basically, all remote access is done through a proxy. I got a bit confused when the middleware started showing errors on the hashCode method. With a bit of debugging, it turned out that org.jruby.javasupport.JavaSupport.getJavaObjectFromCache was using hashCode and equals to load the Java object (the proxy) from the instance cache. These methods (which weren’t explicitly implemented by the proxy), were sent to the invoke proxy method, which then tried to send them over the wire to the middleware!

If you wish to continue to use an explicit proxy, then find your invoke method :

public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {}

and simply add the lines :

if (method.getName().equals("hashCode")) {
return hashCode();
}
if (method.getName().equals("equals")) {
return equals(args[0]);
}
if (method.getName().equals("toString")) {
return toString();
}

to the start.