techblog 

I’ve just released version 0.9.7 of the dnsjnio project. Here are the release notes :

Release notes for dnsjnio version 0.9.7

o Added LookupAsynch (thanks Stefano Bagnara!) and LookupAsynchTest.
o Working version of ExtendedNonblockingResolver (and test code) added.
o Added non-single-port usage of dnsjnio. Instead of always running on
the same port (when possible), dnsjnio can now be configured to run
a new port for every query (to help prevent spoofing).
o Header ID (uniqueId) now short as ID is only 16 bits - thanks
Norman Maurer!
o Threads now call setDaemon(true) - thanks Stefano Bagnara!

If you’re interested in an asynchronous single-threaded dnsjava implementation then check out dnsjnio.

I’ve been having a think about how the threading model should work in the dnsruby library (more on that in another post), and have been looking at those of other DNS libraries. I thought it might be useful to note them down here :

dnsjava

This library comes in layers. A SimpleResolver targets a single nameserver, performing no retries (other than normal TCP retries if appropriate). When a query is sent by a SimpleResolver, a new socket is opened and a new thread is fired up to handle the query (if asynchronous).

An ExtendedResolver sits on top of a collection of SimpleResolvers, with each one configured to communicate with a different nameserver. The ExtendedResolver then handles retries over different nameservers. Each query in each nameserver (and retried queries on the same nameserver) runs in a new thread on a new socket.

dnsjnio

dnsjnio is an extension to dnsjava that uses the java.nio library. It runs multiple queries in just 3 threads (one client, one select and one timer) over either a single or multiple ports. It allows heavy asynchronous use of the dnsjava library.

Net::DNS / pnet-dns

[pnet-dns is a Ruby port of perl’s Net::DNS and therefore has the same threading model]

These projects have a different approach to dnsjava. Instead of many resolver objects each speaking to a different nameserver in different threads, with one controlling thread handling retries, etc., the Net::DNS approach is to have everything related to a lookup run in one thread. However, with no asynchronous calls, you’d have to start a new query in a new thread if you wanted to run concurrent queries. You would also have to configure a Resolver specially if you wanted it to target a single nameserver with no retries (although this is possible).

Each time a new lookup query is issued, a new socket is opened by Net::DNS (unless a persistent socket is requested). This socket is then used in a select loop with timeouts. When a timeout occurs on a nameserver then a new query is sent, along with the next nameserver being tried, all on the same socket.

Ruby resolv.rb

This file is a pure Ruby DNS implementation. It can replace the OS calls made as standard by the language, thus bypassing the notorious “blocking lookup” problem. This problem occurs because all Ruby “threads” are in fact interpreter threads - so if one of them blocks on a system call, the whole of Ruby blocks.

This meant that native DNS lookup calls could grind the whole of Ruby to a halt. To avoid this, the resolv.rb library was written, which can be used by invoking :

require 'resolv-replace.rb'

The resolv.rb library is written in pure Ruby and allows the interpreter to continue whilst one thread is waiting for a response.

However, the library uses a single socket for each instance, which is not encouraged. I don’t think it could work if it were to use a different socket for each query, which will present a challenge for dnsruby.

It also provides no asynchronous calling mechanism, although different threads can be started to run different queries if desired.

Again, there is no mechanism to issue a single query packet (other a pathologically configured Resolv instance). The library will, by default, attempt retries over multiple nameservers (if configured).

While working in a rails project which involved several date and time fields, I hit some formating issues. After some research on the net and following a few forum threads related to dates and times in RoR, I come up with a partial solution to my problems in this post:

http://www.methods.co.nz/rails_date_kit/rails_date_kit.html

The article explains the problem for dates and I needed a solution for both, dates and times. I also wanted to simplify the solution so I dug on to the root of the problem and came up with a different solution.

I am describing in this post how I fixed the problem for dates and times when we want other than the default format in a rails application. I separated the issues into two groups Output and Input. Output is when rails gets a date or datetime object from database and converts it to string to show it on a view. Input is when rails reads the string representing a date or time from a text field and converts it to a date or time object. However both are interrelated and at some point both involve the same objects and method calls.

Output formating problem

If we have the date 20/06/2007 on a date field in your database and want to show it textually on a view it is presented in the format “2007-06-20″, similarly if you have a datetime field you get something like “Wed Jun 20 20:30:00 +0100 2007″

This may not be the format we want when displaying dates and times, so we look at ways to change the default format. The default format can be changed by setting the preferred date display format into ./config/environment.rb:

ActiveSupport::CoreExtensions::Date::Conversions::DATE_FORMATS.
    merge!(default => '%d/%m/%Y')

ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.
    merge!(default => '%d/%m/%Y %H:%M')

This will make Rails to show dates on the format you have chosen. However as Rails uses the default date format (which we just changed) to store dates to the DB, if your DB does not understand the format you set then null values will be stored.

The problem in this case is (at version 1.15.3 of activerecord) located in the file lib/active_record/connection_adapters/abstract/quoting.rb and can be fixed by dropping the [quoting_patch.rb] file into your application lib directory and add the following to your ./config/environment.rb file:

# Apply patch for date and date times quoting
ActiveRecord::ConnectionAdapters::Quoting.
    send(:include, QuotingPatch)

This patch corrects both, date and datetime conversion to string and quoting for the database.

Input formating problem

If we use a text field to input a date and enter “10/06/2007″ we get into the database “2007-10-06″. So the parser has interpreted the string as an American date format and set the month to be 10. If we write into the field “20/06/2007″ The parser returns null as 20 is not a good enough month and the database gets a null or fails if null are not permitted. A similar problem arises if we introduce times such as “10/06/2007 20:44″

Rails date handling when reading from a field calls ActiveRecord::ConnectionAdapters::Column.string_to_date(string) which in turn calls ParseDate.parsedate(string), but the parser does not have the current display format into account. The method tries to guess the date as best as it can but some formats are highly ambiguous. Analogously, when a time is being handled, a similar problem appears as rails calls ActiveRecord::ConnectionAdapters::Column.string_to_time(string).

To solve these issues we need to change the behavior of the methods

ActiveRecord::ConnectionAdapters::Column.string_to_date(string)

and

ActiveRecord::ConnectionAdapters::Column.string_to_time(string)

so they first try to match the date with the display format we have set as mentioned above. I do this by parsing the date using the default format with the DateTime.strptime method.

This solution alone, however, creates a new problem: The method string_to_date and string_to_time are called twice on the column class, one when the date is loaded from the database and another when it is loaded from a form field. Therefore the first call will carry a string in the form "yyyy-mm-dd" for a date field or "yyyy-mm-dd hh:mm:ss" in case of a datetime field which is the format the database returns. But in the second case the string will be on the default format mentioned above "dd/mm/yyyy" or "dd/mm/yyyy hh:mm" as the user is expected to input it on the field.

To avoid this problem, both patched methods fall back to the guessing procedure when the date or time fails to be matched to the default format so returning correct values in both cases.

To override the string_to_date method just drop the patch file [column_patch.rb] into your lib directory and require it from your ./config/environment.rb:

# Apply patch for date and date input
require 'column_patch'

This have been tested with "%d/%m/%Y" and "%d/%m/%Y %H:%M" formats, let me know the results of your experiences with other formats.

Very silly, I know, but it took me some time to change the project status on one of my Rubyforge projects. Just in case anyone else out there is suffering, here’s how I did it :

First off, log in to Rubyforge and navigate to your project’s Admin tab. You might have thought that there would be an obvious mechanism to change your project status here, but it’s actually hidden away in the “Trove categorization” settings. You need to edit the Trove category; the project development status is in the second box down.

We have recently released version 1.0 of a port of the perl Net::DNS library to Ruby. This port has been done line by line, and preserves the original perl API. This allows existing perl applications to be ported easily to Ruby, as well as giving new Ruby programs access to a complete DNS implementation (although it should be noted that the Net::DNS::SEC module has not been ported).
All resource record types are implemented and EDNS is supported, as are zone transfers and dynamic updates.
The project can be found on Rubyforge here.

A native Ruby implementation (dnsruby) will be released soon. This version presents a clean, full DNS implementation in Ruby with a well-designed API borrowing the best aspects of Net::DNS and dnsjava.

Functional indexes are a very useful feature of Oracle that in effect pre-calculate the result of applying a function to a table column and index that. This is often used to allow case insensitive queries. So if you have a table CUSTOMERS with a NAME column you can create an index on LOWER(NAME) to index the name converted to lower case. So long as your code also uses LOWER(NAME) (and not UPPER(NAME)!) in its queries, then your case insensitive lookups will be nice and fast.

I recently ran across a bit of a gotcha when using functional indexes with outer joins. I will describe this with the following simple tables. They describe some data held by a large supermarket:

CREATE TABLE CUSTOMERS(ID NUMBER, NAME VARCHAR2(100));
CREATE TABLE LOYALTY_CARDS(ID NUMBER, CUSTOMER_ID NUMBER, EMAIL VARCHAR2(1000));

So there is a table of customers and a table of loyalty cards (and hopefully a foreign key constraint between the two, which I’ve left out). Suppose a customer can either have a loyalty card or not. They can’t have more than one. The EMAIL column holds their email address - they must provide one to have a loyalty card. We also have a PL/SQL function Clean_Up() that removes extraneous whitespace from the email, puts it in lower case etc.

Now, since we often want details about a customer complete with their loyalty card (if any) we define a view that encapsulates this join. We want customers with and without loyalty cards to be returned, so we use an outer join:

CREATE OR REPLACE VIEW CUSTOMER_AND_CARD AS
SELECT C.ID, C.NAME, L.ID LOYALTY_CARD_ID, Clean_Up(L.EMAIL) EMAIL
FROM CUSTOMERS C, LOYALTY_CARDS L
WHERE C.ID = L.CUSTOMER_ID (+);

(I use the old fashioned Oracle syntax for outer joins because I’m an old time Oracle user - sorry). Now suppose we want to run queries against this view, based on the cleaned up email address. Fair enough, so we create a functional index on Clean_Up(EMAIL). So when we run a query such as:

SELECT *
FROM CUSTOMER_AND_CARD
WHERE EMAIL = 'chris@nominett.orc.uk';

we would expect it to use the functional index to quickly find if there was a matching loyalty card. If not, it would quickly return no rows, if so, it would quickly return the data.

Unfortunately this is not the case. I think this is because the function is applied to the result of the outer join. So in theory it could be applied to a ‘null row’ (is that the correct terminology?) returned when there is no matching row in the outer joined table. When I hit this particular problem it ended up doing a full table scan on the equivalent of the CUSTOMER table which, as in this example, would likely be catastrophic for performance.

The solution is to reassure the optimizer that you are not interested in the possibility of a ‘null row’ coming from the LOYALTY_CARD table:

SELECT *
FROM CUSTOMER_AND_CARD
WHERE EMAIL = 'chris@nominett.orc.uk'
AND LOYALTY_CARD_ID IS NOT NULL

Now the optimizer will start using the functional index again and all is well with the world.

I can’t help but think that this is in fact a bug, or at the very least a limitation of functional indexes. If you are using an ordinary (i.e. non-functional) index this problem does not occur.