techblog 

I got bitten by a silly bug in my dnsruby code recently - I thought I’d share it here in case anyone else starts pulling their hair out over this in future!

DNSSEC RRSIG records contain the signatures required to prove that a DNS zone has been correctly signed by an entity which possesses the correct keys for the zone. DNS clients can obtain the correct keys for the zone, then use the RRSIG records to prove that the zone (or record of interest) is correct. Of course, these records shouldn’t last forever - the data records are periodically re-signed (possibly with different keys), and the RRSIGs updated. The RRSIG includes an inception time, and an expiration time, to show the period over which it is valid. To verify a set of records, the DNS client must first produce an array of bytes, the digest of which is taken and used as the signature for the records - the salient data of of the RRSIG record (including the inception and expiration times) is included in this set of bytes.

The code I had written to do this was working fine - I had coded in the examples from the RFCs, and done a lot of work with actual signed zones (reading the data from the authoritative servers, and proving that it was correct). However, when I started to try to do this with real-world zone files, I started noticing that the signatures weren’t verifying. At least, they *sometimes* weren’t verifying. Very odd. Of course, with this kind of work, all you get is a Pass/Fail - no clue as to what is going wrong. I could see that the records were being translated to and from text format correctly - all the data in the RRs was showing as fine. However, when I compared the byte sequences prepared by my DNS client and Net::DNS, I noticed that four bytes were different. It turned out that, of the four byte sequences written to for the RRSIG inception and expiration time, there was a difference of 0xE100 - this works out to 16 hours worth of 3600 seconds. At last, I was onto something!

There are several ways to express time in the presentation format of the RRSIG record - “1234567890″ (seconds since 01/01/1970), or “YYYYMMDDHHMMSS” (e.g. “20090608123435″). Dnsruby worked fine with the first, and even translated the second from presentation format and back to presentation format correctly. However, when read using the following line :

return Time.mktime(year, mon, day, hour, min, sec).to_i

I got a 16 hour offset from the correct time! [When converting this time to/from the text format, the translation worked perfectly - it was only when inspecting the internal epoch time that the difference could be noticed]

I changed this to :

return Time.gm(year, mon, day, hour, min, sec).to_i

And suddenly everything worked just fine.

I’m sure that seasoned Rubyists will sneer at me for my stupidity - it did take me some time to track this one down! So, if you start noticing strange 16 hours offsets in your Ruby code, it’s worth checking your usage of the Time class…

In this post, I’d like to look at how to use Dnsruby with DNSSEC. As before, I’ll run these examples in irb, and assume that you’ve included Dnsruby there.

Dnsruby has DNSSEC support switched on by default. This means it will attempt to validate any DNS responses against its trust anchors. However, by default, no trust anchors are configured - to get dnsruby to validate responses, you must first configure a trust anchor (or DLV repository).

Trust Anchors

DNSSEC works by following a chain of trust from parent zone to child zone. This chain of trust must start somewhere - the “trust anchor”. In a world with a signed root, the root would be the anchor. Delegations to children zones would be signed, all the way down to the domain that is being queried. The querier can then be sure that the signed response is genuine.

Unfortunately, the root is not yet signed - we have many “islands of security”. Each island is signed, but has no chain to it from the root. It is possible to configure dnsruby with the keys for these zones using Dnsruby::Dnssec#add_trust_anchor() - it’s also possible to define an expiration time for each anchor. Dnsruby will then follow the chain of trust from the anchor down to the queried domain in the signed zone.

Managing these trust anchors quickly becomes a headache. You need to have secure means of obtaining and verifying them, and rolling over to new keys as time goes on. Fortunately, there are two mechanisms to help with this : IANA’s TAR and ISC’s DLV repository.

IANA (who manage the root zone) have created a Trust Anchor Repository (ITAR) which can be used until the root is signed. This holds delegation records for the DNSSEC-signed TLDs. It is possible to download this repository and configure dnsruby with the anchors. A method to do this is defined in Dnsruby::Dnssec#load_itar, but it is not currently secure. If you need to use the ITAR securely, you are currently advised to add the trust anchors from the ITAR directly into dnsruby. A secure method will be provided in future releases.

DLV

Even if the root was signed, there will still be some domains in unsigned zones, which wish to benefit from DNSSEC security. For example, signed-zone.unsigned-zone.example.org - there can be no chain of trust from the root to signed-zone. A solution exists for signed-zone : DNSSEC Lookaside Validation (DLV). Here, a DLV repository holds secure delegation records for zones like signed-zone. Instead of following the chain of trust from the root, a validator follows the chain of trust from the closest parent zone known to the DLV repository. Of course, this method involves more validation queries for each application query.

As an example, considering querying for random.example.com - first, the query itself must be made. Then, if unsuccessful, a DLV query for random.example.com.dlv.isc.org must be made, followed by a query for example.com.dlv.isc.org, followed by a query for com.dlv.isc.org. If none of these succeed, then the message cannot be validated. Imagine that a response was received for the com.dlv.isc.org zone : then, the chain of trust could be followed through example.com down to random.example.com. Keys discovered from the DLV repository are cached.

Configuring Trust Anchors

To configure a trust anchor (in this case for the uk-dnssec.nic.uk DNSSEC test zone) :

    trusted_key = Dnsruby::RR.create({:name => "uk-dnssec.nic.uk.",
        :type => Dnsruby::Types.DNSKEY,
        :flags => 257,
        :protocol => 3,
        :algorithm => 5,
        :key=> "AQPJO6LjrCHhzSF9PIVV7YoQ8iE31FXvghx+14E+jsv4uWJR9jLrxMYm sFOGAKWhiis832ISbPTYtF8sxbNVEotgf9eePruAFPIg6ZixG4yMO9XG LXmcKTQ/cVudqkU00V7M0cUzsYrhc4gPH/NKfQJBC5dbBkbIXJkksPLv Fe8lReKYqocYP6Bng1eBTtkA+N+6mSXzCwSApbNysFnm6yfQwtKlr75p m+pd0/Um+uBkR4nJQGYNt0mPuw4QVBu1TfF5mQYIFoDYASLiDQpvNRN3 US0U5DEG9mARulKSSw448urHvOBwT9Gx5qF2NE4H9ySjOdftjpj62kjb Lmc8/v+z"
      })
    Dnsruby::Dnssec.add_trust_anchor(trusted_key)

Dnsruby will now attempt to validate any responses from the uk-dnssec.nic.uk zone (or its children).

To configure dnsruby to use ISC’s DLV repository, you must first obtain the key (from here). You can then configure dnsruby :

    dlv_key = RR.create("DLV_KEY_STRING_FROM_ISC")
    Dnssec.add_dlv_key(dlv_key)

This method queries the DLV registry to get the ZSK (zone signing key) from the above KSK (key signing key). Dnsruby will now attempt to validate all responses against the DLV repository, if it can’t validate from any trust anchors.

Configuring Validation Policy

It is possible to configure the validation policy to vary the precedence of search order - from the root only, or local anchors only, or either first. Separate key caches are maintained by each validator, making it possible to configure them dynamically. DLV validation is only performed once the DLV key has been added. Here is an example of changing the validation policy :

    Dnsruby::Dnssec.validation_policy = Dnsruby::Dnssec::ValidationPolicy::ROOT_THEN_LOCAL_ANCHORS

It is possible to clear all trusted keys (which will also stop DLV validation) by calling :

    Dnsruby::Dnssec.clear_trusted_keys()

You can remove just the trust anchors (leaving DLV keys and validation from the root, and all keys generated from them), and the keys generated from them, by calling :

    Dnsruby::Dnssec.clear_trust_anchors()

Configuring Validation Resolver

When a response is validated, it may be necessary to make several more queries in order to follow the chain of trust. As more queries are made, more chains are followed. Trusted keys are cached as they are discovered (for the length of time they are indicated to be good for) - this means that future queries for domains in those zones will not require so many validation queries to be performed.

It’s possible to configure dnsruby to use different methods for performing the validation queries. They can either be directed to recursive nameservers (which can be the system defaults, or a client-supplied set of addresses), or they can be performed recursively. I have found that many resolvers do not yet speak a perfect dialect of DNSSEC - performing validation queries recursively ensures that the correct DNSSEC-signed responses are received. The default is to perform validation recursively. Of course, while the caches are being built up when dnsruby starts, more queries will be performed than if the queries were directed to recursive nameservers.

To ask dnsruby to use query a recursive nameserver, call :

    Dnsruby::Dnssec.do_validation_with_recursor = false

Dnsruby will now use the system default configured nameservers for validation.

To use a specific set of servers to perform validation :

    res = Dnsruby::Resolver.new({:nameserver => ['192.168.1.1', '192.168.2.1']})
    Dnsruby::Dnssec.default_resolver = res

Validating Responses

Once dnsruby has been configured with a trust anchor, it will attempt to validate any responses for domains within that zone (or its subzones). If it detects that validation is necessary, then it will fire up a new thread to handle that validation. Since many queries may need to be performed in order to validate the reponse, this can take some time longer than the original query would have done alone. This means that the query timing settings in the Resolver class apply only to each query - *not* to the whole validation process.

For example, a query may have a Resolver#query_timeout of 5 seconds. As long as the answer for that query is returned in 5 seconds, then no timeout will occur - even if it then takes another 6 seconds to validate that response. Future versions of dnsruby will include the ability for client applications to receive events detailing the progress of each asynchronous query (e.g. RECEIVED, VALIDATED).

It is possible to disable validation on a Message basis. Simply set :

  msg.do_validation = false

before sending the Message - dnsruby will not validate the response to that query.

Message Security Levels

Messages can have one of four security levels (defined in Dnsruby::Message::SecurityLevel) : BOGUS, UNCHECKED, INSECURE and SECURE. Dnsruby will only raise an error if it detects that a response is BOGUS - this means that the message does not contain the correct set of signatures. INSECURE means that the response has been verified to have come from a non-secured zone. SECURE means that the chain of trust has been correctly followed from a configured trust anchor to the response, and that all signatures check OK. Note that an NXDOMAIN response can still be SECURE - this means that the NSEC(3) records have been verified to prove non-existence.

To check the security level of a Message, use Message#security_level :

  if (msg.security_level == Dnsruby::Message::SecurityLevel::SECURE)
      print "Response was validated OKn"
  end

Examples of Use

DNSSEC examples can be found in the EXAMPLES file in the dnsruby distribution.

Limitations

Dnsruby does not yet perform NSEC3 validation (although NSEC3/NSEC3PARAM records can be read from the wire, or presentation format). This will be added to a future release.

In this post, I’d like to look at how to use dnsruby to accomplish some common tasks.

Getting started

To follow these examples, you’ll need to install dnsruby :

$ gem install dnsruby

I’ll run these examples in Ruby’s interactive shell :

$ irb

First, I need to include Dnsruby :

>> require 'rubygems'
>> require 'dnsruby'
>> include Dnsruby

Now I’ll load the system’s default resolvers :

>> res = Resolver.new

And display them :

>> res.single_resolvers.each {|s| print "Server address : #{s.server}n"}
Server address : 192.168.1.1
Server address : 192.168.2.2

Now I’ll use them to run a couple of queries :

>> ret = res.query("example.com") # Defaults to A record
>> print ret.answer
example.com.	172789	IN	A	208.77.188.166=> nil
>> res.query("example.com", "MX") # Query the MX record

This time, I’ll use some defined nameservers :

>> res = Resolver.new({:nameserver => ["ns1.nic.uk",
        "ns1.nic.uk"]})

Asynchronous Queries

To run an asynchronous query, I’ll define a Queue to hold the results, and then prepare the query. This time, I’ll construct a Message to hold the query data, and set the RD (recursion desired) bit on the header to 0 :

>> queue = Queue.new
>> m = Message.new("co.uk", Types.NS)
>> m.header.rd = false
>> message_id = res.send_async(m, queue, 1)

Now my code can get on with other tasks, until I’m ready to get the response. Queue#pop is a blocking call, but you can check if it is empty using Queue#empty?.

>> id, reply, error = queue.pop # id == message_id

The [id, reply, error] tuple is popped off the queue. The id identifies which query the response is for (it should match the id returned by the send_async call), reply holds the best response that was received, and any errors will be held in error (which should be nil in this example).

Message Options

Now I’ll ask for a Message to be sent without checking (or the response being stored in) the cache. I’ll also make sure that no DNSSEC validation is performed on the response :

>> m.do_caching = false
>> m.do_validation = false
>> res.send_message(m)

I can ask for a Message to be sent without any pre- or post-processing. No EDNS headers are applied, the header flags are not adjusted, and no caching or validation is performed. This method is most useful for tools authors :

>> res.send_plain_message(Message.new("nic.uk"))

TSIG and Dynamic Updates

I can also use TSIG signatures to communicate securely with a resolver. In this example, I’ll use TSIG to sign a dynamic update. First, I’ll have to define the server to use, and the TSIG key to speak to it with :

>> res = Dnsruby::Resolver.new("ns0.validation-test-servers.nominet.org.uk")
>> res.dnssec = false
>> tsig = Dnsruby::RR.create({
        :name        => "rubytsig",
        :type        => "TSIG",
         :key         => "8n6gugn4aJ7MazyNlMccGKH1WxD2B3UvN/O/RA6iBupO2/03u9CTa3Ewz3gBWTSBCH3crY4Kk+tigNdeJBAvrw==",
      })

Now I’ll create the dynamic update packet :

>> update = Dnsruby::Update.new("validation-test-servers.nominet.org.uk")
>> # ... add stuff to the update
>> update.absent("notthere.update.validation-test-servers.nominet.org.uk", 'TXT')

And apply the TSIG signature and send the message :

>> tsig.apply(update)
>> response = res.send_message(update)
>> print "TSIG response was verified? : #{response.verified?}n"

I could also have configured the Resolver to sign *all* packets with TSIG :

>> res.tsig=tsig.name, tsig.key

Recursive Queries

In addition to defining nameservers to do recursive queries on my behalf, I can also get Dnsruby to query recursively from the root. A static cache is built up, so the more client queries that are run, the less packets need be sent per client query.

>> rec = Recursor.new
>> ret = rec.query("uk-dnssec.nic.uk", "NS")

In my next article, I’ll look at how use Dnsruby with DNSSEC.

I’m very pleased to announce the release of dnsruby version 1.30 (RDoc available here). This version constitutes a considerable advance of functionality on previous dnsruby releases. Although it is still possible to use dnsruby in just the same way (which proves useful for tools authors), it is now also capable of full DNSSEC validation, using either ISC’s DLV registry, IANA’s Trust Anchor Repository, or any trust anchors it is configured with.

Clients can configure the validation policy (e.g. whether to try local trust anchors first, followed by ITAR, then DLV, or some other order), whether to use recursion or a local resolver in the validation process, and manage the store of trusted keys themselves. Dnsruby still offers the verify method to verify a Message or RRSet against a set of keys, but also includes the validate method (called automatically by default) which kicks off a separate validation thread (if validation is necessary). This thread follows the chain of trust from the closest trusted key, and records the result in the Dnsruby::Message#security_level. Exceptions are only raised if the SecurityLevel is BOGUS.

Dnsruby::Recursor now includes a static authority cache. This means that dnsruby can now be used to perform entirely recursive querying - especially useful if you don’t trust your local resolvers, or they don’t yet speak a correct dialect of DNSSEC.

Other improvements include dynamic management of configured nameservers - more responsive ones will be preferred over slower, and servers which don’t respond at all, or give broken responses, will be pushed to the back of the line. A cache has been added for received (and validated) Messages. It is possible to avoid use of this cache on a Message basis.

Support for EventMachine has been removed. I’ve had to rework dnsruby’s event system considerably in order to include validation support, and it was no longer feasible to maintain two IO loops. The native Ruby event loop in dnsruby has no known issues, and that is the recommended (and only) option. It should still be possible to work dnsruby into EventMachine projects.

I plan to write another article soon, with more detail on how to achieve common DNS(SEC) tasks with dnsruby. In the meantime, please let me know if you have any issues using the latest version!

In order to register an ENUM, a XML token (RFC5105) is signed by a Validation Agency to say that the user has a given phone number & so has the right to a ENUM registration; this token is sent by an ENUM Registrar to Nominet’s EPP server.

This process of signing a token isn’t entirely straightforward:

The simplest way to sign a token is to use the templatesign tool from the Apache XML Security project.

$ templatesign -r testVA.key password -c -x testVA.cert unsigned_token.xml > signed_token.xml

A token signed by templatesign is valid, as can be checked using checksig (from the same toolkit) or Oxygen.  However when this token is included as part of an EPP Create command, the signature becomes invalid.  It seems that templatesign doesn’t do the XML canonicalization quite right.

Aside:
To make an XML signature, a digest is taken of the XML data being signed; the data is “canonicalized” before calculating the digest.  It seems that canonicalisation is more-or-less just a standard way of inserting whitespace.  I’m surprised this is how it works because it’s rather fragile, compared to calculating the digest using the element names, attributes etc & ignoring the formatting.

One workaround for this is to run templatesign on the entire Create.  Although this is satisfactory for testing, it does not work in the situation where a Validation Agency has to send a signed token to a Registrar, who then wraps this up into a Create command & sends that to Nominet’s EPP server.

It is possible to use Oxygen to sign a token (example) and this will do the canonicalization in such a way that when enveloped in a Create command, the signature remains valid.  However, Oxygen is an interactive GUI tool and not suited to making an automated system.

Oxygen says that it uses the Java version of Apache XML Security. Looking at the examples that come with the Apache package (specifically GenEnveloped.java) gives us another option:

We’ve taken this example from the Apache library and made the appropriate changes to sign an ENUM token. I’ve packaged some example code which contains:

• EnumTokenSigner.java - signs a token in such a way that it will validate even when enveloped in a Create command.  It takes an unsigned token (as a Java String) for input, signs it and returns the signed token
• EnumTokenCreator.java - is one way to create an unsigned token.  This could be done in any number of ways but using a DOM implementation easily handles absent optional elements.
• EnumTokenSignExample.java - is a simple test harness - it just loads files & calls the two above utility classes.  You will at least have to edit this to set the path names of the keys & token properties.  It does not (for example) do any of the error-checking that you would expect in production code.
• a Makefile

Key formats

Java uses a different key format (DER) than OpenSSL (PEM).  It is necessary to convert a key+certificate to this format before using the Java tools.  OpenSSL can perform the conversion:

$ openssl pkcs8 -topk8 -nocrypt -in yourVA_key.key -inform PEM -out yourVA_key.der -outform DER
$ openssl x509 -in yourVA_cert.pem -inform PEM -out yourVA_cert.der -outform DER

I attended the 2009 Hotsos Symposium, an excellent Oracle database performance tuning conference, in Dallas. The event was a great opportunity to hear world-renowned Oracle performance experts present.

One of the most interesting talks was “The Helsinki Declaration: A set of Principles for the IT Community regarding Application Development” by Toon Koppelaars.

Toon described the expansion of features in the Oracle database over the years. He went on to explain that since the advent of Java, more and more functionality has been implemented outside the database. However new frameworks, methods and languages are appearing frequently and often disappearing quickly, sometimes within a couple of years. Many developers are constantly chasing the latest technology because it’s cool and will allegedly solve all presently-experienced problems. This leads to code quickly becoming legacy, having to be re-written and/or no developers having the necessary skills to maintain it. For example how happy or able would your Java developers be to maintain a system built using Struts, a relatively young framework, but now commonly seen as legacy. Would they first spend ages rewriting it, these days called refactoring, to use Spring, the effort for which gives no value to the user.

Although these technologies are changing, what users want has not changed; they still largely want “window on data” applications.

While this is happening the database technology is remaining stable and under-utilised.

Toon recommends replacing this traditional architecture. He has successfully deployed systems using a new architecture, named The Helsinki Declaration (that’s where it was first proposed). This architecture has a thin user-interface layer, deployed in whatever technology/framework is flavour of the month, and business logic and data logic layers implemented in the comparatively very stable database. Only the thin user interface is then vulnerable to the latest fad.

This is described well on his blog. I recommend starting with his first observation and then proceeding to the second, third and fourth observations.

Talking with conference attendees afterwards I was surprised (or maybe I shouldn’t have been) by how many had experienced exactly the issues Toon described on systems development and maintenance projects.

Yubikeys are pretty neat! They provide a simple, open and convenient package for generating one time passwords. The price is also at a point where it becomes cheap enough to add a reasonably good layer of additional security for two factor authentication.

The basic idea is to plug in the USB key and press its button. The key registers itself as a USB keyboard, so no special software is required, and the one time passwords are generated in about a second. The key and the authentication server share an AES key and the one time passwords are validated by keeping track of how many times the key has been used.

Quite a lot has already been written about Yubikeys but it did take a bit of searching to find the finer details. The following two links were the most helpful for understanding the implementation and being able to write my own.

1. Yubikey password format and test vectors
2. Yubikey simulator

See what’s inside a Yubikey generated password after the jump…
Continue Reading »

This follows on from my previous posting about Day 1. Unfortunately I was unable to attend the third day, so this is the final installment.

Web as a Platform - Dion Hinchcliffe

This keynote was about the changing landscape of application development and how cloud computing affects it.  The main point of interest was that even though cloud computing may mean vendor lock-in rearing its ugly head, the cost advantages are going to make it very difficult to ignore. He has much of the same stuff on his blog at ZDNet.

What I’ve learnt since the book - Eric Evans

In this talk Eric Evans discussed what he has learnt about domain driven design since he wrote the book on the subject.

He began by  talking about what is essential in DDD.  Collaboration with domain experts to explore and experiment is important - don’t frighten them away by going through an enormous spec line by line.  Then evolve the model and language going forward, establish context boundaries and focus on the core domains.

He said that the building blocks (services, entities etc) are less important than the book might suggest.  But he also said that “Domain Event” is a missing building block.  It can lead to clearer models by representing the state of entities (e.g. a bank account balance based on the transactions on that account) and can also decouple systems (e.g. a reporting service separate from a transaction service).

Aggregates are consistency boundaries first and foremost and should represent a conceptual whole.  You need to take care putting aggregate level properties and invariants on the root.  Sometimes you might need to make a special aggregate object or add a method to the repository.

Something that was in the book that he thought could be left out was Large Scale Structure, which doesn’t come up too often. On the other hand, he suggested a couple of new context patterns:

• Partners - Mutually Dependent and Cooperative
• Big Ball of Mud - Common.  Know that you can’t do sophisticated modelling within and accept that.

On that second pattern, he said we should accept that not all a large system will be well designed and that precision designs are fragile. It is also possible to spread modelling too thin. Ensure the core domain has a clean context boundary and focus your efforts there. To do the context mapping follow these steps (it isn’t a creative activity):

1. What models do we know of? Draw a blob and a name for each.
2. Where does each apply?
3. Where is information exchanged?
4. Which pattern maps the relationship?

Finally he talked about DDD and SOA, which is something he is often asked about.  He thinks the confusion arises because SOA services carry much more baggage than DDD services, which are just something that can do something for you.  SOA services are defined within some context and it is quite possible that the service interface may define a context boundary.

OSGi - Neil Bartlett

This was an introductory talk on modularisation using OSGi.  He started by saying that while jars have dependencies, these are implicit.  OSGi uses modules, which are just jars plus dependency metadata.  He had a tongue-in-cheek dig at SOA by pointing out that since we can’t use classes in isolation due to the dependencies we have to call them remotely instead. I was surprised to find out that Spring-OSGi had to be renamed Spring-DM (”Dynamic Modules”) due to pressure from the OSGi Alliance. The takeaway message was that you need to use something like Spring-DM to take care of OSGi for you and don’t code directly to services.

Coaching Self-Organising Teams - Joseph Pelrine

This session used cooking soup as a metaphor for coaching an agile team.  So you need to keep things cooking nicely without burning.  I wasn’t quite sure what to take from this.

Value Objects in Domain Driven Design - Dan Bergh Johnsson

This was an entertaining talk on value objects in DDD, which seem to be the poor relation compared to entities.  Lots of this was just good old fashioned object oriented design, with a domain driven slant. Value Objects differ from Data Transfer Objects since DTOs are a technical construct, whereas VOs are true domain objects.

He showed how pulling Value objects out simplifies the code and helps to avoid bugs, awkwardness and duplication.  It also helps the business logic concentrate on just that, without getting bogged down in other issues.  He worked through some examples, with his credit card exchange rate example particularly compelling.  Having orthogonal classes means that in general m+n tests will be needed where previously m*n would be required.

Good candidates for refactoring include strings or integers with limitations and return values from service methods.  For example, you could encapsulate a quanitity so that it is never negative.  Apparently a friend of his working for “a large Swedish corporation” found that anyone entering negative quantities in their online shop would get sent a credit note!

He moved on to discuss how encapsulating multi-object behaviour with composite value objects can improve matters further.  So you might have DateFrom+DateTo=TimeInterval or Amount+Currency=Money or CurrencyFrom+CurrencyTo+BigDecimal=ExchangeRate. He mentioned Qi4j, which is a framework designed to help do this.

I was lucky enough to be able to go along to QCon London 2009, but not quite lucky enough to go to all three days.  So I’ll summarize my thoughts from the two days I did attend.  Here are my notes from Day 1:

Tony Hoare - Opening Keynote

This was a talk about the differences between Computer Scientists and Software Engineers.  He characterised the former as being idealists who think about generalities and the long term, while the latter are pragmatists and specialists who generally consider the short term.  It was a philosophical talk and since I fall into the latter camp I would have liked something more concrete to take away.  He seemed to be very optimistic about what Computer Science can offer the practicising engineer.  In fact, he came out with the quote:

One day, software will be the most reliable component in any system that contains it.

It wasn’t clear quite how we’ll get to this point, although a colleague of mine quipped that this would only happen because less reliable components would be introduced!

David Pollak - Scala and Lift

This was an interview with the maker of the Lift Framework, which is a web framework based on Scala. It was filmed and should appear in due course on InfoQ (who were co-organisers of the conference).  His pitch for this framework was similar in many ways to that of Rails, in that Java frameworks tend to suffer from too much boilerplate code, because it is not possible to easily pass blocks of code around.  In Ruby and Scala this is possible and so this sort of duplication can be removed.  In fact, Lift started out as a port of Rails to Scala.  At least he didn’t call it Scails!

Scala allows you to use type inference to keep the quantity of code down.  But where you are keen to document what types are needed, you can do this too.  He said that research has shown that the number of lines of code written per day varies very little between languages.  This means we should use more expressive languages where possible.

Lift itself uses a more event-driven style of development to avoid the HTTP request/response cycle.  He described it as being more like Visual Basic where you add a button and then add the code to handle the button being pressed.

Programming in the Small - Ivan Moore and Mike Hill

This was an excellent session.  It turned into something like a 60-way refactoring session.  They gave out samples of code and then asked for ideas on how to improve it.  Their premise, which is similar to that in the book Clean Code, is that you need to start out by cleaning up the really small stuff.  Once that is sorted, bigger issues will become more obvious and easier to solve.

To begin, remove useless comments and unneeded code, fix formatting and rename variables to explain their purpose.  One top tip was to check in reformatting as a revision by itself so as not to hide real changes.  The next step was to remove duplication, simplify conditionals and use tiny policy objects if need be to achieve this.  Finally, look to remove statics and NoJos and  hide primitives inside domain objects where possible.

Clojure - Rich Hickey

Clojure is a dialect of Lisp that runs on the JVM and this talk was given by its creator.  He said that it was the interaction between your code and the compiler that makes a Lisp a Lisp.  He did try to convince us that Lisp does not have a problem with too many parentheses, but it turned out he’d tried to solve this (non-)problem in Clojure, so I’m not sure.  He talked about the use of functional languages and how they simplify programming for our new multi-core world.  Clojure solves this with mutable references to immutable objects.  He gave various reasons why he saw object orientation as a problem and said that polymorphism is great, but should be available based on the runtime state of an object,  because “You are not born a father”.

 Three Years of Real-World Ruby - Martin Fowler

This talk looked at the experience of ThoughtWorks using Ruby in real projects.  Generally it seemed to be a success and even those where it was not, the issues were more sociological than technological.  It also seemed that having some experienced developers who knew about dynamic languages was important.  Once a small team had put the foundations in place it would be possible to expand the team. He suggested that this idea (start small and grow out) would be more widely applicable.

Some of the issues traditionally associated with Ruby were mentioned.  He said that so-called Monkey Patching could be controlled by putting all such changes in one place, or by putting the new code into a module to include into the class. On the performance front, he said that Ruby is indeed slow, but that most of the time your application is going to be database bound anyway.

He finished off by talking about the difficulties of testing code based on Rails’ ActiveRecord. Two big projects went different directions with this. One decided to just go with the flow and let the tests talk to the database, whereas another decided to use dynamic mocking to avoid this. It turned out that the mocking caused the tests to become brittle, so that project is now moving towards using the database instead.

 Continuous Integration from the trenches - Julian Simpson

This talk began with the observation that continuous integration is a human activity.  It is not sufficient to install CruiseControl, TeamCity or whatever on a server.  If developers are not checking in frequently, it is not happening.

There was discussion of how to signal a broken build and how to label builds.  He pointed out that there is a danger using subversion revision numbers as build labels, as these can be fragile when repository maintenance is carried out.  Deployable builds should be labelled, but big binary files like this should not be kept in version control. An ivy repository is an alternative.

He talked about deployment scripts and how these should be kept under version control.  He gave the example of a bash script used to deploy a system which be put under version control and ‘tested’ but simply running it and checking the exit value.  This actually picked up an error introduced into the script by mistake.  It is also a good idea to have the CI server grab its own configuration from version control.

He then moved onto functional tests.  He suggested starting with smoke tests, but warned that building fully automated acceptance tests was harder than you might think.  He then moved on to talk about running static analysis and other metrics against the code and speeding up feedback by parallelizing the builds or by having a build pipeline.  But you need to be careful.

I’ve recently been struggling to check out a large subversion repository on my Mac.  Each time I tried I’d get an error like:

svn: Can't open file '..../.svn/tmp/some_path/some_file.svn-base': No such file or directory

I had no such problems checking out the same repository on a Linux system. This was somewhat puzzling.

Finally, the answer dawned on me. The default HFS file system on Mac OS X is case insensitive, and an examination of the repository with a web browser showed the problem - there were two files that had the same name, but one had mixed case, and the other was lower case. When checking out the repository, both files are dropped into a temporary directory, but one overwrites the other. When SVN tries to move the temporary files, one of them no longer exists!

To prove the theory, I created a case-sensitive .dmg file with Disk Utility, and was able to check out the whole repository into that virtual disk without any errors. I’m considering whether I should convert my whole disk to case-sensitive mode, although some Google search results suggest that some applications don’t like that either!