I wouldn’t normally cross-post like this but you may not have seen that we have launched a new blog for the management team here at Nominet to post articles. I’ve just posted one on our position on signing the root, which is a summary of our recently published position paper.

For some reason, I decided to upgrade my laptop to Leopard yesterday. Not sure why, as there isn’t anything that I simply must have. Everything went well after the upgrade, with only one thing broken: The login window. As soon as it received focus I got the dread spinning beachball of death. I found that there were some other folks getting the same issue. As mentioned there you need to do an Archive and Install to fix the problem. So I tried that with the ‘Preserve User and Network Settings”. Same thing. Finally I tried it without preserving any settings and I was in. So then I had to copy across all of my files and preferences from the archived copy of my system. Phew, not bad for a day’s work.

I think the problem is that I am a tweaker. I can’t help installing 3rd party apps to change this and that. But that’s because I’m a techie and we jealously guard our freedom to tinker. It looks like if I want a trouble free life I should learn to live with the defaults.

Chatting to a colleague this morning, and it looks like Leopard’s Time Machine just won’t work with Filevault when he tried on his laptop. As Apple state on their Time Machine page in their marketing blurb: “Time Machine: a giant leap backward” …when working with Filevault.

Time Machine monitors your disk drive by checking for changed files on the hour and backing these up incrementally. Filevault works by encrypting and storing the entire contents of your Home folder into a safely encrypted disk image, then reading and writing to that, encrypting and decrypting on the fly.

Because of this, your home directory is essentially a single file as seen by Time Machine, so every time you try to make a change to your Filevault protected home directory, Time Machine tries to backup this whole disk image.

Now as a business user, I can see why Filevault would be used to protect sensitive business data on a laptop in a business environment, but really businesses should have a more robust backup solution should be in place already, rather than depending on a consumer grade solution, and businesses should not really depend on Time Machine as their sole reliable backup solution. Time Machine won’t work reliably across a network (unless to another Mac) anyway, which is what a lot of businesses will be doing backup-wise.

However as a home user, on my machine at home, I can see the benefits of Time Machine, and really running Filevault on my home directory would be pointless, as the amount of RAW image processing I do would seriously be hampered by encrypting/decrypting on the fly, and I have absolutely no need to encrypt my MP3s! At home, most of the document processing I do now is web based anyway, and short of a few applications and music/photos, I have precious little on my home hard drive that really needs encryption, but would benefit from something like Time Machine for occasional file recovery/chance of component failure. At work, I use Filevault on my laptop, and our source code repository for storing code and Lotus notes for storing project related info, so have no need for Time Machine, but Filevault on the other hand is very useful.

Now obviously my particular computer usage will work well with this situation, but for those who store more sensitive documents and want encryption and to use time machine, another solution might well be needed.

The only workaround I can think of is to use the Disk Utility to create an encrypted AES-128 disk image. This is the same technology Apple uses for Filevault. Them while using this, mount it and write files to it, and close it when done. Time Machine will back this up as usual, but as it is storing just the files you want encrypted, it should be a lot snappier, due to much smaller file size. It’s not an ideal situation, but if someone had to use both encryption and Time Machine it might help.

Demon’s own Gary Feldman wrote some alternative lyrics to “American Pie” in the hotel bar, and performed the next day during Ripe-55’s closing planery. If you haven’t attended the ripe meeting, it contains the yeast of several presentations about the need to migrate to IPv6.

Enjoy.

I’ve been working on something in Wordpress with my colleague Ewan, and we ran into a very strange problem with Firefox yesterday. The theme which Ewan was designing for it looked fine to him in Firefox, but to myself and a colleague had a few issues visually. The header image wasn’t in the correct place, and a couple of elements disappeared that should have been there. The curious thing was that we were all running Firefox 2.0.0.6 and above.

After trying various things, we realised that Ewan’s version was upgraded from 1.5 to 2.0, whereas the rest of us were using clean installs of 2.0. Ewan cleaned Firefox off his machine, installed the latest version, upon which he could see things the same as us.

So what caused this? Were some elements of Firefox referring to an older version of Gecko (Mozilla’s rendering engine)? What did trouble us was the idea that other people may be designing websites for Firefox, and just not seeing what their users see.

I’ve been installing Oracle 10g on Solaris 10 x86-64. I was keen to avoid doing the old fashioned method of editing /etc/system but instead set the various shared memory segments dynamically using resource management and projects. The official Oracle installation guide for Solaris 10 x86-64 mentions how to do this, but it actually contains incorrect and incomplete information. Ignore the Oracle documentation and set your shared memory kernel parameters by doing the following:

sudo projadd -U oracle -G oinstall user.oracle
sudo projmod -sK "project.max-shm-memory=(priv,14294967295,deny)" user.oracle
sudo projmod -sK "process.max-sem-ids=(priv,1024,deny)" user.oracle
sudo projmod -sK "process.max-sem-nsems=(priv,1024,deny)" user.oracle
sudo projmod -sK "process.max-shm-ids=(priv,1024,deny)" user.oracle 

You can look at /etc/project to ensure you have correctly entered the parameters. I would recommend looking at id -p as the oracle user to make sure that the user you are running oracle as, is in the correct project:

id -p
uid=2000(oracle) gid=2000(oinstall) projid=100(user.oracle)

You can see the oracle user is using the user.oracle project, as opposed the default project.

At Nominet we use DataCash as our payment service provider. The idea is that they give you the relevant libraries to allow you to process various types of payments without needing to get tied up in all of the implementation details. We use them to process credit/debit cards and bank direct debits. This has generally worked pretty well for us.

We provide online services to our customers, we don’t ship physical products. So if we get a chargeback, we can simply stop the service. It’s a different matter if you’ve sent hundreds of pounds of electronics to a crook. This means that we’ve never signed up for the bells and whistles that are available to verify transactions because that would just force our customers to jump through a bunch of extra hoops.

Recently though, the big two credit card companies have been pushing their online security system, 3-D Secure. They are rolling this out over their different card types, making it mandatory to use online. As an example, since 1 July 2007 it has been mandatory for Maestro cards. The only get out clause is if you are in the process of implementing it.

So this has meant that we’ve had to do some development work to change all of our online credit/debit card processing to add this extra step. It isn’t a simple one either, as it involves redirecting the cardholder to their bank’s website, where they enter a password or somesuch, which then POSTs a bunch more data back to your site. It makes what was once a one-step process into a multi-step one.

So, we did the development work necessary and put it on our test systems. We used DataCash’s test server and tested out all of the various error messages it could send back. The test server is only set up to send a subset of the long list of return codes, but we were confident we could handle the common ones and display an error message for the obscure ones. There are in effect three classes of return code:

1. ok so far, redirect to this URL: …
2. ok so far, but this card is not setup for 3-D Secure, so continue without.
3. oh dear, something went wrong.

We were pretty sure we knew which codes were which so that our payment logic would work. The day of deployment came. We put the system live. Suddenly we were hit by a bunch of new codes that we weren’t expecting. Lots of people couldn’t make payments. The result is that we look like a bunch of amateurs who haven’t tested our systems properly. After a couple of phonecalls to DataCash I got hold of someone who knew what they were talking about. It turns out that there are 5 codes in the second class (”ok so far, but this card is not setup for 3-D Secure, so continue without”), rather than the 2 we had been expecting. So we were telling the user that an error had occurred when everything was fine.

Unfortunately, the piece of information we were missing is not mentioned anywhere in DataCash’s current documentation and the error code we were seeing is not sent back by the test server. So unless you are psychic, you are only going to find out the truth once you’ve gone live and you look like an incompetent idiot. I was told that these error codes are not DataCash’s codes, but are part of the 3-D Secure specification, so we should have known about this by reading about 3-D Secure elsewhere. Frankly I don’t believe this for the following reasons:

1. I’ve searched the web for a list of standard 3-D secure return codes. I can’t find one anywhere. The only places where I find something that looks promising, I am asked to sign up to a credit card company’s licence agreement.
2. It seems a little coincidental that in their list of return codes, the 3-D secure ones lie neatly in the range 150-189 with only a handful of numbers separating them from codes for completely different services above and below.

So if you are in the process of implementing 3-D secure with DataCash, here’s the secret that will spare you a lot of grief. The following codes all mean that the card is not enrolled in 3-D Secure and you should proceed without a redirection:

158, 162, 163, 173 or 183

Good luck!

I’ve installed NSD on Apple’s OSX. Since rc functionality will be deprecated in the near future, I decided to have a go with launchd. Launchd offers a single method to any programs started automatically by the system. You can do a lot more, like specifying resource limits and environment variables, but I’ll leave that out of this post.

Opposed to writing scripts and reference them through rc.local, I have to write an xml plist file. This file should be put in the /Library/LaunchDaemons directory. There is also a /System/Library/LaunchDaemons directory, though that is reserved for system provided daemons, and hence may have changed after an update. Have a look though at some of the plist files in those directories to understand how things are done.

Let’s construct a plist file for NSD. Since the plist files are in XML, we’ll start of by referring to a document type definition (dtd). We need to include the property list version (1.0) as well.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>

Now we’re ready to fill out the property list keys. We’ll use “nl.nlnetlabs.nsd” as a label. Though this is required, it could really be anything, as long as its unique. Since the labels of other daemons seem to look like a name space definition, let’s mimic that.

        <key>Label</key>
        <string>nl.nlnetlabs.nsd</string>

One thing to note is that Launchd takes care of forking and daemonizing, so we’ll need to configure NSD in such a way that itself does not daemonize. (We can’t get it to stop forking). This can be done by using the “-d” flag, which causes NSD to run in debug mode, and not daemonize. Take care of the path if your NSD resides elsewhere than /usr/local/sbin . You could also configure that in your nsd.conf.

        <key>ProgramArguments</key>
        <array>
                <string>/usr/local/sbin/nsd</string>
                <string>-d</string>
        </array>

We need to instruct Launchd that NSD is run at load time.

        <key>RunAtLoad</key>
        <true/>

We also need to instruct Launchd that NSD must be kept running continuously, i.e. restarted immediately after a crash.

        <key>OnDemand</key>
        <false/>

Finally, we need to add the closing tags:

</dict>
</plist>

The entire file should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>nl.nlnetlabs.nsd</string>
        <key>ProgramArguments</key>
        <array>
                <string>/usr/local/sbin/nsd</string>
                <string>-d</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>OnDemand</key>
        <false/>
</dict>
</plist>

After you’ve copied the above in the file /Library/LaunchDaemons/nl.nlnetlabs.nsd.plist you can now start the daemon. You can do it by using launchctl, as follows:

$ sudo launchctl load -w /Library/LaunchDaemon/nl.nlnetlabs.nsd.plist

There is a very handy GUI editor for these kind of launchd plist files by Peter Borg called Lingon. This allows you fine grained configuration over your launchd plist files, and acts as an interface to control launchd.

I have been doing a little bit of playing around with our new Sun X4500 box. I’ve already discussed elsewhere how compelling the price/GB of this box is. I have now had the chance to get some out-the-box performance numbers for running ZFS on the X4500.

First off, I created a zfs pool using a mirror-stripe combination:

zpool create -f testpool  mirror c0t0d0 c1t0d0 mirror c4t0d0 c6t0d0 
         mirror c0t1d0 c1t1d0 mirror c4t1d0 c5t1d0 mirror c6t1d0 c7t1d0 mirror c0t2d0 c1t2d0 
         mirror c4t2d0 c5t2d0 mirror c6t2d0 c7t2d0 mirror c0t3d0 c1t3d0 mirror c4t3d0 c5t3d0 
         mirror c6t3d0 c7t3d0 mirror c0t4d0 c1t4d0 mirror c4t4d0 c6t4d0 mirror c0t5d0 c1t5d0 
         mirror c4t5d0 c5t5d0 mirror c6t5d0 c7t5d0 mirror c0t6d0 c1t6d0 mirror c4t6d0 c5t6d0 
         mirror c6t6d0 c7t6d0 mirror c0t7d0 c1t7d0 mirror c4t7d0 c5t7d0 mirror c6t7d0 c7t7d0 
         mirror c7t0d0 c7t4d0

I then created an 8GB test file with the following:

time dd if=/dev/zero of=/testpool/test.dbf bs=8k count=1048576
1048576+0 records in
1048576+0 records out

real    0m15.330s
user    0m0.375s
sys     0m14.941s

This gives a sustained data write transfer of 523MB/s. I also looked at read speed:

time dd if=/testpool/test.dbf of=/dev/null bs=8k
1048576+0 records in
1048576+0 records out

real    0m7.007s
user    0m0.313s
sys     0m6.694s

This gives a sustained read rate of 1145MB/s.
As a simple comparison I created a RAID-Z pool as well:

zpool create -f  testpool  
raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 
raidz c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 
raidz c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 
raidz c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 
raidz c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 
raidz c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 
raidz c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 
raidz c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 
raidz c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0

I also tested read and write preformance on this pool:

time dd if=/dev/zero of=/testpool/test.dbf bs=8k count=1048576
1048576+0 records in
1048576+0 records out

real    0m15.107s
user    0m0.381s
sys     0m14.637s

This gives a sustained data write rate of 531MB/s, very similar to the RAID10 performance. The read performance was as follows:

time dd if=/testpool/test.dbf of=/dev/null bs=8k
1048576+0 records in
1048576+0 records out

real    0m6.715s
user    0m0.311s
sys     0m6.404s

Again giving a data transfer rate of 1194 a pretty similiar rate as that achieved with RAID10.

No one is saying these tests in any way model a real world situation, however I would argue they are pretty indicative of maxium possible sustained data transfer rate. It’s interesting to me that RAID-Z and RAID10 performed pretty much identically, not quite what i would have expected, perhaps the write penalty associated with parity calculations would be more apparent with multiple random I/O’s.

The other really interesting thing is the comparison of maxium transfer rate with Fibre Channel. We use a lot of fibre here at nominet for connecting databases to storage, the theoretical maximum transfer rate of 2Gb/s fibre is only around 250MB/s, so even a pair of fibres ain’t touching the X4500. You’d really need to go to dual connected 4Gb/s fibre to start competing on a transfer rate basis. Of course as I said at the start, the X4500 will still win in the price/performance department hands down.

Dnsruby can use either its inbuilt (pure Ruby) event loop, or
EventMachine (a native extension to Ruby which must be installed
on the local platform).

Configuring Dnsruby to use EventMachine

I left a couple of switches in Dnsruby::Resolver :

Dnsruby::Resolver.use_eventmachine(on=true)

Dnsruby::Resolver.start_eventmachine_loop(on=true)

The first of these tells Dnsruby to use EventMachine, rather
than its own event loop.

The second tells Dnsruby whether to start the EventMachine loop
or not.

If standard Dnsruby client code is used, then Dnsruby needs to
call EventMachine::run{} in order to start the EventMachine loop.
However, if more than one EventMachine loop is started in a Ruby
process, then the process terminates.

So, if client code is written in an EventMachine style, contained
in an EventMachine::run{} call, then it will need to tell Dnsruby
NOT to start the EventMachine loop (on pain of sudden death!).

Example code

Here is an example of using the code in an EventMachine style :

require 'Dnsruby'
require 'eventmachine'
res = Dnsruby::Resolver.new
Dnsruby::Resolver.use_eventmachine
Dnsruby::Resolver.start_eventmachine_loop(false)
EventMachine::run {
  df = res.send_async(Dnsruby::Message.new("example.com"))
  df.callback {|msg|
     puts "Response : #{msg}"
     EM.stop}
  df.errback {|msg, err|
     puts "Response : #{msg}"
     puts "Error: #{err}"
     EM.stop}
}

And an example in a normal Dnsruby style :

require 'Dnsruby'
res = Dnsruby::Resolver.new
Dnsruby::Resolver.use_eventmachine
Dnsruby::Resolver.start_eventmachine_loop(true) # default
q = Queue.new
id = res.send_async(Dnsruby::Message.new("example.com"),q)
id, response, error = q.pop