Anoto

If you’ve never heard of it before, Anoto is a company with a clever piece of technology that enables digital note taking. It starts with a special paper printed with a pattern of tiny (0.1 mm) dots on it. The pattern is based on a grid, but each dot is slightly off centre of from the grid intersections:

They then have an algorithm that can produce squares of this grid, where each square is 2×2 mm and has 36 dots on it, each with a unique pattern. The total surface area they can cover with this unique pattern is 1.8 million square miles.

Then they have a pen, which has a miniature camera built into it that takes 50 or so photographs of the nearby dots every second. From that the pen can tell exactly where you are, where your stroke was going and where it has gone, basically recording everything you write. Most importantly, if you raise the pen and then lower it to the paper, it can tell immediately where it is.

The other clever thing about this pen is the ink. The dots themselves are printed in black and are visible in the infrared (IR) portion of the spectrum, which is how the camera sees them. But in order to ensure that the ink you write does not obscure the dots they have developed IR transparent ink.

That’s the technology, which is easy enough but Anoto then complicate things drastically by their business model and appalling web site. They don’t sell anything direct but work through partners, on whom they provide precious little information. These partners then develop products and, in many cases, buy a section of the virtual Anoto paper to use for their applications.

Most of these partners have developed “business applications” such as pre-printed forms for recording medical questionnaires. All very interesting I’m sure, but no use to me and by definition a niche product.

Moleskine notebooks

This brings me onto to Moleskine notebooks. I love these. They have a way of making me feel I can create anything and anything I do write looks good.

But that’s not enough. I want a digital Moleskine, or something very similar so I get the same feeling of productivity and yet have it all digitally recorded.

OSX and sharing my data

The final thing I want is a way of getting the data off the pen onto my Mac and from there outputting it to my note taking program (SOHO Notes) or a wiki (Confluence) or even to ebook format.

The requirements

It appears that most of the Anoto partners have inherited the same “give your customer only half the information” attitude to web sites as Anoto so finding out exactly how it all fits together was painful to say the least. Even now I still don’t have some crucial answers.

Just to be clear, the three things I am after are:

• An Anoto-enabled pen.
• OSX software to talk to the pen above that outputs in a way I can actually use. It would be nice if it did character recognition as well but I guess that is pushing it.
• An Anoto printed notebook that feels like a Moleskine.

I’ve found the last one quite easily, made by Livescribe. They also seem to make great pens, complete with built-in audio recording and and OLED display. Unfortunately they don’t make OSX software though they are developing some. Even with their Windows software I can’t tell what formats you can output to, they say so little about it. Sigh.

I’ve also come across two separate providers of OSX software, Pen-It and Paperium. The Paperium software claims to output in lots of ways. Both of these come with their own pens to link to their software, which gives me my first major question:

Can any Anoto-enabled pen work with any pen software or is software unique to a specific pen and vice versa?

Pen-It, who resell the Maxell pen make it clear their software only works with that pen and they don’t sell it differently. Given just how little information there is on their web site I am amazed that nugget slipped through.

If the answer is ‘no’ across the board then I’ll have to buy the Paperium pen.

The next question is:

Will any pen work any Anoto paper product?

Again, I would have hoped this answer is yes but Pen-It again say their pen doesn’t.
Until I get that one answered I won’t be able to buy anything!

The products

To save you having to search for hours like I did, here is a roughly useful set of links.

Pens:

• Maxell DP-201. This is also branded as Pen-It (not the same as the people above). They only sell through the channel but there appear to be plenty of people who resell it.
• Nokia SU-27W. They do sell direct and lots of others sell it. This is the latest version of the discontinued Nokia SU-1B.
• Destiny io. This was formerly the Logitech io pen (and io2) but they sold the business. This is possibly the least informative web site I have ever come across.
• Magicomm g303. I’m not sure this is actually an Anoto-enabled pen. They are listed as a partner by Anoto but they don’t mention anything about it.
• Livescribe Pulse. This is easily the pen with the highest features and can also come in a 2GB variant.
• Paperium. I’m pretty sure this pen is just another pen rebadged but I haven’t looked at the pictures closely enough to be sure.
• Flypen”. This is a version especially for kids

Paper:

• Oxford easy book. These notebooks appear everywhere and again are sold through resellers not direct.
• Orignote. These are supplied by Magicomm and again I don’t know if these are Anoto-enabled or not.
• Livescribe notebooks. These are the nice ones that look like Moleskine books.
• Paperium notebooks. Again these are probably rebadged but I can’t tell.

And finally …

Confused? I certainly am.

We had some issues in getting an MS SCOM 2007 (formerly Microsoft Operations Manager) install up and running this week. This product has a fairly heavy set of pre-requisites before it will install and amongst these is asp.net. Try as we could myself and a fellow sysadmin could not get the install to recognise that asp.net (which is included in the .net framework) was installed. Luckily after a little search I stumbled upon MS KB article 934759 (http://support.microsoft.com/?kbid=934759) which states that this problem can occur if you install IIS after the .net framework as asp.net does not then get registered in the IIS metabase, it goes on to recommend that you run the ASPNet_RegIIS.exe command to register asp.net, however this command seems to be 32 bit only and would not run on our 64 bit installation of Windows, the only options were to modify the IIS metabase manually or re-install from scratch, in the interests of application stability we opted for the latter, when we then ensured that IIS was added before the .net framework the SCOM 2007 went through without any issues.

I like cycling and walking. What I also like to do is record where I have been. My Nokia N95 has built-in GPS, but it is pretty poor reception. Besides, the software I use to record where I have been, AFTrack, doesn’t work with the internal GPS. So I now use a GPS receiver.

AFTRack is fine as a mechanism for recording GPS tracks, but I wanted to have a way of viewing them after the event. Of the various formats available for export I have only played with KML and GPX so far. Google Earth is the natural app for viewing KML files, and very good it is too. But there is a great site, Trailguru, which accepts GPX uploads and shows you altitude and speed profiles for the whole track. The intention of the site’s creator, Tim Park, is to build a database of tracks through a wiki interface. This will enable you to plan routes based on existing track segments, and to average over repeated track uploads, reducing the effects of GPS ‘jitter’.

You can see my first upload, a Sunday bike ride with my son, here.

For some time now, I’ve been convinced of the inherent evil of threads. I’ve gone to some lengths to mitigate their use in projects over the last few years. For example, my dnsjnio package uses java.nio to conduct many thousands of concurrent DNS queries in just a couple of threads. This package has been pretty stable for the last couple of years, and I know that it is in reasonably heavy use at several sites (at least).

So, I was very surprised to discover it has a nasty race condition!

I was alerted to this by a very helpful user. I’m still in the discovery phase, so certainly no fix yet, I’m afraid. However, the problem seems to be a race condition between a timer thread (which looks after query timeouts), and the select thread, which handles the I/O. A separate thread is used for timers because so many queries may be outstanding at any one time - and the select thread is busy enough handling I/O to worry about ordering timeouts. If a timeout is generated immediately after data has been received, but not yet promulgated to the user, then a connection can be closed twice, leading a NullPointerException (oops!).

Of course, some sort of lightweight Actors implementation would have been ideal here - in Erlang, I would have an Erlang process maintain the timeout queue, and the select() function ask it for the next timeout before starting the loop. And with a message-passing system, it would be simple to detect that the close had already been processed, as messages are processed sequentially.

Away from the details of this particular issue, I think there’s a broader point to make. It seems that even a relatively simple project, with lots of use, that has been designed to avoid the evils of threading, is still likely to contain some nasty threading bugs.

I will be coding future projects to avoid threading altogether. Well, that’s maybe a bit extreme. But I’d certainly want to design all my threads around blocking queues, with no other means of interaction.

My favoured design for my next project involves a collaboration of many communicating sequential processes - one on each core. Each process may be written in whatever language is best for the functionality of that process - C for speed-critical areas and the JVM or Erlang for most others. Of course, once you design your system like this, it’s then easy to move some components onto the network.

Some might invoke the generic tenth rule, and say that I may as well use Erlang for all of the nodes. I would agree, if there weren’t such a small amount of communication required between the nodes (for this project). And if my need for speed in certain areas wasn’t so acute.

I shall also be looking for other ways to learn from the functional languages - for example, enforcing immutability (perhaps with something like this).

If anyone is particularly interested in this, I’d recommend this paper which I was pointed to recently. I agree with almost everything in it - especially the recommendation to avoid C where at all possible.

Anyway, enough rambling - I’d better get back to fixing my nasty threading bug!

A third person has now asked me about using Dnsruby to write an authoritative nameserver. It seemed worth jotting some notes here.

A nameserver at its most basic simply provides a mapping service between names and addresses. Of course, the RFCs that specify DNS have more than that to say on the subject, even at the start. And, in the 25 years since the first DNS specifications, there’s been a whole load more! For example, an authoritative nameserver might support views, ACLs, different flavours of zone transfers, dynamic updates, etc. And then there’s the cryptographic extensions : TSIG, DNSSEC and so on.

If you want the latest, fully RFC-compliant authoritative nameserver, then your best bet is definitely just to download BIND or NSD. It will run a lot faster than anything you can do with Ruby, and will support most of the currently fashionable DNS features. You won’t even need to debug it! ;0)

However, I don’t think that the people asking about Dnsruby are necessarily after a fully-featured nameserver : “I would like to get some kind of simple ruby dns server running, authoritative-only, for development purposes as it can interact nicely with existing ruby infrastructure and i have no need for high volume”

Of course, you could ask why you’d need to use Dnsruby at all - after all, if only an A record is to be requested, how much DNS functionality is required to respond? I guess there are two reasons :

1) Code reuse. Even if only a few lines of the parser in Dnsruby are used, it still beats re-implementing (and debugging) them.

2) Future extensions. You might not think you’ll want that fancy new feature, but who knows how things will change in the future? The Dnsruby library already provides support for may DNS features (most notably those related to security).

One issue is that I wrote Dnsruby as a client-side library. So, although it supports zone transfers, it only supports initiating and receiving them - not serving them. Currently, support is only there for verifying DNSSEC signatures (although very little code would be needed in order to support signing packets). Similarly, there is currently only support for sending queries; there is no listening server. And there is certainly no data store (or cache).

Anyway, the question was : “If I could have dnsruby sit on a machine (using EM - awesome!) and receive dns requests, and I could put some hook code in there somewhere to look it up in a DB or YAML file or whatever, it sounds like I could do that, but I just can’t find where to begin.”

I think most of the answer is in the question! The main tasks would be to write a server to listen for incoming queries, and some kind of storage to hold the data you want to serve (be that in memory or on disk).

I’d use EventMachine to sit on (presumably) port 53 listening for incoming UDP queries. You should really also listen for TCP queries (especially if you want to support zone transfer). When you receive something, call Dnsruby to decode it, and look up the response in your data store.

As for the storage, you’d need to decide if it should support updates, or whether it can be read-only. You could use the Dnsruby::RRSet class to hold resource records which have the same label, class and type. When your server receives a DNS request, do something like :

def NameServer.createResponseToIncoming(packet)
    message = Dnsruby::Message.decode(packet)
    question = message.question()[0]

    # If the question is a Dnsruby::Update or an AXFR/IXFR transfer request then deal with it separately
    # if (message.header.opcode != Dnsruby::OpCode.Query) ....

    # Find the records which were requested
    rrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, question.qtype)
    rrset.each {|rr| message.add_answer(rr)}

    # And get the NS records for the authority section
    nsrrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, Dnsruby::Types.NS)
    rrset.each {|rr| message.add_authority(rr)}

    # We also want the NS A records for the additional section
    nsrrset.each do |ns|
         rr = DataStore.findArecord ForNS(ns)
         message.add_additional(rr)
    end

    # Make sure that we are authoritative for the response! Otherwise, return REFUSED
    message.header.rcode = Dnsruby::RCode.NoError # Or Refused

    # Now encode the packet and return it for the server to respond with
    response = Dnsruby::Message.encode(message)
    return response
 end

[As you can see, Dnsruby is not doing a great deal for you here. Indeed, it would be possible to do away with most of the requirement for understanding DNS : you could store the data in wire format, thus removing the need to decode/encode the message. However, you’d still need to construct the encoded data store, and you’d have problems if you decided to secure your zone with DNSSEC with this approach]

If anybody does end up implementing a namserver with Dnsruby, I’d be happy to consider adding the code to the Dnsruby library! ;0)

Over the past few months I have been working with Iris Explorer to develop visualisation applications. Iris Explorer uses a module based approach to application development, where tasks are split into individual modules which can be then wired together. The modules represent processes on the data and the wires between them represent the flow of the data between the modules. A simple drag and drop interface is provided for adding and linking the components of an application, or map, together.

Iris Explorer is distributed with a set of standard modules for accomplishing a wide variety of standard tasks such as, reading data, writing images, and standard visualisation techniques, such as plotting histograms. An example of an Iris Explorer map is shown below:

I have been working on creating a 3D graph to represent the volume of queries to our WHOIS system for each hour of a given date, or range of dates. This could largely be accomplished using the standard module set avaialble with Iris Explorer, however there were no modules for the labelling of the axes of graphs which were particularly suited to the task, which meant I had to write my own custom module.

Iris Explorer provides the ability to write your own modules, so it is entirely possible to add custom processes and functionality to an application. The method for developing these custom modules is a two step process, first of all the the layout of the module is defined using the Iris Module Builder, secondly a user function, written in C/C++, is added.

The Module Builder provides a GUI to creating the module window, its data input and output ports and specifying the flow of data through the module internally. This is a logical process where you simply specify the type of input data which is accepted as well as the format of the output data. The appearance of the module itself and the user interface of the module are also specified using a drag and drop WYSIWYG style editor.

The second stage, the user function, controls the processing of the data. So in the case of the Graph Labelling module I have written this would extract the number of dimensions in the data set and number bars of the graph accordingly, as well as creating and positioning axis text labels.

The Iris Explorer Map produced collects data for a given date range, and split this into hourly segments to give a clear picture of the volume of queries to our systems at any given hour of the day. An example of the results of this are shown below:

Need a quick way to decide who does something:

http://www.youdrawstraws.com/index.cfm

There are two modes:

1. An instant on-screen version, useful if everyone is in the same room.
2. A group session for use when working remotely. These sessions are time-limited.

We used the on-screen version to decide who would go to a remote site to install servers next week. It saved me the job of picking on someone! All it required was to give the session a name, enter how many people there are to choose from, name them, then let the application choose at random.

The group session is a bit more involved. Those involved in the draw can add categories, and options within categories. Once everyone has given input, or the deadline for choosing has been reached, the options are selected at random.

I have a SE P990i - not a bad device if over the last 3 years you’ve been able to stomach the infinite software updates/bug fixes/patches and workarounds - mine’s actually now usable.

Running the bundled software ‘PC Suite’ is required for backups (amongst other things) - and backups are essential. But having changed to a x64 system (XP) it turns out that Sony Ericcson do not support x64 systems (except within Vista Ultimate x64 - but not confirmed) - ie. PC Suite won’t install - and that’s SE’s offical line and last word on the matter. They’re not replying to customer service emails on the matter, either. Not even an automated reply - just silence (I think this will be my last ‘phone from SE).

Some get round this by dual booting 64 and 32-bit Windows and running the software when necessary under 32-bit - but I’m not too confident in this area and wanted a more ‘familiar’ solution. There are other solutions (while staying within x64) for SE’s slackness kicking around the web - however all of them involve only partial sucess - ie. you can have the ‘phone recognised by the system as a mass storage device, have the drivers working so the USB cable can charge it, even update the OS via SE’s update service - but you still can’t get the PC Suite to synchronise and perform backups. After numerous attempts and tweaking I have however managed to get them all working correctly. This obviously was the result of sheer hard work and luck was not involved in any way. Cough. I can’t guarantee it will work for other SE models, though my P990 was very happy.

Needless to say, backup your ‘phone before messing with it - although this concerns the software on your PC, not your ‘phone, so you should be just fine.

First, here’s a summary of the current solutions and drawbacks, with links:

• http://webmotion87.googlepages.com/sonyericssonp990(i)undermicrosoftwindows Summary: Extract the .exe and 64-bit .msi file - modify and re-run. This will give access to the SE Update Service but not the PC Suite.
• http://www.planetamd64.com/lofiversion/index.php?t13052-50.html Summary: Extract the .exe and retrieve the .msi file. Mess about with the installation .msi and then re-run it. This solution nearly got me up and running - but failed - I got the same old “This operating system is not adequate..” error in the final stages.
• http://www.wilson-it.com/sony_ericsson_download.htm. Summary: Some clever people at Wilson-it rewrote the SE drivers; quote “Basically the DSS-25 has a USB to Serial converter chip in it made by FTDI Ltd that was slightly modified for SE. We modified a BETA x64 driver of that chip to work with the SE DSS-25. Why SE couldn’t do this I have no idea.”. This solution will, on it’s own, allow you to recognise the device (the ‘phone, not just the sync-station as claimed) on your system via USB so you can charge it. Allegedly the update service will work as well (needing just a USB connection) but I haven’t tested this bit (you’ll know why, if you own a P990 ;-). This is not a fix for the installation of PC Suite, mind -however some people have reported success.
• Do both. Install the drivers and then re-run the modified .msi. Summary: You can get a bit further but sadly, for me, it was the same old installation error in the end.
• Bluetooth only: Apparrently, “floAt’s Mobile Agent 2.1 beta 3 ( http://fma.sourceforge.net/ ) works great with Win x64 and my Z800i. Synchonizing and file transfers, no problem”. Summary: Not tested this as I’d like to stay with the OEM software, so cannot comment. And it’s Bluetooth only, as stated.

Tweaked version:

1. Download the Wilson-it drivers (http://www.wilson-it.com/sony_ericsson_download.htm) and install them (all of them - when one is finished, it will prompt another). This will enable USB charging.
2. Get the latest PC Suite (67mb) from SE’s site (http://www.sonyericsson.com/cws/support/softwaredownloads/p990i?cc=gb&lc=en) which is “PC Suite for Smartphones 1.5.8″ or use an existing (older) version - I used my older version that came with the original CD.
3. Download an .msi extraction tool such as Orca (I used this from http://www.technipages.com/download-orca-msi-editor.html) or other program (strangely the most ‘popular’ seems to be something called “Less MSIérables” - but their site is offline and still is so I have no idea what it’s like)
4. Use a program such as WinRar (http://www.rarlab.com/download.htm) to extract the PC Suite for Smartphones .exe package. Extract to a new folder where you can get at the resulting files.
5. From the resulting WinRar extracted files, run the “PCSuite.exe” (not the PCSuitex64.exe some solutions say “do” but that didn’t work for me) until you hit a wall (error report or such) - do not cancel or abort the installation just yet. Search your C:\WINDOWS\temp folder for the “still running but about to disappear when you abort the installation” .msi file - this will be called “PC Suite for Sony Ericsson.msi” or possibly “ah7v5gdo6.msi” or similar ‘random’ string. Tip: if you search for this before you begin and leave the window open, the window will update when you run the .exe showing you the right file.
6. Right click on this to edit within Orca (or drag and drop into Orca). Whilst open in Orca, locate the entry “LaunchCondition”, right-click and and “drop table”. So far, this is the standard solution which didn’t work (yet) for me.
7. My tweak - an additional error for me was “The current version of msxml6.0 is not compatible with…”. So, whilst still in Orca, locate (using the find tool) “msxml6″ in all entries (I found two) and choose “drop row”, thus deleting that call/check. I suspect the critical one is in the InstallExecuteSequence table.
8. Save the resulting file (note that’s Save and not Save as) - this will overwrite the .msi in memory. Still without having aborted the initial installation, double click and run this .msi file - this will bring up a new install window. Keep going and it should install the PC Suite. If it complains about drivers, decline Windows’ offer to search the web and locate the extracted Wilson-it drivers in the same directory as last time.
9. If you have no luck at this point with the installation of the PC Suite try the x64 version of the .exe and the subsequent .msi. I know I ran both or all variations before finally running the “PCSuite.exe” and resulting .msi which worked.
10. At this point, although PC Suite runs (hopefully) it wil still not recognise the ‘phone. I tried restarting the PC, as well as unplugging the ‘phone repeatedly. I swear this did not work - until around 48 hours later when I plugged my ‘phone in to charge - when suddenly Windows detected ‘new hardware’ and asked for “drivers”. Pointing Windows to the same extracted Wilson-it drivers as before (surely they’ve already been installed?) allowed PC Suite to recognise and connect to the ‘phone. Maybe it’s just a question of a few restarts and re-re-plugging in of the ‘phone. Not very technical this very last bit, but at least it worked.

In hindsight, I suspect that if you install the drivers last - after the “.msi PC Suite” part - you may have a quicker result than I did.

Useful links:

http://www.planetamd64.com/lofiversion/index.php?t13052-50.html
http://www.esato.com/board/viewtopic.php?topic=170665

I have been introducing Coverity Prevent for Java into our continuous integration stream,  so I decided to post some comments about it.

Installing the software is not difficult at all, just uncompress the files and get a license. Starting to analyze code is also quite easy following the Quick Start Guide, once you get it working you can progressively tune the analysis for your needs . The program supports a great variety of platforms and operative systems so it is very likely that it will work in any kind of machine you happen to have your code and binaries on. It also does a great job detecting failures, quite clever stuff.

However when it comes to integrate Coverity Prevent with your build system it is not as straight forward. You code in Java, don’t you? so it is very likely that you use ant to build the system, but there is no support for ant to run Coverity Prevent. The program runs as a number of native applications or shell scripts called with several parameters, therefore I had to write a few shell scripts to run the different programs with the correct parameters and then use the exec ant task to call those scripts. I would have been quite happy if I could just drop some libs into ant directory and run an hypothetical prevent_analyze task from the ant build script.

Coverity Prevent stores found defects into a proprietary database and you can manage and query this database via web using an embedded web server or via command line. The embedded web server is another annoying thing, as we are already running an application server in the machine for the continuous integration testing, I think it would be good that the web interface was also provided as a java web application deployable into the web application server which you are likely to be using and managing.

Apart from that, the access via web looks nice,  you can see the defect description and explanation. It allows you to classify the error as intentional or a real bug. You can also assign a severity to a given defect, and specify an action to mark the defects to be resolved, expected to be fixed or ignored. When finding defects, it allows you to visualize defects by product, by component into a given product, sort them by any field, etc.

When accessing via the command line you can perform all the same operations and even more, allowing a great degree of automation and produce reports on the information stored in the defect database. This functionality allowed us to extract some statistics on the number of defects and defect listings to be presented in our build results main page. Using the web access was less useful as user need to log into the web application to be able to see results and we wanted some output to embed into the reporting system and have the current state at a glance.

The need to log in and the fact that the graphs produced in the web view are done with interactive flash prevented us to have a nice main page showing graphs along with the statistics. It would have been nice to have the command line query to produce PNG images with those graphs instead of interactive flash applications. It could be done with the query command line and some plotting package such as ploticus or gnuplot but it would require too much time to be invested on.

In conclusion, I have to say that it is a good analysis tool which detects many pitfalls in the code and it really helps, but it is yet another tool to add to the defect management pool of tools as we are already using Jira for that purpose, asking a plugin to integrate them would probably be too much ;)

Wireshark is a free, open source network protocol analyzer. It can be used as a graphical interface when viewing tcpdump files. Alternatively, it can be used to capture live network traffic. For this, it must be able to read raw data from network interfaces. Using the standard install from DMG package under Mac OS X this is not straightforward.

By default the /dev/bpf* interfaces are owned by root with permissions set to rw-------. They can be opened for reading using sudo chmod go+r /dev/bpf*, but this is not persistent across reboots. To permanently ensure that Wireshark can be used in capture mode it is necessary to run this command on start up. The mechanism to do this is explained here. I made a slight modification to the ChmodBPF script to run the command above. The gzipped file is attached. Simply unzip this and put it in /Library/StartupItems.