techblog 

Last week I attended an ITIL v3 Foundation Training course. Non technical training courses usually leave techies a little bored and uninspired but for me this definitely was not the case. All the stories I have heard from colleagues in the past is that ITIL is a bad thing, has huge overhead and costs huge amounts of money to implement. From what I was taught last week I think this is a reflection on the way ITIL has been implemented and not a criticism of ITIL itself. One of the cornerstones of what we were taught last week is to pick up the pieces from ITIL that are relevant to your business and change/apply them to your needs, this seems like a very good philosophy to me.

Anyone who has done much looking into these kind of frameworks will tell you the cost is high mainly due to the expansion in human resources which can be huge and it is certainly true that looking through the course notes (and if you are really sleepless the ITIL books themselves) that it seems to be a job creation scheme, there are new roles all over the place, however they are just that ‘roles’ many of these tasks can and should be taken on by somebody who has another job and can dedicate a little of their time to this ITIL role to ensure that things are running smoothly from an ITIL perspective. Of course this depends on the size of the organization in question, but in the case of Nominet I would imagine that one person could indeed take on several of these roles, the important thing is that the roles and responsibilities are clearly defined.

ITIL version 3 has expanded from the previous version to now include Service Strategy and Service Design and as these were new topics there was quite a lot of focus on them in the three day course. These subjects however I believe are of more interest to a management and/or software development audience, my interest mainly was on the Service Operation angle of the course as it is in that area that I have spent the last 13 years of my working life. I was very interested in ITIL’s take on Incident, Problem and Change Management, these are all critical areas within the operation of any critical service (IT or Non IT) and doing these efficiently in a repeatable manner as ITIL suggests is very important for any service based organisation, to be fair I was already aware of this but ITIL does enable you to easily spot any weakness in the way you operate and most importantly use the experiences of other people.

A third person has now asked me about using Dnsruby to write an authoritative nameserver. It seemed worth jotting some notes here.

A nameserver at its most basic simply provides a mapping service between names and addresses. Of course, the RFCs that specify DNS have more than that to say on the subject, even at the start. And, in the 25 years since the first DNS specifications, there’s been a whole load more! For example, an authoritative nameserver might support views, ACLs, different flavours of zone transfers, dynamic updates, etc. And then there’s the cryptographic extensions : TSIG, DNSSEC and so on.

If you want the latest, fully RFC-compliant authoritative nameserver, then your best bet is definitely just to download BIND or NSD. It will run a lot faster than anything you can do with Ruby, and will support most of the currently fashionable DNS features. You won’t even need to debug it! ;0)

However, I don’t think that the people asking about Dnsruby are necessarily after a fully-featured nameserver : “I would like to get some kind of simple ruby dns server running, authoritative-only, for development purposes as it can interact nicely with existing ruby infrastructure and i have no need for high volume”

Of course, you could ask why you’d need to use Dnsruby at all - after all, if only an A record is to be requested, how much DNS functionality is required to respond? I guess there are two reasons :

1) Code reuse. Even if only a few lines of the parser in Dnsruby are used, it still beats re-implementing (and debugging) them.

2) Future extensions. You might not think you’ll want that fancy new feature, but who knows how things will change in the future? The Dnsruby library already provides support for may DNS features (most notably those related to security).

One issue is that I wrote Dnsruby as a client-side library. So, although it supports zone transfers, it only supports initiating and receiving them - not serving them. Currently, support is only there for verifying DNSSEC signatures (although very little code would be needed in order to support signing packets). Similarly, there is currently only support for sending queries; there is no listening server. And there is certainly no data store (or cache).

Anyway, the question was : “If I could have dnsruby sit on a machine (using EM - awesome!) and receive dns requests, and I could put some hook code in there somewhere to look it up in a DB or YAML file or whatever, it sounds like I could do that, but I just can’t find where to begin.”

I think most of the answer is in the question! The main tasks would be to write a server to listen for incoming queries, and some kind of storage to hold the data you want to serve (be that in memory or on disk).

I’d use EventMachine to sit on (presumably) port 53 listening for incoming UDP queries. You should really also listen for TCP queries (especially if you want to support zone transfer). When you receive something, call Dnsruby to decode it, and look up the response in your data store.

As for the storage, you’d need to decide if it should support updates, or whether it can be read-only. You could use the Dnsruby::RRSet class to hold resource records which have the same label, class and type. When your server receives a DNS request, do something like :

def NameServer.createResponseToIncoming(packet)
    message = Dnsruby::Message.decode(packet)
    question = message.question()[0]

    # If the question is a Dnsruby::Update or an AXFR/IXFR transfer request then deal with it separately
    # if (message.header.opcode != Dnsruby::OpCode.Query) ....

    # Find the records which were requested
    rrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, question.qtype)
    rrset.each {|rr| message.add_answer(rr)}

    # And get the NS records for the authority section
    nsrrset = DataStore.findRrsetsWithNameClassType(question.qname, question.qclass, Dnsruby::Types.NS)
    rrset.each {|rr| message.add_authority(rr)}

    # We also want the NS A records for the additional section
    nsrrset.each do |ns|
         rr = DataStore.findArecord ForNS(ns)
         message.add_additional(rr)
    end

    # Make sure that we are authoritative for the response! Otherwise, return REFUSED
    message.header.rcode = Dnsruby::RCode.NoError # Or Refused

    # Now encode the packet and return it for the server to respond with
    response = Dnsruby::Message.encode(message)
    return response
 end

[As you can see, Dnsruby is not doing a great deal for you here. Indeed, it would be possible to do away with most of the requirement for understanding DNS : you could store the data in wire format, thus removing the need to decode/encode the message. However, you’d still need to construct the encoded data store, and you’d have problems if you decided to secure your zone with DNSSEC with this approach]

If anybody does end up implementing a namserver with Dnsruby, I’d be happy to consider adding the code to the Dnsruby library! ;0)

Hopefully by now many of you will have read of the new and serious vulnerability in DNS servers that allows them to be spoofed easily and bypassing detection systems.  We at Nominet are privy to the details of this vulnerability and can assure you that it is every bit as serious a threat as being portrayed in the various advisories being released by CERTs around the world.

Whilst it is based around a known vulnerability in DNS (the 16 bit ID field) it enables spoofing of a caching resolver with a very small number of packets, far less than might trigger any normal detection system.  It can also be triggered remotely by various techniques, so an attacker does not need query access to your resolver to exploit the vulnerability.

The full details of the vulnerability will be released at BlackHat on August 6th, which gives a four week window for the upgrade of _all_ caching resolvers.   This is something that we strongly urge you to do and a process that we have already begun.  To be clear, this only applies to caching resolvers, not authoritative servers.

The main CERT vulnerability note is here (with links in it for every product)

If you use BIND then details of the patched versions are here.

If you use Microsoft DNS server then details are here.

If you wish to use a caching resolver that is built from the ground up for security, and already implements the mitigation technique, then please consider Unbound.  If you use djbdns then that too already implements the interim mitigation technique of source port randomisation.

It should be clear to all those who work with DNS on a daily basis that the only true mechanism to prevent DNS spoofing techniques is DNSSEC and we all need to begin taking that seriously.

So please use this short window to upgrade your caching resolvers and iron out any issues that might arise before exploits of this vulberability are seen in the wild

Over the past few months I have been working with Iris Explorer to develop visualisation applications. Iris Explorer uses a module based approach to application development, where tasks are split into individual modules which can be then wired together. The modules represent processes on the data and the wires between them represent the flow of the data between the modules. A simple drag and drop interface is provided for adding and linking the components of an application, or map, together.

Iris Explorer is distributed with a set of standard modules for accomplishing a wide variety of standard tasks such as, reading data, writing images, and standard visualisation techniques, such as plotting histograms. An example of an Iris Explorer map is shown below:

I have been working on creating a 3D graph to represent the volume of queries to our WHOIS system for each hour of a given date, or range of dates. This could largely be accomplished using the standard module set avaialble with Iris Explorer, however there were no modules for the labelling of the axes of graphs which were particularly suited to the task, which meant I had to write my own custom module.

Iris Explorer provides the ability to write your own modules, so it is entirely possible to add custom processes and functionality to an application. The method for developing these custom modules is a two step process, first of all the the layout of the module is defined using the Iris Module Builder, secondly a user function, written in C/C++, is added.

The Module Builder provides a GUI to creating the module window, its data input and output ports and specifying the flow of data through the module internally. This is a logical process where you simply specify the type of input data which is accepted as well as the format of the output data. The appearance of the module itself and the user interface of the module are also specified using a drag and drop WYSIWYG style editor.

The second stage, the user function, controls the processing of the data. So in the case of the Graph Labelling module I have written this would extract the number of dimensions in the data set and number bars of the graph accordingly, as well as creating and positioning axis text labels.

The Iris Explorer Map produced collects data for a given date range, and split this into hourly segments to give a clear picture of the volume of queries to our systems at any given hour of the day. An example of the results of this are shown below:

Background
In many ways, name servers are standardised: the format of queries and responses are defined by standards, as are the ways of transferring zone information into and out of them (zone files, dynamic updates, AXFR, IXFR). This is not the case of the commands and files used to configure and control them, which are specific to each type of server.

Having a common means of interacting with servers would stimulate the development of a common management client, so simplifying operations. Although particularly benefiting users of multiple name servers, good management software should make it easier for occasional users to securely configure and manage their systems.

We have been investigating the idea of a common management interface, as have the IETF, who set up the DCOMA (DNS Configuration Management) committee to consider the problem. We have contributed to the DCOMA discussions, the result of which has been the publication of an Internet Draft containing the requirements for the system.

Implementation
Our approach has to define a protocol (NSCP - Name Server Control Protocol) layered on top of NETCONF, an XML-based protocol for the configuration and control of network devices. In NETCONF, a data model is defined for a network device, with configuration commands being framed in terms of it. NSCP defines a generic object model for a name server, and extends the NETCONF command set with name server-specific ones.

Although the long-term aim is to get NSCP understood by name server implementations, a more pragmatic approach is to put the control into server-specific wrappers, so avoiding the need for changes to the server software. The wrapper accepts NSCP commands and, on the basis of them, modifies the configuration file (and zone files), and causes the server to reload the data. The way this operates is shown in the figure below:

The first step is to create an XML version of the server configuration file. Name server configuration files tend to map into XML quite well as they usually have a hierarchical structure. In our tests, this was accomplished for BIND and NSD by modifying the parser module to emit XML as the configuration file was processed. Although this required a modification to the server software, the modification is a small, localised, change and does not otherwise affect its operation.

Next, an XSL transformation is used to convert the server-specific XML into NSCP, a process that involves mapping server-specific objects and attributes into NSCP objects and attributes. A two-step approach is used to keep changes to the name server software to a minimum: the parser module only needs to create XML that is isomorphic to the configuration file, something that is relatively simple to do. The intelligence needed to convert the representation into an NSCP is held in an (external) XSLT.

Once the NSCP representation of the configuration file is obtained, it can be manipulated with NETCONF commands. “Listing” commands (such as “get-config”) extract the relevant part of the configuration from the representation and send it back to the client. “Modification” commands (e.g. “edit-config”) are applied to the NSCP representation of the configuration file (again using an XSLT) to obtain NSCP describing the updated configuration: this is converted back into the configuration file format. As before, the conversion is a two-stage process to separate out the logic of the object model mapping from the mechanical process of creating the configuration file.

Once the configuration file has been updated, the wrapper forces the server to reload it to apply the changes.

Current Work
A small proof of concept project has shown that this approach is both feasible and practical. Effort is now being put into a pilot implementation.

We have just purchased a 40 port Avocent Mergepoint device for remote server console access management. This saves us using up valuable switch ports and separates these devices from the main network which results in a much more secure console access lan. This device can be managed through Avocent’s DSView3 software, but we are currently running it as a standalone device for testing. The DSView software will also manage their ACS console servers, presenting us with a single solution for console access whether they use a network or serial method of console access.

It is basically a switch that can run DHCP on its ports, and (Linux flash based) software to access and configure everything. It all sounded great so we deployed it out into the field for further testing at a site that was running out of switch ports. At under £3k it is probably cheaper than an enterprise level switch to manage these devices. Using it we connected up 21 remote servers and freed up 20 valuable switch ports. It has dual power, redundant network connections and a serial port for when all that fails.

All well and good so far. Next thing was to configure the web interface and create a new SSL certificate signed by our Nominet CA. This is where it all started to go wrong. The manual linked to on the Avocent website is wrong in so many ways. Firstly the web interface is completely different. Our Mergepoint came with firmware version 4, but the manuals (linked to from the product page) seem to be a previous version. I can cope with a different GUI, but the instructions for creating the certificate used the command line - and were wrong. They said to use

openssl req -new -nodes -keyout private.key -out public.csr

but of course you also need a config file, so the command should be

openssl req -new -nodes -keyout private.key -out public.csr -config /path/to/openssl.conf

with openssl.conf containing (for example) this:

[ req ]
default_bits            = 1024
default_keyfile         = privkey.pem
distinguished_name     = req_distinguished_name

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
countryName_default             = GB

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Oxfordshire

localityName                    = Oxford
localityName_default            = Oxford

0.organizationName              = Organization Name (eg, company)
0.organizationName_default      = Nominet

organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Tech

emailAddress                    = Email Address
emailAddress_default            = example@nominet.org.uk

commonName                      = Common Name (eg, YOUR name)
commonName_default              = MergePointDeviceName

After that you can use the CSR to created a new SSL certificate. The manual says the certificate should go into /etc/httpd/conf/ssl.key (it actually says use the command “cat cert.cert-/etc/httpd/conf/ssl.crt/server.crt” to do this. Does anyone every proof read manuals these days?). This is wrong and the private key and certificate should actually go into /etc/httpd as server.crt and server.key.

Next you should restart apache. Again the manual is wrong and says to use “daemon.sh restart APACHE”. Wrong - that’s the command that you would have used on an ACS console server. The Mergepoint is much more like standard unix here and a simple

/etc/init.d/apache2 restart

or

apachectl restart

is all that is required. All well and good and your new certificate is now in place and working. However, this is a flash based linux so you’ll need to ensure that these new files get saved to flash or they will be lost at the next reboot. There’s the handy manual that tells you to use the saveconf command (correct for once), but it is incorrect in telling you that all files listed in /etc/config_files get backed up. There is no /etc/config_files file (there is one on an ACS console server which is obviously what the manual was based on). The actual file to edit is backup_list.txt. Add these lines to the end:

/etc/openssl.conf
/etc/httpd

Finally if you want to add users to this device and allow ssh access via public key then add /home to the /backup_list.txt file. The users must be added through the web interface as this also updates a database allowing access to the web interface. Then add the users keys, update backup_list.txt and run saveconf. Optionally edit /etc/ssh/sshd_config - we remove root access and password access as we use non-root key based logins only.

I have voiced my concerns about the poor quality manuals to Avocent so that no-one else has to try and reverse engineer things. They originally said that creating a new CSR was impossible, but have since provided a draft of how to do it which was still missing some of the points above (specifically about getting the files saved to flash). A new firmware version is due out in July and hopefully the manuals will be better this time. They still maintain that public key ssh access is impossible without using the DSView software.

It seems I have done something I have been trying to do for years and achieved the impossible.

Background

OFCOM (the UK telecommunications regulator) recently mandated that any VoIP Service Provider that allows end users to make calls to the PSTN must also provide access to the Emergency Services number.

They also mandated that “to the extent technically feasible” the Emergency Handling Agencies (EHAs) must be supplied with the location of the caller. This speeds up the emergency response, and also helps in those cases where the caller is unable to provide their location.

This is already done by the existing PSTN and mobile telephone companies and is not a particularly difficult problem for them as they control all elements of the telephony service. However for VoIP the problem is substantially more difficult - VoIP users can connect to their VoIP service provider over any internet connection, from anywhere in the world.

In some parts of the world the customer is required to notify their VoIP service provider of their location. This isn’t feasible however for users who regularly use their VoIP service away from home, e.g. with a WiFi VoIP handset. Also, if this information is out of date then the consequences can be tragic as seen recently in Canada where the initial ambulance dispatch went to the customer’s previous address, 2500 miles away from where it was needed.

Proposed Solution

The proposed architecture currently being developed by NICC (the UK telecoms industry standards group) is to have the VoIP Providers, the Internet Service Providers, and the Access Network Providers all cooperate to provide location information in real-time.

The VoIP provider will know what the end user’s public IP address is, but they don’t know where it is. When they receive an emergency call they’ll pass the call over the PSTN to the EHA, but at the same time they’ll also send a separate message (using TCP/IP) containing that end user IP address.

The EHA will maintain a database derived from BGP4 real-time routeing data that maps from IP addresses to ISPs. Once they know the ISP, they’ll send the ISP’s “Location Information Service” a HELD protocol request containing the IP address as the lookup key (see IETF Draft geopriv-http-location-delivery). The response from the ISP is an XML document in PIDF-LO format (see RFC4119 and RFC5139) which contains either a “civic address” or a “geodetic location” (i.e. latitude/longitude).

A further complication is that many types of internet access run over a separate Access Network Service which is independent of the ISP. Most ADSL in the UK, for example, is provided using BT Wholesale’s “IPStream” access product which is then packaged up by hundreds of different ISPs.

In these networks it’s common for individual users not to be tied to a specific access line. Hence the Access Network needs to tell the ISP exactly which line is being used. In some cases that might be sufficient for the ISP to determine the address, but in many cases it’s expected that the ISP LIS will need to proxy the HELD request onwards to the Access Network where they should have the most accurate and up-to-date address information.

The diagram below is a simplified representation of how it’s expected to work.

Policy Issues

In my opinion the biggest problem with the architecture at the moment is that neither the ISPs nor the Access Networks actually operate Location Information Services. Partly this is because the relevant standards are still in development, but particularly because as yet there’s no regulation requiring them to do so.

All of the OFCOM regulatory changes so far have been focussed on the VoIP providers, without sufficient acknowledgment of the fact that it’s only the ISPs and the Access Networks that really know where any particular IP connection is being made from. However Ofcom are committed to revisit the regulations for the location issue very early next year, once technical standards (national and international) are clarified.

In my opinion most ISPs are at the moment completely unaware that they might need to do anything, and I hope that this article will stimulate further involvement from the ISP industry. I believe that regulation requiring ISPs and Access Networks to operate Location Information Services is inevitable and it would be better to work now towards a practical solution than to be stuck with an unworkable one later.

It should be noted that the implementation costs for this will be significant, and ISPs looking to recover their costs might start looking at how they could sell-on their customers’ location information to third parties. The potential market for location-based advertising is enormous, but so too are the privacy implications.

Ray Bellis - Senior Researcher

I use a MacBook Pro at work, but of course there are always one or two Windows applications that I need to run too.  Consequently, I recently installed Parallels to allow me to run these both in the office and at home.  I knew that others were using this method to run these apps, so I wasn’t too worried about getting it to work.

When I fired up Parallels for the first time though, the CPU usage shot right up and the whole OS X interface became incredibly sluggish for several minutes.  The strange thing was that this was just starting Parallels itself, not booting the virtual machine.  Once the application was started, sanity was restored.  The offending process was “WindowServer”, so it seemed to have something to do with the graphics processing.  There were reports of other folks having similar problems on the Parallels forum, but none of the solutions seemed to be definitive.

Then I tried starting Parallels while I was away from my desk.  It came straight up with no problem at all.  What was different?  It turned out to be a Good Way USB Display Adapter I am using to allow me to run 2 external monitors from my laptop.  Once this was disconnected the problem went away. So now if I’m at my desk, I unplug the USB cable, start Parallels, then reconnect.  I don’t understand what is going on here, but at least I have a workaround.

Need a quick way to decide who does something:

http://www.youdrawstraws.com/index.cfm

There are two modes:

1. An instant on-screen version, useful if everyone is in the same room.
2. A group session for use when working remotely. These sessions are time-limited.

We used the on-screen version to decide who would go to a remote site to install servers next week. It saved me the job of picking on someone! All it required was to give the session a name, enter how many people there are to choose from, name them, then let the application choose at random.

The group session is a bit more involved. Those involved in the draw can add categories, and options within categories. Once everyone has given input, or the deadline for choosing has been reached, the options are selected at random.

SJphone is a popular free VoIP client (or “softphone”), available for Windows, Linux, and MacOSX.

Most softphones have separate settings for the username and password used to authenticate the SIP REGISTER command, and another to set the SIP Address of Record. SJphone, by default, does not. It has the username and password fields, but it generates a default SIP AoR of username@domain.

There didn’t appear to be any way to support our configuration where the Authentication username is different to the left-hand side of the SIP AoR. After much digging a solution was finally found. In the “Profiles” dialog box there is an “Initialization” tab. On this tab there is a “Caller ID” setting and a tick box marked “Inquired”:

Ticking this box tells SJphone to prompt for the left-hand side of the SIP AoR along with the username and password fields. The other two tick boxes control whether the application remembers the supplied value, or prompts for it each time.

If necessary you can tick the “Full Address of Record” box instead, should you need to supply a SIP AoR with a different right-hand side.