techblog 

I attended the 2009 Hotsos Symposium, an excellent Oracle database performance tuning conference, in Dallas. The event was a great opportunity to hear world-renowned Oracle performance experts present.

One of the most interesting talks was “The Helsinki Declaration: A set of Principles for the IT Community regarding Application Development” by Toon Koppelaars.

Toon described the expansion of features in the Oracle database over the years. He went on to explain that since the advent of Java, more and more functionality has been implemented outside the database. However new frameworks, methods and languages are appearing frequently and often disappearing quickly, sometimes within a couple of years. Many developers are constantly chasing the latest technology because it’s cool and will allegedly solve all presently-experienced problems. This leads to code quickly becoming legacy, having to be re-written and/or no developers having the necessary skills to maintain it. For example how happy or able would your Java developers be to maintain a system built using Struts, a relatively young framework, but now commonly seen as legacy. Would they first spend ages rewriting it, these days called refactoring, to use Spring, the effort for which gives no value to the user.

Although these technologies are changing, what users want has not changed; they still largely want “window on data” applications.

While this is happening the database technology is remaining stable and under-utilised.

Toon recommends replacing this traditional architecture. He has successfully deployed systems using a new architecture, named The Helsinki Declaration (that’s where it was first proposed). This architecture has a thin user-interface layer, deployed in whatever technology/framework is flavour of the month, and business logic and data logic layers implemented in the comparatively very stable database. Only the thin user interface is then vulnerable to the latest fad.

This is described well on his blog. I recommend starting with his first observation and then proceeding to the second, third and fourth observations.

Talking with conference attendees afterwards I was surprised (or maybe I shouldn’t have been) by how many had experienced exactly the issues Toon described on systems development and maintenance projects.

Heard of iTunes? Yes, so have I. Now, have you heard of something called x64, or 64-bit? Yes?

Apple haven’t.

You see, after buying an iPod to use on my x64 system, I’d quite like to use iTunes as well. Would come in really handy, wouldn’t it?

Helpfully, Apple provide a link (a very tiny one) to the x64 download for iTunes. And clicking on it gets you….

Nowhere. Page not found.

Cheers Steve. Nice one. But hey, I can simply contact Apple’s website bods, because this is just a simple error, right?

You can find the link easily enough. The page comes with a nice, simple form. Name, email address, category of error, and comment. Oh, and a submit button. So I filled them all in. Correctly. It isn’t hard.

But it is hard for Apple.  Back comes the form submit error:

“We’re sorry but your feedback regarding Apple’s website was incomplete. Please return to Website Feedback and fill in all of the fields.

Thank you for taking the time to contact us. We greatly appreciate your comments regarding Apple’s website.
Sorry, You are not using a good form.”

But I have filled them all in! The syntax is correct, the email format is correct, there are no special “tripped you up” characters, etc, etc. You can go back as many times as you like, nothing will get you past this.

But at least we have an explanation.

I am not using a good form.

So it’s my fault. I wrote that page, that html, that form. “Your Honour - It Was Me Wot Done It!”

I’m putting this down to a bad Job.

And taking my iPod back. If this experience is anything to go by, God help me if I ever need real support.

I’m currently in the process of adding some reports based on data stored in a database. With annotated configuration in Spring 2.5 and the flexibility of JFreeChart, I was surprised at what can be achieved with such little effort.

The new features added to spring-mvc provide a nice way to avoid having to manage large configuration files and meant that I could concentrate on writing the code for my charts. I’ve added a quick example after the jump

Continue Reading »

Need a quick way to decide who does something:

http://www.youdrawstraws.com/index.cfm

There are two modes:

1. An instant on-screen version, useful if everyone is in the same room.
2. A group session for use when working remotely. These sessions are time-limited.

We used the on-screen version to decide who would go to a remote site to install servers next week. It saved me the job of picking on someone! All it required was to give the session a name, enter how many people there are to choose from, name them, then let the application choose at random.

The group session is a bit more involved. Those involved in the draw can add categories, and options within categories. Once everyone has given input, or the deadline for choosing has been reached, the options are selected at random.

For a number of years I’ve been running Firefox (FF) with advert blocking extensions to make the whole web experience more pleasant.  Originally this was using AdBlock but for the last couple of years it has been with AdBlock Plus (ABP) and Filterset G.

However I recently got a bit fed up with the speed of FF to restart with 30 tabs open and so decided to investigate a bit.  Firstly I got to the bottom of the difference between AdBlock and ABP.  The former has been dead for a long time whilst the latter is live and apparently better developed.  No problem there I was already using the right one.

Then I discovered that the authors of ABP specifically caution you not  to use Filterset G because it will slow down FF and uses a different subscription mechanism.  Instead they recommend that you use a filterset from a recommended list, so I thought I would give it a try.

First though I had to remove Filterset G, which was not quite straightforward.  First you have to remove the FF add-on, easy enough.  But the last set of data it drew down is still present.  So then I de-installed and re-installed ABP but this kept the same data somewhere, so then I manually had to delete the filters added by Filterset G.

The next step was to add another add-on that ABP recommends, the element hider.  Again a simple FF add-on installation.  Finally I subscribed to some new filters, namely EasyList, EasyElement and ABP Tracking Filter, all accessed from the recommended list.

The end result is all the same ads blocked (it may even be better), a much faster FF and fewer of those obvious white spaces where ads used to be.  Altogether a good result.

I’m building a web site for my village and I decided to use a new CMS for some fun.  The two apparently popular choices were Joomla and Drupal, both PHP based with similar funtionality on the surface.  Between the two Joomla is regarded as the easier of the two so I decided to start with that.  In hindsight this was a mistake: Joomla is only half a CMS, as I hope to explain.

As an aside, all the documentation I have read on Joomla is pretty poor at explaining how it all fits together so this post also acts a decent basic guide.

Content structure

Articles are the basic text of web pages and are displayed in the main body of the page.  When you write one it can be either uncategorised or placed inside the content structure.  The content structure is made up of sections at the top level and categories within the sections.   Yes that’s right, Joomla imposes a two level content structure, no more and no less.

Sections and categories are used in various mechanisms to display content, in particular in menus.  For example you can easily show a list of the section or categories or documents in a category. However they are not part of the navigation structure, that is entirely independent.

Site structure

The site structure is the hierarchy within the URLs for your site and in Joomla this is created entirely by your menus.  Your location on the site is always within a specific menu path.  When you move to another part of the site you always move menu path at the same time.

Breadcrumbs always show your current menu path.  If you have friendly URLs configured then these are generated from the menu structure.

It is possible to simulate a menu system that is entirely independent of the site structure by creating a hidden set of menus for the site structure and using the visible menus as redirections into this hidden set of menus.

Interestingly URLs are generated entirely from menus.  Each menu item has an alias and that is used for the URL segment created by that menu.  You can’t assign an alias to an article that is used whenever that is displayed.   This also means that if you have a menu item that displays a list of articles then the URLs for those cannot be search engine friendly.  You simply have to use menus whenever you want a URL like that.

Templates

Templates define the look and feel of the site and you can find a wide variety of free ones out there.  Switching between them is pretty easy.

Modules

Modules generate blocks of content.  Each module is a specific instance of a type of module with individual parameters.  The built-in types of module include breadcrumbs (called mod_breadcrumbs), banners (called mod_banners) and footer (called mod_footer).  You can create any number of modules of the same type each with different parameters.

What appears where on the page

Each template has different areas for content on it called positions.  For example the template ‘beez’ defines the following set of positions:

left
right
top
breadcrumb
user1
user2
user3
user4
debug
syndicate

These positions are defined for each template in the file templateDetails.xml.  Within the Template Manager you can select Preview to see where the various positions are on a page.

Interestingly these positions do not include the central area of the page that the article displays in, they only define the areas around it.  This is one of the “half a CMS” features of Joomla.

The Module Manager is used to put modules into different positions.  You can assign more than one module to the same position and the order field is used to determine the order in which they appear.  For example the position ‘left’ is often used for the left-hand column of a three-column layout and multiple menu modules are commonly assigned to this position.  The first one appears at the top, then the second below it and so on.

For each module you can also select on what pages it appears.  This is done within the Module Manager and is configured by selecting within which menu items this modules appears.

Suppose you want a number of modules to appear in the right hand column of a three-column layout on the front page but the rest of the pages not to have right-hand column then you configure the modules to only appear on the home page and none of the menus.  They then effectively disappear when you leave the front page.

In some other CMS you can have one article replace itself by any other article and keep the same menu path, which would mean that a particular content item might appear in any menu path and theoretically different modules would be visible each time.  However within Joomla one article cannot replace itself with another.  Instead all one article can do is link to another by specifying the full URL including, by definition, the menu.  This means that the modules that are visible for any article are always determined by where it appears in the menu structure.

If you wanted to have one article appear in two ways, with different modules appearing in each way then you need to have two different menu paths for the same article with the modules set to appear differently for each menu context.

Main body of the page

The main body of the page is different from the positions explained above.  It is its own unique area.  The way this main body of the page looks is determined by the menu item that is used to access it.  Remember that even if you link from another page, it still goes via the menus so this always holds true.

Joomla uses the term ‘content layouts’ for the different ways the main body can be displayed.  The built-in content layouts include:

• The article itself
• A blog style layout of the articles in a particular section or category.  This normally displays one article at the top with two columns of articles below it.
• A blog archive of the articles in a particule section or category.  This is the same as the blog style layout but includes drop downs to select articles from a specific date.
• A list style layout of the articles in a particular section or category.  This is presented in tabular format with sortable columns.

Content layout can also be generated by components.  So if you see a Joomla plugin that describes itself as a module only then it cannot generate content for the main body, it has to have component functionality also.  The built-in components include:

• Search form
• Login form
• A wrapper around some external content.  This is implemented using an iFrame.

To select any content layout you have to create a menu item and specify which type of content layout that menu item leads to.  For example if you want to create a search form then you create a menu item (called ’search’ I guess) which is configured to generate a search form content layout.  The oddity from this is if you want a link to the search form that is separate from a menu item then it has to link to a menu item path for the form, there is no other way to create the search form.

Conclusion

This poor control over the main body of the page and the weird way that menus are used for URLs are my main disappoinments with Joomla.  Had I found a number of good third party components that provide real flexibility for content layouts with good URL support then I might have been pacified.  As it is I switched to Drupal and found that light years better - but more on that later.

I’ve recently found myself using the Firefox Extension Scribefire more and more. It hides away nicely as a single button on your status bar, but when opened shows itself to be a well-functioned editor for writing blog posts. You can drag and drop text into a post, work on posts you’ve already published, and most handy for myself, choose from several different blogs to post to, across multiple blogging platforms. I’ve found it works well with WordPress.

The only proviso I’d make is that it appends a “Powered by Scribefire” message to the end of your posts. You can however turn this off. With the editor open, click on the double arrows in the top left-hand corner. In Settings > Publishing is an option to remove this. Apart from this minor annoyance, I do like to use it, it seems a little more natural and easier to compose in than WordPress’s own interface.

Prism is a simple new application from the Mozilla group.  Basically you use Prism to create an application icon for a single web site.  For example, for accessing this blog, I used Prism to create an icon on my desktop that points to the blog URL.   The configuration was without the navigation bar, toolbar or any other ephemera normally found in a browser.  The end result is a single screen with just the web site I want in it.  It looks so much like a desktop application it is uncanny, but underneath Prism is just a lightweight browser instance for a single web site.

Of course a bookmark can get me there pretty quickly as well, but I tend to have at least three Firefox windows open, each with several tabs and it takes a few moments of thought to remember where a particular tab is.  With Prism, for the sites I visit regularly, I have them as icons on my OSX dock.  Yes, there are versions for Linux and Windows as well.

The only problem I’ve found so far is that it does not accept self-signed X.509 certs, but then it is still in beta.  The only thing I don’t like is that it doesn’t automatically grab the favicon for the application icon.

There is even a plugin for Firefox 3 betas that allows you to create a Prism app directly from Firefox.  I haven’t tried that yet though.

We run a “digital signage” screen in our foyer - in other words it’s a large plasma screen displaying some static content (and also ‘live’ content such as TV) that is simply served from a box. The box (aka “net-top-box”) is a certain ‘NTB115′ from a company named OneLan; it runs Mandrake and has been described elsewhere as a “Mini-ITX board in a nice case with a video output card”. It has a reasonable web interface for customisation of the content layout but you can also configure it via local xml files directly.

Whilst it is reasonably flexible in displaying TV or static web content, some of it’s features are not so design friendly, such as RSS feed inclusion. With these feeds, you’re only able to style the font of the heading and subsequent description - not the width of the RSS area on the screen, nor the length of the content itself (first 200 characters only? Sorry.). Consequently if you run out of space the RSS feed simply truncates, and so you’re limited to displaying an RSS area as one long line (no line breaks) in order for it to be readable, as you can’t wrap it.

As I wanted to pull and also display (via the net-top-box) our own RSS content from our CMS (we use InfoGlue), I thought that it would be easier to ‘pull’ the content into a CMS resident HTML ‘page’, style it there and then (whilst still within the CMS), and *then* call that resulting page into the net-top-box layout as an ‘HTML content area’ (you only get two choices here - RSS their way or an HTML link). Since our RSS content is already XML, the best way to generate a styled HTML page from it is obviously to transform it via XSLT. We don’t really create a page as create a static url to a xml file which in turn references an xslt file - so the transformation (and resulting HTML) is done on the net-top-box and not within InfoGlue. There’s nothing in the manual to say you can’t do this, but also nothing to say you can, either.

Simple in theory - but it turned into a bit of a pain. It’s difficult to know (unless you’re a linux expert - I’m not, I’m a web designer) exactly how the innards of the net-top-box work as regards reading HTML straight off the bat or having to render it from a transformed XML file, but I do know it transforms it’s own XML files, so there’s a XSLT processor in there somewhere. When it came to test the files from the CMS on the box, the result on the display screen was simply a blank area, which usually denotes whitespace issues, character-encoding issues or simpy badly formed XML, but having eliminated any of these causes I was left with a CMS vs. net-top-box puzzle - eg. something in the way the files are generated and then called didn’t match.

It turned out to be the media types (after much testing) - no errors as such, but just a rather odd way they are treated within InfoGlue or within the net-top-box (exactly which system, I’m still unsure of). For an xml or xsl file to exist within InfoGlue it has to have (in our CMS structure anyway) a content-type defined - additionally a media or output method can be explicitly defined in the file itself. The RSS xml file contains a reference to <?xml-stylesheet type="text/xml" href="xslstylesheetname"?> - at this point, I expected the content-type contained within the xml file to be referencing the stylesheet as type=”text/xsl” - not type=”text/xml” - but it needs to be “text/xml” if it’s to work. The CMS at this point has this same xml file itself as being of content-type “text/xml” (which is also correct, as it’s not an XHTML file).

The xsl file has its output-method as <xsl:output method="html"/>, again as expected as the first child of the root node is going to be <html>, but the InfoGlue content-type for this file, for some reason, *has* to be “application/xml” - or it simply won’t marry the two files up.  Again, at this point, I would have expected the content-type to be type=”text/xsl” (even though “text/xsl” it is not considered to be the appropriate MIME type for XSLT files, it has to be for Internet Explorer to render it, and IE is the engine in the net-top-box for displaying content…).

So my conclusion is that it’s a MIME type issue - whether it’s the net-top-box not having the MIME types needed configured, or whether it’s an IE issue, or both, I’m not sure. Buy hey, it works, at last. ;-)

On 10 January 2008 our web site was unavailable for several hours. The cause for this outage, and our failure to detect it threw up some interesting points.

We have always seen examples of abuse of our systems. Usually this takes the form of high volumes of requests sent to the whois server. We respond by throttling the traffic or, in more extreme cases, blocking the originating IP address from accessing the server. In early January we became aware that two IP addresses were responsible for 68% of the bandwidth to our web site. Each address was pulling our list of tags once a second, 24 hours a day. This is the biggest page on the website. Between them the two IP addresses were responsible for more than 20GB of traffic per mont. Either one was using more than ten times the bandwidth of any other address that accessed the web site.

Our web site sits in the DMZ, outside of the firewall. We have a Juniper router sitting in front of the DMZ and use ACLs to limit access to the web site. Now I like Juniper routers, particularly the CLI. Editing the config on a Juniper, especially if you are only used to Cisco, is a pleasure. However, you do have to be aware of the consequences of your actions. When the decision was made to block these IP addresses a new term was created in the ACL to block access to the web site.

This term consisted of:

1. Source addresses - the offending IPs
2. Destination address - our web site
3. Action to be taken - in this case, discard all packets received

Pretty soon after the block was imposed we were contacted by the owner of the IP addresses. It seems they intended to pull the tag list once a day and had misconfigured the script. I’m of the opinion that there is no need to pull the list like this, but we decided to remove the block anyway. The engineer who had imposed the block decided to leave the term in place, in case it needed to be re-applied. He chose to remove the source addresses only. In doing this we were left with a term that read:

1. Destination address - the web site
2. Discard all packets received

Which blocked any access to the web site from outside of Nominet. This is the first interesting point. The decision to leave the term there was a reasonable one, and if only one IP address had been removed then we would have been fine. [There is an option to deactivate a whole term, but he was unaware of this.] It makes me realise that we need a proper firewall for the DMZ. Router ACLs are only really applicable at layer 3.

The next interesting point is that we were unaware that the web site was not visible to the outside world. The term was removed once we were made aware of the outage, but this information came from outside of Nominet. We have a sophisticated monitoring system, based around nagios. This gives us a fully configurable and timely view of our systems, but only as seen from within Nominet. We already put our authoritative nameservers within other people’s ASes, so these would be candidate sites for monitoring stations. But one thing I want to do is make more use of things like the RIPE NCC DNSMON service. This gives us a global view of .uk authoritative nameserver availability. At present we use this on an ad hoc basis when diagnosing nameserver incidents. I want to incorporate the raw data (which we have access to) into our monitoring system to ensure we see events that would not be detected by our monitoring system. Including incidents which segmented the network, for example, where we could still see the nameserver but half the internet could not.

I would encourage anyone who can to sign up for a RIPE TTM box to increase the coverage that DNSMON has.