techblog 

For any number of reasons, you may find yourself wanting to create only part of a Spring context without satisfying all of its @Required dependencies (and their dependencies, and their dependencies’ dependencies…) with real objects:

• Reducing the number of Spring beans your integration tests need can drastically speed up your unit tests, since Spring container startup can take a while and consume a lot of heap space.
• In my case I wanted to write an integration test that only needed a thin vertical slice of beans to get Hibernate working, without the (rather large number of) other beans in the rest of the context definition.

In conventional unit testing, this is easy: you use mocks or stubs at the boundaries of your test objects. Why not extend this to Spring beans?

By rejigging some import statements, I was able to use Spring’s instance-factory bean declarations and dynamic mocking to do just that. The process has two parts: creating the factory class, and declaring the stub beans…

Creating the factory

package uk.nominet.testing;
 
import net.sf.cglib.proxy.Enhancer;
import net.sf.cglib.proxy.MethodInterceptor;
import net.sf.cglib.proxy.MethodProxy;
 
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
 
/**
 * Stub spring bean factory class. Call createStubBean("classname") to stub out a bean.
 */
public class StubBeanFactory
{
    /**
     * Factory method to create stub beans.
     * @param className to create a stub bean for.
     * @return stub bean.
     * @throws ClassNotFoundException if className doesn't match a real class.
     */
    @SuppressWarnings("Unchecked")
    public <t> T createStubBean(String className) throws ClassNotFoundException
    {
        // get the class to proxy
        Class</t><t> proxyClass = (Class</t><t>)Class.forName(className);
 
        // the specifics of proxy creation depend upon whether we need to proxy a class
        // or an interface - which is it?
 
        if (proxyClass.isInterface())
        {
            // proxying interfaces is easy with the standard java proxying classes
            return (T)Proxy.newProxyInstance(proxyClass.getClassLoader(),
                                             new Class[]{proxyClass},
                                             new StubMethodHandler());
        }
        else
        {
            // proxying classes requires some cglib magic...
            Enhancer enhancer = new Enhancer();
            enhancer.setSuperclass(proxyClass);
            enhancer.setCallback(new StubMethodHandler());
            return (T)enhancer.create();
        }
    }
 
    /**
     * Helper class to handle method invocations on stub beans.
     */
    private static class StubMethodHandler implements InvocationHandler, MethodInterceptor
    {
        /**
         * @see InvocationHandler#invoke(Object, Method, Object[])
         */
        public Object invoke(Object proxy, Method method, Object[] args) throws Throwable
        {
            // you could put some logging here...
        }
 
        /**
         * @see MethodInterceptor#intercept(Object, Method, Object[], MethodProxy)
         */
        public Object intercept(Object o,
        			Method method,
				Object[] objects,
				MethodProxy methodProxy) throws Throwable
        {
            // you could put some logging here...
        }
    }
}
</t>

Creating stub beans

Stubbing out a spring bean is simply a case of creating and using the factory:

<beans>
    ...
    <!-- this is our stub bean factory -->
    <bean id="stubBeanFactory" class="uk.nominet.testing.StubBeanFactory"></bean>
 
    <!-- stubbing out either interfaces or classes is exactly the
         same - just call the factory -->
    <bean id="tokenDao">
</bean>          factory-bean="stubBeanFactory"
          factory-method="createStubBean"&gt;
        <constructor -arg value="uk.nominet.authentication.TokenDao">
    </constructor>
    ...
</beans>

What’s the point?

By stubbing out the unneeded beans:

• the number of beans in the testing context was reduced from nearly 400 to less than 40;
• the running time for the integration test was reduced from 40 seconds to 10 seconds.

The second point here is key - faster tests leads to tests that get run at all…

Having been brought up on C# and .Net, Java enumerations are a breath of fresh air - you can do all sorts of fun stuff like give them methods and properties like any other object. Well, almost.

I’ve been writing some code to interact with a payments system. The rules as to whether specific types of payments can be taken are rather procedural, and so don’t instantly lend them to OO’s usual easy encapsulation. Typically I’d end up writing something like this:

public enum ProductType { SHOES, SOCKS, SHIRTS; }
 
public static boolean isBuyable(ProductType product, Account payeeAccount)
{
    // nasty procedural rules and switch statements to 
    // determine whether things are buyable for the given account...
    switch (product)
    {
        case SHOES:  /* ... */ break;
        case SOCKS:  /* ... */ break;
        case SHIRTS: /* ... */ break;
        default:
            // panic! someone added an enum instance 
            // without updating the business process!
            throw new PanicException("run!");
    }
}

This time I thought I’d try putting the rules about product types on the product type itself. Easy enough in Java: stick an abstract method on the enum instances and let them do their own work:

public enum ProductType
{
    SOCKS
    {
        @Override
        public boolean isBuyable(Account payeeAccount)
        {
            // rules about sock availability only
        }
    },
 
    // other product types go here, each looking after its own rules
    // and only its own rules:
 
    SHOES  { /* ... */ },
    SHIRTS { /* ... */ };
 
    // abstract base method for all enum instances in the group
    public abstract boolean isBuyable(Account payeeAccount);
}
 
// the nasty method previously required could then be reimplemented easily:
public static boolean isBuyable(ProductType product, Account payeeAccount)
{
    return product.isBuyable(payeeAccount);
}

The advantages of this approach include:

• Business rules are ‘close to’ the objects they concern. In this case I could have written a method like isBuyable(ProductType) on the account object, but the payments logic is naturally separate from the rest of the system, and this approach keeps it that way.
• Business rules for the availability of specific products are cleanly separated - each product only has to manage its own rules.
• There are no switch statements (the equivalent for enums of the dreaded instanceof for objects) and the associated PanicExceptions.
• It’s just not possible to add enumerated instances that don’t respect the business rules - that abstract prevents their compilation…

Unfortunately, having instance methods on enums breaks their serialisability. Or at least it does with our combination of Hessian and Java for remoting - I’ve not tracked down the precise cause. My first thought was that enum deserialisation was using reflective instantiation under the hood, so I removed the abstract keyword from the root declaration of isBuyable, but no such luck.

A quick refactor put each enumerated instance’s business logic into a discrete interface implementation, but this slightly bloats an otherwise neat solution to the problem. Reality trumps design once again…

IntelliJ, the extremely good Java IDE from Jet Brains will often fail to pick up external file changes; for example performing a subversion “svn up” on your source code.

The problem can be fixed by removing or renaming fslogger in IntelliJ’s bin directory (e.g. /Applications/IntelliJ IDEA 7.0.3.app/bin) as is well documented.

Jet Brains continue to ship this library — enabled by default — for version after version of IntelliJ. Each time I install a new version, the first thing I have to do is disable fslogger and if I forget, then I tend to end up wrongly accusing a colleague of checking-in broken code.

The fslogger library doesn’t work for anyone here as far as I am aware. Shipping a library that actually stops a product from working, when it works so well without it is something I find hard to understand. Perhaps Jet Brains would consider removing this library from the IntelliJ distribution or shipping it disabled by default.

In the recent Jolt Awards you may (or may not) have noticed an award to Structure101 by Headway Software. We’ve been using this tool for a while now and I thought it might be worth giving my impressions. Put simply it allows you to visualise the dependencies within your Java codebase and highlights areas of complexity. It does this using a concept called ‘Excess Complexity’ which Headway Software explain on their website. It shows this complexity at every granularity from top-level package right down to method.

Obviously decisions about this sort of thing can never be entirely automated, but this tool does give you a good way to track down design ‘smells‘ in the code. It may be that the complexity is not a problem, but it certainly gives you cause to reconsider your design decisions in that area.

You may be thinking at this point “But my IDE already shows dependencies, so why would I need to pay good money for this?”. That may be true and I know for a fact that IntelliJ IDEA at least will display a grid of package dependencies for you. But if you decide that you want to unpick some circular dependencies it is very hard to see what’s causing the problem. Compare:

I know which of these is more intuitive. The Structure101 diagram also has the advantage that clicking on the links show exactly which piece of code is causing the dependency.

At the moment we are using the tool to keep our dependencies under control by just analyzing them and graphing the overall complexity figure on a whiteboard. This helps to motivate you to keep it down. We will be fully integrating it with our continuous integration server when we move over to a new system in the next week or so. After that we may consider using the IDE plugins to highlight design problems to individual developers as they work.

Overall, a nice piece of work to keep control of complexity and one that is now free for open source projects.

We have a Cisco CallManager (CCM) VoIP telephone system here at Nominet and I’m currently investigating how this system might be adapted to support ENUM lookups for outbound calls so that calls can be made directly to other VoIP systems without going over the PSTN. There’s no built-in support for ENUM in CCM at all, so some sort of plugin would be required.

I’ve not been able to find any internal plugin APIs, but CCM does support most of Version 1.2 of the Java Telephony API. My hope is that this API will provide tight enough integration with the system to allow the necessary outbound call routeing.

I found that there are very few examples around of how to use JTAPI to actually do anything. Cisco does provide a sample application, but it’s unnecessarily complicated. The code below is therefore presented as a much simplified version of that application that does nothing except tell a CCM extension to dial a specified number.

To use this code you’ll need to obtain the jtapi.jar file from the CCM Administration software via Applications->Plugins->JTAPI. You’ll also need to create a new “Application User”, and add that user into the JTAPI Groups, and also grant that user access to some terminals.

With jtapi.jar in your CLASSPATH, this application can be run as:

% java MakeCall {host} {user} {password} {extension} {number}

As soon as the program has told CCM to place the call it will exit.

As this is only an example there’s only minimal input and error checking, but the application will generate an appropriate error message if you supply incorrect JTAPI login details, or if you try to trigger a call from an extension that your JTAPI user doesn’t have access rights to.

There are two specific parts of the code that deserve special mention:

1. The com.cisco.cti.util.Condition class is not part of the JTAPI standard, but provides a useful semaphore object. In this case it’s used to make the main thread block until the ProvInServiceEv event is received by the anonymous ProviderObserver object.
2. It appears that it’s always necessary to add a CallObserver to the Address object (even if that observer subsequently does nothing useful itself) since the JTAPI library throws an exception if you don’t. I couldn’t find this behaviour documented anywhere.

MakeCall.java

import javax.telephony.*;
import javax.telephony.events.*;
import com.cisco.cti.util.Condition;
 
public class MakeCall
{
	public MakeCall(String[] args) throws Exception
	{
		String hostname = args[0];
		String login = args[1];
		String passwd = args[2];
		String src = args[3];
		String dst = args[4];
 
 		/* start up JTAPI */
		JtapiPeer peer = JtapiPeerFactory.getJtapiPeer(null);
 
 		/* connect to the provider */
		String providerString = hostname;
		providerString += ";login=" + login;
		providerString += ";passwd=" + passwd;
		Provider provider = peer.getProvider(providerString);
 
		/* wait for it to come into service */
		final Condition	inService = new Condition();
		provider.addObserver(new ProviderObserver() {
			public void providerChangedEvent (ProvEv [] eventList) {
				if (eventList == null) return;
				for (int i = 0; i < eventList.length; ++i) {
					if (eventList[i] instanceof ProvInServiceEv) {
						inService.set();
					}
				}
			}
		});
		inService.waitTrue();
 
		/* get an object for the calling terminal */
		Address srcAddr = provider.getAddress(src);
		srcAddr.addCallObserver(new CallObserver() {
			public void callChangedEvent (CallEv [] eventList) {
				/* ignored */
			}
		});
 
		/* and make the call */
		Call call = provider.createCall();
		call.connect(srcAddr.getTerminals()[0], srcAddr, dst);
	}
 
	public static void main(String[] args) {
		try {
			new MakeCall(args);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			System.exit(0);
		}
	}
}

Here at Nominet we use Hessian to build our Java middle-tier services. We’ve been using version 3.0.20 for a while and, once you get used to its idiosyncrasies, it’s pretty workable and stable.

Yesterday I tried to update to the latest version, 3.1.3. Suddenly all hell broke loose. A bunch of domain entity objects we’ve been persisting through Hibernate stopped deserialising cleanly at the Java client end. A little debugging revealed that:

• Instances of java.sql.Timestamp would drift by a few seconds from their transmitted version. We’d been using just such fields along with a @Version annotation to provide optimistic locking for Hibernate-managed entities. Lots of StaleObjectStateExceptions get thrown.

“No problem” I thought, “they’re just there because we haven’t moved them over to cleaner, less coupled Integer versioning yet. Refactor!”. Half an hour of refactoring class and table definitions later, and the local unit tests pass again. Back to service-level tests:

• Deserialisation now fails for a couple of alternating reasons, each of which throws a HessianFieldException:
  • Enumerations randomly fail to deserialise, complaining that they couldn’t be assigned from HashSets;
  • Sets of enumerations also fail, with a root IndexOutOfBoundsException from an ArrayList.

At this point I gave up and reverted to the previous version of the library. Another developer here also had the same problems with the updated version, so I think that for now we’ll be sticking with the tried and trusted version 3.0.20. Just because a new library is there doesn’t mean you have to upgrade straight away, or even at all.

Having said that, if anyone can shed any light on these problems it would be very much appreciated…

This probably isn’t news to many but I only recently noticed that Oracle AQ can manage JMS queues. We don’t use a full blown J2EE application server so our options when using a JMS broker is to either have a standalone server/cluster (more work for SAs) or to embed the JMS server in to our middleware (more work for me). By using the database we add no new dependencies or software so this is a pretty neat solution. Another big advantage for us is that it looks like we would be able to create messages inside PL/SQL procedures but I may be getting ahead of myself here.

There isn’t much to it but it was a lot harder to get to the same point than with the other JMS brokers I have been testing. There is a lot of documentation to go through and it is all fairly general. I also couldn’t find a good example so I’ve created a quick howto… Continue Reading »

I’ve just started to look in to the various message queuing servers for our applications and first up is Sun’s Open Message Queue. I’m drawn to this as it is open source, has C and Java client APIs and also has high availability features using a persistent store.

Setting up a test environment was quite straight forward, as was using it from Spring. Here’s how I did it…
Continue Reading »

Having got the basic querying functionality and configuration sorted in dnsruby, I’m now implementing the high level lookup. I had hoped there would be an RFC to guide me here, but the closest I can find is in RFC1034, section 5.3.3, which is really for nameservers rather than stub resolvers. It says :

1. See if the answer is in local information, and if so return it to the client.

2. Find the best servers to ask.

3. Send them queries until one returns a response.

4. Analyze the response, either:

a. if the response answers the question or contains a name error, cache the data as well as returning it back to the client.

b. if the response contains a better delegation to other servers, cache the delegation information, and go to step 2.

c. if the response shows a CNAME and that is not the answer itself, cache the CNAME, change the SNAME to the canonical name in the CNAME RR and go to step 1.

d. if the response shows a servers failure or other bizarre contents, delete the server from the SLIST and go back to step 3.

In order to figure out the exact algorithm for dnsruby, I thought I’d look at some current open source stub resolver implementations, in dnsjava and Net::DNS / pnet-dns. I also thought I’d check out Ruby’s native resolv.rb (incomplete) DNS implementation.

dnsjava

dnsjava has several layers which perform lookups. The SimpleResolver targets a single nameserver and sends queries to it. No retries are performed in the case of timeouts or other errors.

The ExtendedResolver class has a clutch of SingleResolvers which it uses to fire queries at multiple nameservers. The class implements the retry mechanism in the case of failure. The basic algorithm is as follows :

• Send the query to the first nameserver (this could be first in a list, or a randomly selected nameserver from the list)
• If the response is good, or an NXDOMAIN is returned, then return it to the client.
• If it is a timeout, then retry that server, but also, if it is the first timeout for that server, send the first query to the next server on the list. If it’s not the first timeout, then simply try that nameserver again up to the maximum number of retries
• If an error is returned as the last response from the last nameserver, then send that to the client

The Lookup class adds another layer of complexity. This class implements a results cache, and performs lookups on names with the search list applied. When a query is made, the cache is checked first. If no records are found, then the ExtendedResolver which belongs to the Lookup is used to query all configured nameservers for the name (with the previous algorithm). The results are then added to the cache. CNAMEs are followed by the Lookup class.

Dnsjava can therefore be seen to follow what specification there is in the RFC.

In the case of totally unresponsive servers, this algorithm may take a total time of (retries*timeout + num_nameservers * timeout). However, if using the dnsjnio asynchronous extension, multiple retrying queries can be run concurrently in a single thread (plus a worker thread to do the I/O).

Net::DNS

The Net::DNS project has a different model. The Resolver class implements both the SingleResolver and ExtendedResolver functionality of dnsjava. It does not implement any form of result cache, and query forms exist to either append or not append the search list to query candidates.

The algorithm, the implementation of which is horrendously long and complicated, is as follows :

• If the search list is to be applied, then for each element in the list, that element is appended to the query name, and the remaining steps are run for each result. Otherwise, the remaining steps are only run for the queried name
• retry Retries of the following steps are attempted, and for each round of retries, retrans is doubled
• If udp_timeout is more than 0, and udp_timeout seconds have elapsed since the query was started, then the query times out to the client
• The first nameserver is queried, and the response awaited for retrans/num_nameservers seconds.
• If NXDomain or the result is returned, then that is passed back to the caller
• If the server returns any other error, then it is removed from the list of nameservers for this query
• If the query timed out, then the next nameserver is tried.
• Once all nameservers have been tried once, and all timed out, then this process is repeated up to retry times (see above)
• If still no response after all of this, then the query times out to the client

If udp_timeout is set, then this is the maximum time that the query can take.

If udp_timeout is set to 0, then the retrans and retry parameters control the query time. In this case, the query could take up to ((2^retry) * retrans) seconds. [It could take an additional num_nameservers * retrans seconds if the nameserver is totally duff]

Ruby resolv.rb

The original Ruby DNS implemenation, resolv.rb, has a slightly different algorithm. This implementation only performs retried lookups, and will apply the searchlist. The algorithm is as follows :

• All nameservers are tried sequentially. If a valid response is returned then that it is returned to the client. If an error is returned, it is returned to the user and the search stopped
• Once all nameservers have been tried, and all timed out, then they are all tried again with double the timeout value. This is tried three times (so all nameservers can be tried up to 4 times with increasing timeouts)
• If this is not successful, then the whole process is repeated with the next “candidate name”, which is formed by appending the next element of the search list to the domain

There is no way to set a per-client-query timeout.
This algorithm is probably the worst of the three - at least with dnsjava, the retries are concurrent once the first round of queries have timed out. However, Net::DNS has the nicest algorithm; splitting the timeout per retry between the number of nameservers limits the worst case scenario, and gives all nameservers a chance to respond. However, if the initial retry interval is set too short by the client, then more queries could be sent than are actually required (especially if several nameservers are targetted).

I think I’m going to go with a combination of the Net::DNS and dnsjava approach (as with so much in this project!), and have the option of :

• A total timeout for the query
• A retransmission system that targets the namervers concurrently once the first query round is complete, but in which the total time per query round is split between the number of nameservers targetted for the first round. and total time for query round is doubled for each query round

This system should limit the total number of queries sent in the best case, but allow good response from a poor or worst case.

Of course, unlike Net::DNS, all these queries will be done in the background by a worker thread.

Any comments?

I’ve been having some issues with Ruby IO#select() recently, which I think I now have my head around.

My Java non-blocking DNS project, dnsjnio, has a select loop in one thread, which processes all of the IO for all DNS queries. The client thread can wait on a blocking queue for more responses from the select thread. I had hoped to follow this model for Dnsruby. However, there are some differences between the select implementation in Ruby and Java which make things a little more tricky.

In Java, there is a Selector.wakeup() method which a client thread can call to rouse the select thread from its slumber. This means that you can send something (or ask the select thread to send something), and then wake the select thread up to deal with it. Otherwise, nothing would happen until the select() call timed out, or something happened on one of the descriptors it was monitoring - which could be a long time on a quiet system.

Ruby doesn’t have this facililty. So once a thread calls select(), it’s not going to do anything until either the timeout or a descriptor event occurs.

I read a comment by Dave Thomas in which he suggested waking the select thread up with an Exception. Well, I tried that, and it does work, but it doesn’t seem happy under JRuby. OK, it is possible to set up simple test code to get this working in JRuby. However, when you start hitting it hard it just doesn’t seem to work right, and I haven’t had the time to dig around in JRuby to figure out why.

UPDATE - I think this probably has something to do with it. Sounds like JRuby is having problems with Thread.critical=, which was getting used heavily in my multi-threaded stress testing code!

It would be nice to try to get the exception signalling working again in the future - the code is all still there. However, it is necessary to use excessive synchronisation between the threads in order to ensure that the exception is only fired at the time the select call is blocking. Basically, all the processing of the results of select has to be done in a synchronized block, guarded by the same Mutex that guards the exception raising.

Another possibility is to use pipes to signal the select that it is time to wakeup. Whilst this is a hack, it was still worth trying. The problem with it is that (once again) the behaviour or Ruby varies across platforms. Pipes aren’t implemented in Windows MRI, so we can’t use them.

This leaves the only option being to run select with a short timeout. Of course, this leaves us with a thread effectively in polling mode, which is not good. So, once the thread has not serviced any queries for a defined period of time (currently 1 second), it shuts itself down, to be restarted the next time a client query is sent. This leaves us with a worst case not much worse than the standard resolv.rb, and a best case significantly better!

It would be nice if there a non-hacky way to do this. If anyone knows of one, I’d love to hear it!!