techblog 

Mutation testing is a technique for checking how good your unit tests are.  It mutates a class by for example swapping a subtraction for an addition or by negating an if statement.  It then runs the unit tests for that class.  If none of them fail, then maybe your tests are not good enough.

We took a look at Java mutation testing tools some time ago as a possible addition to our continuous integration system.  But at that time the tools fell short.  Jester works by mutating the source files.  We found that this was just too slow with all the compilation it needs to do.  Jumble is a bit smarter in that it mutates the bytecode of the compiled class files.  However, when we evaluated it only JUnit 3 tests were supported and we were already well on the way to transitioning most of our tests to JUnit 4.

Recently though, Jumble was modified to work with JUnit 4 tests, so I thought it was time to take another look at it.  I found it quite tricky to get it working as there is no ant integration and it only runs as a command line application.  However, with a bit of persuasion I managed to get it running over our codebase using ant.  The details of how I did this are given below, but I think that a better solution would definitely be a fully fledged custom ant task.

So how did I get on?  Initially I had some niggly classloader problems. It seems that you need to tell Jumble’s mutating classloader to defer the loading of various sets of classes to the default classloader.  I needed to do this for the JMX classes found under javax.management by specifying the command line flag --defer-class=javax.management. Once I’d done this I had it working and did indeed find some interesting things.  I found tests that had been cut-and-pasted and not changed to test what they claimed to test. I found some test data that wasn’t up to the job and I found an actual bug in the code.

However, I hit a roadblock once the code under test used a database.  For some reason Oracle’s JDBC driver would not behave. Even before any mutations were applied it would insist on trying to connect with a null password.  This meant that it tried a number of times before locking itself out of the database.  I assume that this is some kind of classloader thing, but it seems strange that it manages to successfully contact the database only to completely mess up the credentials.  I’ve contacted the developers of Jumble (who are based in New Zealand incidentally), but no solution has been forthcoming as yet.  Until this problem is fixed we won’t be able to add this to our continuous integration system, which is a shame, as I think it could be a useful tool.

So, how did I integrate Jumble into ant?  I used the follow macrodef to run Jumble:

<macrodef name="do-mutation">
    <attribute name="class-to-mutate"/>
    <sequential>

        <!-- Use this trick to convert a reference to a classpath to classpath as a string -->
        <property name="the-classpath-as-a-string" refid="execute-test-classpath"/>        

        <java dir="${base.directory}" classname="com.reeltwo.jumble.Jumble" fork="true">

            <!-- all we really need in this classpath is the jumble library -->
            <classpath refid="ants-own-classpath"/>

            <!-- but then it needs everything in the JVM it forks: -->
            <arg value="--classpath=${the-classpath-as-a-string}"/>
            <arg value="--exclude=equals,hashCode,toString"/>
            <arg value="--defer-class=javax.management."/>
            <arg value="@{class-to-mutate}" />
        </java>

    </sequential>

</macrodef>

To get this to work you will need the classpath used to run the tests set up with the id execute-test-classpath and the classpath used by ant to find custom tasks set up with the id ants-own-classpath. The latter will need to include the jumble jar. As you can see I have excluded the equals(), hashcode() and toString() methods from being mutated as the first two are often generated by the IDE anyway. In the case of toString, this rarely contains important logic.  As mentioned before, the JMX classes are deferred to the default classloader. You may need to add some other packages to get it to work in your environment.

The above macrodef will mutate a single class. But we want to run this across our whole code base. To do this I had to resort to using some further ant tricks in the shape of the ant-contrib library which adds additional functionality to ant. As I said before, a better solution would be to write a proper ant task. Here is how the macrodef is called to run Jumble over a whole project:

<target name="mutation.test">    

    <!-- The ant contrib jar contains the "for" task -->
    <taskdef resource="net/sf/antcontrib/antlib.xml" classpathref="ants-own-classpath"/>

    <!-- Strip off the leading directory and the .class. Then replace the slashes
         with dots and separate each one with a comma.
         Results go into the classlist property -->

    <pathconvert dirsep="." pathsep="," property="classlist">
        <mapper type="glob" from="${build.dir}/*.class" to="*"/>
        <fileset refid="classes-to-mutate"/>
    </pathconvert>

    <!-- Iterate through the comma separated list and call jumble -->
    <for list="${classlist}" param="class">
        <sequential>
            <do-mutation class-to-mutate="@{class}"/>
        </sequential>
    </for>

</target>

This takes a fileset with id classes-to-mutate which consists of classes under the directory ${build.dir}. It turns the path into a package and removes the .class from the end to get a list of classes to mutate. (NB This was written to work with Unix style paths, it may need alteration to work under Windows). Then the macrodef given previously is called for each. Note that we refer to the ants-own-classpath classpath again which must contain the ant contrib jar this time.  The code given above was put in a standard ant build file included by other projects. The fileset to mutate could then be defined in each like this:

<fileset id="classes-to-mutate" dir="${build.dir}" includes="**/*.class">
        <exclude name="**/WeWantToLeaveThisOut.class"/>
</fileset>

UPDATE: The story gets even weirder. One of the Jumble developers got back to me with a suggestion for how to fix the Oracle problem, which was to add the JDBC driver to the list of classes deferred to the parent classloader. This didn’t help, but I then discovered bizarrely that the JDBC problem goes away if you are connecting to an 11g database as opposed to a 10g one. So it means that somehow the 10g JDBC driver fails to connect to 10g when run by Jumble, but succeeds against a later version. Curiouser and curiouser…

I’m pleased to announce the release of enumdroid.

This application adds ENUM (E.164 Number Mapping) support to your Android phone.

Each time you dial a full international number (i.e. starting with a ‘+’) your phone will check the DNS for additional routing information and offer you a list of alternate contact methods.

The application is open source (under the Apache License) and the code is available for download from Google Code.  The application can be downloaded from the Google Market under Applications -> Communication

Here are some screenshots, which show in turn:

1. Nominet’s switchboard number being dialled
2. ENUM results being returned
3. A call being placed over the PSTN to a tel: URI
4. The ENUM application’s settings page

I attended the 2009 Hotsos Symposium, an excellent Oracle database performance tuning conference, in Dallas. The event was a great opportunity to hear world-renowned Oracle performance experts present.

One of the most interesting talks was “The Helsinki Declaration: A set of Principles for the IT Community regarding Application Development” by Toon Koppelaars.

Toon described the expansion of features in the Oracle database over the years. He went on to explain that since the advent of Java, more and more functionality has been implemented outside the database. However new frameworks, methods and languages are appearing frequently and often disappearing quickly, sometimes within a couple of years. Many developers are constantly chasing the latest technology because it’s cool and will allegedly solve all presently-experienced problems. This leads to code quickly becoming legacy, having to be re-written and/or no developers having the necessary skills to maintain it. For example how happy or able would your Java developers be to maintain a system built using Struts, a relatively young framework, but now commonly seen as legacy. Would they first spend ages rewriting it, these days called refactoring, to use Spring, the effort for which gives no value to the user.

Although these technologies are changing, what users want has not changed; they still largely want “window on data” applications.

While this is happening the database technology is remaining stable and under-utilised.

Toon recommends replacing this traditional architecture. He has successfully deployed systems using a new architecture, named The Helsinki Declaration (that’s where it was first proposed). This architecture has a thin user-interface layer, deployed in whatever technology/framework is flavour of the month, and business logic and data logic layers implemented in the comparatively very stable database. Only the thin user interface is then vulnerable to the latest fad.

This is described well on his blog. I recommend starting with his first observation and then proceeding to the second, third and fourth observations.

Talking with conference attendees afterwards I was surprised (or maybe I shouldn’t have been) by how many had experienced exactly the issues Toon described on systems development and maintenance projects.

In what could be seen as further proof of the imperfection of multi-threaded programming, I have just released another version of dnsjnio. This version includes fixes from Allan O’Driscoll that deal with very obscure synchronisation issues.

It also includes changes to the test system which cope with the fact that Java DatagramSockets have slight differences of behaviour on different platforms. Regardless of the operating system you use, and the power of your computer, you should now have no problems with the dnsjnio test code. Previously, some versions of Linux (and Solaris), and slower machines, could have some issues.

If you’re using dnsjnio in a very high volume environment, I’d recommend upgrading to the latest version.

I’ve been asked why Dnsruby is hosted on Rubyforge; asked often enough to write a note here!

I could have hosted Dnsruby on Nominet’s servers (perhaps using TRAC and subversion). However, if I were looking for a useful open-source library, I’d prefer to see it hosted on a well-known public repository. That way, I’d know what I could expected from the hosting site (fora, trackers, source repositories, downloads, etc.). I’d also know that the information which had been published to that site already, would stay on that site in the future. I’d have less confidence about a privately hosted project.

These reasons are enough for me to host dnsjnio on sourceforge (indeed, the Apache JAMES SPF folks were unable to use the project until it was moved to sourceforge from Nominet’s site).

However, for Dnsruby, there is an additional advantage to using Rubyforge - simple interaction with gems. I can publish a new release of my project to rubyforge, and within an hour or two, anyone can type “gem install dnsruby” and have the latest version installed to their machine.

The downside of not using Nominet to host these projects is that it’s not always obvious what great open-source projects we produce!

At the risk of spamming this blog with updates on dnsjnio, I’d like to announce that version 1.0.0 of dnsjnio has now been released.

The changes include :

o UDP single-port race conditions fixed.
o Single-port operation removed for UDP.
o TCP single-port and multiple-port race conditions fixed.
o Retry behaviour in ExtendedNonblockingResolver now doubles timeout for each retry.
o Port randomisation is now done by dnsjnio, rather than left to the host O/S.
o Different QID now used for each query in round-robin or timeouts.
o Timeout semantics changed slightly in SingleResolver.sendAsync().
o Redundant error message removed.
o Test code fixed.

If anyone finds any issues with this code, I’d be very keen to hear about them.

Hopefully, I’ll now be able to keep quiet about dnsjnio for a while!

Well, I suppose we can all be too hasty. It doesn’t stop the embarrassment, though…

After Dan Kaminsky reported his DNS vulnerability, it seemed that port randomisation was A Good Thing. So, I removed the useSinglePort feature from dnsjnio (even though it had been one of the original selling points!).

Then, a potential new dnsjnio user downloaded the beta release, and found that the demo (DemoClient.java) broke horribly. This had been my second mistake - not to test the demo code along with the test code in my automated tests. It had been a long time since I looked at the demo code; it showed how to use dnsjnio to perform 50,000 concurrent lookups over a single TCP port (not a conventional use of the DNS, but still a valid one). Of course, since I’d removed the ability to use a single TCP port (my first mistake), this resulted in the poor demo code attempting to run 50,000 queries on a port each!

I’ve now reinstated the useSinglePort feature for dnsjnio, but have only enabled it for TCP. If you try to use it for UDP, it will be ignored, and a new, dnsjnio-chosen random port will be used. I guess I should briefly note that a TCP session will be continued for as long as there are outstanding queries. It will be closed when all queries are complete, and a new, random port will be used for the next query session.

Please let me know if you see a problem with this! I’d quite like to put this issue to rest…

The JUnit 4.5 release includes a new standard test runner, BlockJUnit4ClassRunner, which is part of the published API and is designed to be an extension point for building your own custom test runners.

The flow for executing a particular test method is built up by chaining a sequence of Statements, as seen by inspecting the source code for BlockJUnit4ClassRunner:

protected Statement methodBlock(FrameworkMethod method) {
  Object test;
  try {
    test= new ReflectiveCallable() {
      @Override
      protected Object runReflectiveCall() throws Throwable {
        return createTest();
      }
    }.run();
  } catch (Throwable e) {
    return new Fail(e);
  }
 
  Statement statement= methodInvoker(method, test);
  statement= possiblyExpectingExceptions(method, test, statement);
  statement= withPotentialTimeout(method, test, statement);
  statement= withBefores(method, test, statement);
  statement= withAfters(method, test, statement);
  return statement;
}

The statement returned by methodBlock is evaluated for each test method in the test class. The statement base class is straightforward, with a single evaluate method to implement:

public abstract class Statement {
  public abstract void evaluate() throws Throwable;
}

This makes it relatively easy to hook into a particular point in the execution flow to introduce custom behaviour. As part of our upgrade to 4.5, I needed to port a test runner which executed each test method within a transaction which is rolled back on completion. By overriding withAfters, I can obtain the statement which represents the standard JUnit execution flow:

protected Statement withAfters(FrameworkMethod method,
  Object test, Statement statement) {
  statement = super.withAfters(method, test, statement);
  // Evaluate the standard junit statement within a spring transaction
  statement = withinTransaction(test, statement);
  // Evaluate before and after transaction annotations outside transaction
  statement = withBeforeTransactions(method, test, statement);
  statement = withAfterTransactions(method, test, statement);
  return statement;
}

I then create a new Statement which wraps the standard flow in a Spring managed transaction:

private Statement withinTransaction(final Object target, 
  final Statement statement) {
  return new Statement() {
    public void evaluate() throws Throwable {
      PlatformTransactionManager txManager;
 
      try {
        txManager = ((HasTransactionManager) target).getTxManager();
      } catch (ClassCastException e){
        throw new RuntimeException("Test class must implement HasTransactionManager", e);
      }
 
      new TransactionTemplate(txManager, new DefaultTransactionDefinition(
      TransactionDefinition.PROPAGATION_NESTED)).execute(
      new TransactionCallbackWithoutResult()
      {
        public void doInTransactionWithoutResult(TransactionStatus transactionStatus)
        {
          try {
            // evaluate the standard flow within a transaction
            statement.evaluate();
          } catch (Throwable throwable) {
            throw new RuntimeException(throwable);
          }
          finally {
            // Force a transaction rollback
            transactionStatus.setRollbackOnly();
          }
        }
      });
    }
  };
}

Finally I add in some hooks which mirror the @Before and @After annotations to run methods outside the transaction scope:

protected Statement withBeforeTransactions(FrameworkMethod method, 
  Object target, Statement statement) {
  return new RunBefores(statement, 
    getTestClass().getAnnotatedMethods(BeforeTransaction.class), target);
}
 
protected Statement withAfterTransactions(FrameworkMethod method, 
  Object target, Statement statement) {
  return new RunAfters(statement,
    getTestClass().getAnnotatedMethods(AfterTransaction.class), target);
}

It was quite easy to figure out how to write a custom test runner by examining the source code for the provided statements, which I think reflects well on the new design.

I’m currently in the process of adding some reports based on data stored in a database. With annotated configuration in Spring 2.5 and the flexibility of JFreeChart, I was surprised at what can be achieved with such little effort.

The new features added to spring-mvc provide a nice way to avoid having to manage large configuration files and meant that I could concentrate on writing the code for my charts. I’ve added a quick example after the jump

Continue Reading »

As a follow-up to my previous post about a race condition in dnsjnio, I thought I’d explain the issues that arose, and what has been done to fix them. These fixes are now in beta, so if you’d like to help test them, please let me know soon!

Basically, I found two bugs in the 0.9.9 release of dnsjnio; both were related to a race condition between the timer thread and the reception of data. The first bug was pretty much as reported in my previous post. The timer thread would fire a timeout for a query. In the meantime, data has been received, but the client has not yet been notified. The connection is being closed as the timer fires. When the timer thread comes to close the connection, it did not check that the connection was still good, so a NullPointerException was generated when it attemped to close the closed connection. This has been fixed by checking the status of the connection, and returning a boolean to the timer thread indicating the status. If the connection has been closed, then the timer thread takes no more action (otherwise, a timeout event would be sent!).

The other bug was similar, but the other way round - the data arrival occurs just after the timeout. This resulted in two client events; both a timeout, and a data receipt! This case was more difficult to fix - although I could make sure that the connection was properly closed, it was difficult to signal this back to the select thread (the Thread : void run() interface is used). In the end, I decided to put some logic in the query Transaction class itself, that enforces the condition that only one result must be generated for each query. In this case, whichever event fires first has the priority.

Note that both of these bugs would only show up if the query timeout value was very close to the normal query time. If a long timeout value was used (say, 10 seconds or more), these bugs would be unlikely to show up. In order to reproduce them (a requirement to prove a fix!), I had to write some pretty pathological test code (which took some time). In normal dnsjnio usage (as I imagine it), a single client thread makes many query requests to a single dnsjnio select thread (an additional timer thread is also present). This client thread uses the ResponseQueue interface to pick up query responses as they return. To reproduce the bug, I had to have hundreds of client threads, each performing sequential queries (using the original dnsjava thread-intensive callback pattern), with timeouts starting at 500ms. Still, even this case now works with no problems.

Please let me know if you have any issues with this latest beta release - the latest test version can be found at :

https://dnsjnio.svn.sourceforge.net/svnroot/dnsjnio/tags/dnsjnio-1.0.0-beta/dnsjnio-1.0.0-beta-1.jar