techblog 

Mutation testing is a technique for checking how good your unit tests are.  It mutates a class by for example swapping a subtraction for an addition or by negating an if statement.  It then runs the unit tests for that class.  If none of them fail, then maybe your tests are not good enough.

We took a look at Java mutation testing tools some time ago as a possible addition to our continuous integration system.  But at that time the tools fell short.  Jester works by mutating the source files.  We found that this was just too slow with all the compilation it needs to do.  Jumble is a bit smarter in that it mutates the bytecode of the compiled class files.  However, when we evaluated it only JUnit 3 tests were supported and we were already well on the way to transitioning most of our tests to JUnit 4.

Recently though, Jumble was modified to work with JUnit 4 tests, so I thought it was time to take another look at it.  I found it quite tricky to get it working as there is no ant integration and it only runs as a command line application.  However, with a bit of persuasion I managed to get it running over our codebase using ant.  The details of how I did this are given below, but I think that a better solution would definitely be a fully fledged custom ant task.

So how did I get on?  Initially I had some niggly classloader problems. It seems that you need to tell Jumble’s mutating classloader to defer the loading of various sets of classes to the default classloader.  I needed to do this for the JMX classes found under javax.management by specifying the command line flag --defer-class=javax.management. Once I’d done this I had it working and did indeed find some interesting things.  I found tests that had been cut-and-pasted and not changed to test what they claimed to test. I found some test data that wasn’t up to the job and I found an actual bug in the code.

However, I hit a roadblock once the code under test used a database.  For some reason Oracle’s JDBC driver would not behave. Even before any mutations were applied it would insist on trying to connect with a null password.  This meant that it tried a number of times before locking itself out of the database.  I assume that this is some kind of classloader thing, but it seems strange that it manages to successfully contact the database only to completely mess up the credentials.  I’ve contacted the developers of Jumble (who are based in New Zealand incidentally), but no solution has been forthcoming as yet.  Until this problem is fixed we won’t be able to add this to our continuous integration system, which is a shame, as I think it could be a useful tool.

So, how did I integrate Jumble into ant?  I used the follow macrodef to run Jumble:

<macrodef name="do-mutation">
    <attribute name="class-to-mutate"/>
    <sequential>

        <!-- Use this trick to convert a reference to a classpath to classpath as a string -->
        <property name="the-classpath-as-a-string" refid="execute-test-classpath"/>        

        <java dir="${base.directory}" classname="com.reeltwo.jumble.Jumble" fork="true">

            <!-- all we really need in this classpath is the jumble library -->
            <classpath refid="ants-own-classpath"/>

            <!-- but then it needs everything in the JVM it forks: -->
            <arg value="--classpath=${the-classpath-as-a-string}"/>
            <arg value="--exclude=equals,hashCode,toString"/>
            <arg value="--defer-class=javax.management."/>
            <arg value="@{class-to-mutate}" />
        </java>

    </sequential>

</macrodef>

To get this to work you will need the classpath used to run the tests set up with the id execute-test-classpath and the classpath used by ant to find custom tasks set up with the id ants-own-classpath. The latter will need to include the jumble jar. As you can see I have excluded the equals(), hashcode() and toString() methods from being mutated as the first two are often generated by the IDE anyway. In the case of toString, this rarely contains important logic.  As mentioned before, the JMX classes are deferred to the default classloader. You may need to add some other packages to get it to work in your environment.

The above macrodef will mutate a single class. But we want to run this across our whole code base. To do this I had to resort to using some further ant tricks in the shape of the ant-contrib library which adds additional functionality to ant. As I said before, a better solution would be to write a proper ant task. Here is how the macrodef is called to run Jumble over a whole project:

<target name="mutation.test">    

    <!-- The ant contrib jar contains the "for" task -->
    <taskdef resource="net/sf/antcontrib/antlib.xml" classpathref="ants-own-classpath"/>

    <!-- Strip off the leading directory and the .class. Then replace the slashes
         with dots and separate each one with a comma.
         Results go into the classlist property -->

    <pathconvert dirsep="." pathsep="," property="classlist">
        <mapper type="glob" from="${build.dir}/*.class" to="*"/>
        <fileset refid="classes-to-mutate"/>
    </pathconvert>

    <!-- Iterate through the comma separated list and call jumble -->
    <for list="${classlist}" param="class">
        <sequential>
            <do-mutation class-to-mutate="@{class}"/>
        </sequential>
    </for>

</target>

This takes a fileset with id classes-to-mutate which consists of classes under the directory ${build.dir}. It turns the path into a package and removes the .class from the end to get a list of classes to mutate. (NB This was written to work with Unix style paths, it may need alteration to work under Windows). Then the macrodef given previously is called for each. Note that we refer to the ants-own-classpath classpath again which must contain the ant contrib jar this time.  The code given above was put in a standard ant build file included by other projects. The fileset to mutate could then be defined in each like this:

<fileset id="classes-to-mutate" dir="${build.dir}" includes="**/*.class">
        <exclude name="**/WeWantToLeaveThisOut.class"/>
</fileset>

UPDATE: The story gets even weirder. One of the Jumble developers got back to me with a suggestion for how to fix the Oracle problem, which was to add the JDBC driver to the list of classes deferred to the parent classloader. This didn’t help, but I then discovered bizarrely that the JDBC problem goes away if you are connecting to an 11g database as opposed to a 10g one. So it means that somehow the 10g JDBC driver fails to connect to 10g when run by Jumble, but succeeds against a later version. Curiouser and curiouser…