techblog 

After discussing code coverage with several colleagues recently, and the use of various tools (Clover, Jester - more test validity/mutation testing than coverage), I have since found out that my personal editor of choice, IntelliJ Idea has had some built in support for running code coverage (on a line by line basis) for unit tests using its own built in engine (based on the open source EMMA toolkit). This isn’t without its issues however.

For the uninitiated, Code coverage refers to a software engineering technique whereby you track quality and comprehensiveness of your tests by determining simple metrics like the percentage of {classes, methods, lines, etc} that got executed when the test suite ran. In general, line coverage answers the question, “Was this line of code executed during simulation?” (which is what IntelliJ attempts to show).
You can access code coverage by selecting Run | Edit Configurations (Shift + Alt + F10) through the Run/Debug Configurations dialog (where you can enable and configure).

Code coverage is automatically measured when running/debugging. Statistics on the lines covered by your unit tests are displayed in the Project View by selecting View-> Show Coverage Information or pressing Ctrl+Alt+F6 ( if your keys are set up using the default keymap - Apple key+Alt+F6 on a Mac).

Coverage is given both as a percentage against individual files listed in the Project view, and also visually alongside the code using a traffic light key (green = covered, red = not covered, yellow = partial coverage).

The integration of this feature within the IntelliJ development environment is supposed to provide two key benefits:

• low overhead - because IntelliJ knows the structure of your project, it can calculate statistics based on compiled code, allowing for low run-time overhead as compared with using an external tool.
• On the fly visual coverage info - by comparing new code against a version of the file when coverage was last run, the traffic light indicators in the editor window automatically change as you type. The project view statistics are only updated when you run coverage (run/debug).

At first glance, this looks like an excellent way of making sure you are writing sufficient tests for your code. In practice however, I found that if you selected IntelliJs “merge data with previous results” option, it cached the results after the first run, and I couldn’t get them to update subsequently. Also, if you selected a single class file to record coverage data against (as opposed to a package), you wouldn’t get coverage statistics displayed in the project view. I also found the on the fly updating of traffic light indicators to not work.
There is also an known unresolved issue which I experienced that munges the output like

EMMA: collecting runtime coverage data ...
EMMA: runtime controller started on port [47653]
raw session data merged into [/sers/lpower/ibrary/aches/ntelliJIDEA60/overage/est$FormValidatorTest.es] {in 7 ms}"

…knobbling the first letters of directories in path - this is apparently on the display of the path only.

So all in all, it seems to work, but in a limited respect.

Of course if you were doing TDD (test driven development - i.e. writing tests FIRST) then there wouldn’t be so much of a need for tools like this, but they do act as a valuable sanity check.

Commercial tools like Clover look like they cover more feature-wise and come with fully integrated plugins that hook into IntelliJ and other IDEs. If you do want reporting, you could always try EMMA on its own - its free and open source, and apparently supports several report types: (plain text, HTML, XML). All report types support drill-down, to a user-controlled detail depth. The HTML report even supports source code linking (see EMMA sample report for an example).

When one is working on a project that are a product of several developers, unit test coverage is always likely to vary based on personal development practices, so running code coverage tools looks to be a good way to see quite what is being tested and what is/isn’t partially covered.

In my opinion, the use of metrics should be seen as a useful tool and pointer to guide improving tests for developers, rather than the be-all-and-end-all of test reporting used by non-technical management. It would be easy for someone not familiar with using such a tool to assume that 100% coverage is the ideal, when in fact it is the opposite - you get a lot less quality improvement for the cost of time spent to achieve such “perfection”, and relying only on such data could lead one astray.

The metric isn’t that important - an experienced developer will look at the generated coverage statistics for pointers, drill down a bit into their code, look at the “red” areas, and figure out which, if any, areas of the product were left somewhat under-tested.

Front end developers using anything in addition to Java, such as a framework (which may have functionality coded in its own idioms), must also make sure testing takes into account this additional functionality, which would not be reported by code coverage tools.

For further information on IntelliJ buit-in code coverage and some nice screenshots see IntelliJ’s blog and EMMA on Sourceforge.