Reviewing the NetBeans Unit Tests Code Coverage Plugin

The NetBeans Unit Tests Code Coverage Plugin has been around for several versions of NetBeans. It measures code coverage statistics and displays annotated & highlighted lines in the Source Editor in each class that were executed by unit tests. In a recent release of the plugin, it also provides a report that shows overall, package, and class-level statistics. This article provides a quick overview of code coverage, why it is important, and what the NetBeans Unit Tests Code Coverage Plugin provides. I’ll also cover some pros/cons of the plugin.

Overview of Code Coverage

To quote myself from Chapter 16, Using Code Coverage Tools, of my book, Pro NetBeans IDE 5.5 Enterprise Edition (Apress, March 2007) :

You know that it is important to write accurate and effective tests for your code. But how do you measure the effectiveness of your tests? In a perfect world, you would write a test for every class, every method, and every line of code. Depending on the complexity of your code base, this may be easy or extremely difficult.

How do you know when you’ve written enough tests? If you have 200 tests for your code, and they all pass, then you’re finished, right? Not necessarily. You’re finished writing tests only when you know for a fact how effective your tests are. Since testing often involves visual analysis by programmers and code-based test cases, it’s difficult to know if 100 percent of your code has been tested. This is where code coverage tools come into play.

The first and obvious benefit of using a code coverage tool is being able to measure the coverage of your test cases (and, by association, the test case effectiveness). Untested code can lead to bugs, and bugs lead to many other problems.

Code coverage tools also help you identify areas in your code that are dead or unreachable. Suppose you have written test cases for all the public methods in your code. If your code coverage tool identifies one or more methods with private-level access that have never been executed, then you may be able to remove those methods from your source code. You can then use the NetBeans Safely Delete refactoring to check the method and make sure it is not called by any other code.

A good code coverage tool not only tracks if each line in each method was executed, but also how many times each line was executed. Using this data and the knowledge of what lines were and were not executed, you can understand the flow of program functionality and rearrange code blocks accordingly.

The NetBeans Unit Tests Code Coverage Plugin does not provide access to some of the information mentioned above (such as number of times each line was executed). However, it provides enough of the basic features you will need to be quite useful. Internally, the plugin depends on the Emma code coverage library.

Code Coverage tools like Emma and Cobertura (one of my personal favorites) typically work by instrumenting classes (inserting extra byte codes). The instrumented classes are then executed instead of the original un-instrumented classes. The inserted byte codes in the instrumented classes then collect different bits of data and deposit them in a file for later analysis.

Getting Started

As of the official release of NetBeans 6.1 Beta, the update centers contained a slightly older version of the code coverage plugin. I would suggest going to the official NetBeans code coverage plugin site and download it using the prominently displayed button labeled “Download Code Coverage Plugin”.

Use the Plugins Manager to install the NBM file. Go to the Downloaded tab and click the Add Plugins button. Accept the license, click Next a few times, and you should be all set.

Once installed, you need to activate code coverage on a per-project basis. For this article, I have a standard J2SE Java Library project. In the Projects window, right-click the project name, and you should see a Coverage submenu. It contains the following options :

Activate Coverage Collection: When clicked activates code coverage collection for the selected project. This will also generate the file nbproject/coverage.properties and coverage/emmascript.xml, which are covered below.

Deactivate Coverage Collection: When clicked deactivates code coverage collection.

Show Project Coverage Statistics: When clicked displays a tab in the Source Editor showing the accumulated statistics collected by the plugin for the selected project.

Files Generated By Activating Coverage Collection

As previously mentioned, several files are generated by activating coverage collection.

The nbproject.properties file contains several parameters used by the plugin such as :

coverage.activity=ON
project.type=java
coverage.templateFile=coverage/template.emma
coverage.coverageFile=coverage/coverage.emma

The first parameter obviously indicates whether the coverage collection is active or inactive. The second parameter identifies the type of NetBeans project, and the last 2 parameters identify the paths to specific files used by the plugin.

The coverage/emmascript.xml file contains Ant targets for executing the instrumentation of the project class files.

<?xml version=”1.0″ encoding=”UTF-8″?>
<project basedir=”.” default=”echoit” name=”Coverage Tasks”>

<target name=”echoit” description=”test target”>
<echo message=”Test succeeded.”/>
</target>

<target name=”instr” description=”Instrumenting jars”>
<echo message=”Instrumenting started.”/>

<java classname=”emma” fork=”true”>
<classpath >
<pathelement location=”${emma}”/>
</classpath>
<arg line=”instr -verbose -m overwrite -cp ‘${jarfiles}’ -outdir ‘${output.dir}’ -outfile ‘${output.dir}/template.emma’”/>
</java>
<echo message=”Instrumenting done.”/>
</target>

</project>

The instr target runs the main emma class passing it several arguments to overwrite the application JAR file with a JAR file of the project’s instrumented classes.

Creating Sample Code

First, create a simple Java Class like this :

public class StringUtils {

public static String tryIntToString(int numLoops) {

StringBuffer sb = new StringBuffer();

for (int i = 0; i < numLoops; i++) {
sb.append(String.valueOf(i));
}
return sb.toString();
}
}

It is basically a ‘hello world’ class that takes in a counter variable, numLoops, instantiates a StringBuffer, and loops appending an int converted to a String into the StringBuffer. The method then returns the value of the StringBuffer.

To test this method, create a JUnit test. Right-click the class listing in the Projects window and select Tools >> Create JUnit Tests (or press Control+Shift+U). Select your preferred JUnit version (4.x in this case) and the various other JUnit settings you are prompted for. Once created, I implement a basic test case for the StringUtils class passing in a value and setting the expected return.

public class StringUtilsTest {

public StringUtilsTest() {
}

@BeforeClass
public static void setUpClass() throws Exception {
}

@AfterClass
public static void tearDownClass() throws Exception {
}

@Before
public void setUp() {
}

@After
public void tearDown() {
}

@Test
public void testTryIntToString() {
System.out.println(“tryIntToString03″);

int numLoops = 10000;

String result = StringUtils.tryIntToString(numLoops);

int expectedResult = 38890;

assertEquals(expectedResult, result.length());
}
}

As written, the testTryIntToString method in the JUnit test should pass. When you execute the tests (or trigger Test Project on the Run menu), you should see the JUnit Test Results window displayed, as shown in Figure 1.

NetBeans JUnit Test Results window
Figure 1. The JUnit Test Results window.

Once you are positive that your test passes and everything is working as expected it is time to activate the coverage collection.

Collecting Coverage Statistics

To activate code coverage, right-click the project name in the Projects window and select Coverage >> Activate Coverage Collection. You won’t really see any visual acknowledgement. Next, re-run your JUnit tests. The JUnit Test Results window should list the passed results, but this time, the right pane in the window will list output similar to:

EMMA: collecting runtime coverage data …

EMMA: runtime coverage data merged into [C:projectsCodeCoverageTestingcoverage.ec] {in 50 ms}

Next, open the StringUtils class in the Source Editor. It should contain highlighted lines for each line of code that was executed by your tests and measured by Emma, as shown in Figure 2.


Figure 2. The highlighted lines in the StringUtils class.

Notice that the lines that display the package statement and the class definition are not highlighted. Obviously, these lines are not truly executed so they are not highlighted. Sometimes you will also see lines that contain individual { and } that are not executed. The coverage statistics are not generally perfect and depending on code formatting and other semantics, may not pick up all executed lines.

As you go through and modify your code, you go re-run the test cases and watch the highlights in your source code change. This can help you identify sections in your code that are not being tested, and thus may contain potential bugs.

For example, modify the tryIntToString method like the following:

public static String tryIntToString(int numLoops) {

StringBuffer sb = new StringBuffer(numLoops);

if (numLoops > 10000) {
sb.append(“greater than 10000″);
} else {
for (int i = 0; i < numLoops; i++) {
sb.append(String.valueOf(i));
}
}

return sb.toString();
}

Now, the method has an if-else conditional branch. The statement inside the if clause should never execute using ONLY the JUnit test written to date, since it passes in the value of 10000. Compile the modified class and re-run the JUnit test. Look at the StringUtils class in the Source Editor. The executed lines will be highlighted green, but the line inside the if clause will not, as shown in Figure 3.


Figure 3. The highlighted lines of the modified StringUtils class.

Deactivating Coverage Collection

It is quite simple to deactivate code coverage for the project. Simply right-click the project name in the Projects window and select Coverage >> Deactivate Coverage Collection.

The property coverage.activity in the nbproject/coverage.properties file will now be set to OFF.

Viewing Project Coverage Statistics

The newest feature of the NetBeans Unit Tests Code Coverage plugin is the ability to view coverage statistics. To view coverage statistics you must have coverage collection activated and have executed at least one test. To view the report, right-click the project name in the Projects window and select Coverage >> Show Project Coverage Statistics. The coverage report will display, as shown in Figure 4.


Figure 4. Coverage report.

The report lists the number (and percentage) or packages, classes, and lines “covered” or executed at least once. The middle and bottom of the report also displays specific information for each package and class. You can see the com.pronetbeans.samples.StringUtils class is 71% covered, or 5 out of 7 lines.

The report can be very convenient in that you can easily look across packages and even hundreds of classes to see which ones have a low percentage of coverage. You then have some knowledge of where to look to start writing additional tests to test out your code.

Good luck and happy testing.

Resources :

NetBeans Unit Tests Code Coverage Plugin : http://codecoverage.netbeans.org/

Emma Code Coverage Library : http://emma.sourceforge.net/

Cobertura Code Coverage Library : http://cobertura.sourceforge.net/

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>