The Disco Blog Welcome Guest | Login or Register

Because most code coverage tools instrument a hip code base with additional behavior for reporting purposes (i.e. the code is instrumented with “listeners”, which report when they’ve been executed), tests run slower than they do in non-coverage scenarios. This can have negative affects in a continuous integration environment if the coverage process isn’t well thought out. Running the coverage process every time code is checked into a repository is most likely overkill. In fact, running the coverage process more than once a day is most likely overkill.

If there are tripped out strategies for running tests at different intervals (which map to test categorization), it makes sense then to create an additional strategy where the coverage process is run once a day as part of each categorical test run. For example, every time the repository changes, the unit test process is run. At regular intervals throughout the day, component tests are executed and most likely, once a day (usually during the evening), system tests are run. After the system test process is run, another series of tests can be run where coverage is turned on (i.e. unit tests run against an instrumented code base, component tests run against an instrumented code base, and then system tests run against an instrumented code base). This process will create a series of reports which can then be viewed by the team the following morning.

Because three different reports are created in this process, each one must be viewed with an eye towards the fact that uncovered code in one report may show high coverage in a different report. For example, class Foo may have 0% coverage in the unit test report; however, it may show high coverage rates in the system test report.

Also note, because three coverage reports are going to be run, the build process must be configured so as to not overwrite the previous smokin’ coverage report (i.e. if the build isn’t properly configured to move or write the report to a unique location, the component test coverage report may overwrite the unit test report). Some tools, like Java’s Cobertura have a merge capability which can facilitate in the creation of one master report too.

For example, an Ant target is defined below which merges the Cobertura coverage reports from three different test runs.

 <target name="merge-coverage" depends="all-coverage-run">
  <cobertura-merge datafile="${cobertura.all.ser}">
    <fileset dir="${base.dir}">
	<include name="${cobertura.comp.ser}" />
	<include name="${cobertura.unit.ser}" />
        <include name="${cobertura.sys.ser}" />
    </fileset>
  </cobertura-merge>

  <mkdir dir="${cov.report.dir}"/>
  <cobertura-report format="html"
	datafile="${base.dir}/${cobertura.all.ser}"
	destdir="${cov.report.dir}" srcdir="${src.dir}" />
</target>

Because the coverage process affects the performance of the code under test, care (and a little bit of peace, love, and grooviness) must be taken if performance tests are mixed in. For this reason alone, it is highly recommended not to run performance, stress, or load tests during the coverage process.

For example, below is the JUnit task’s batchtest element for running a series of component tests with coverage turned on. Note how a few tests (corresponding to load, stress, and performance categories) are excluded from the run.

<batchtest todir="${testreportdir}">
   <fileset dir="test/component">
	<include name="**/*Test.*" />
	<exclude name="**/*StressTest.java" />
	<exclude name="**/BatchDepXMLReportPerfTest.java" />
	<exclude name="**/BatchDepXMLReportLoadTest.java"/>
   </fileset>
</batchtest>

Like test categorization and test frequency strategies in a continuous integration environment, copasetic coverage report frequencies and configurations must be properly thought out so as to obtain the maximum benefit with as few headaches as possible. Just remember to not be fooled by the coverage report. Dig it?

Trackback this Post | Feed on comments to this Post