The Disco Blog

Categorizing developer tests into three respective disco buckets (unit tests, component tests, and system tests) has efficiency benefits when it comes to Continuous Integration. For example, running system tests every time the repository changes is a time and resource consuming task that may not be worth the effort. Why not run unit tests every time some one checks code in as they are cheap to run, then schedule periodic intervals to run component tests and then another interval for system tests? Those intervals can be increased as iterations come to a closing and elongated at the initial stages too.

Because it’s their bag, frameworks like NUnit for .NET and JUnit 4.0 and TestNG for Java have annotations, which make categorizing tests quite easy to implement; however, in other frameworks, segregating tests is a bit more challenging.

For example, with pre 4.0 JUnit versions, there is no copasetic mechanism within the framework itself or within Ant to easily divide tests up into three groups. This can be achieved, however, with a simple naming scheme or, even easier, with a hip directory strategy.

One best practice for developer testing is to place unit tests in a separate directory than that of the source code. For example, a project directory structure would have a src folder for the source code and a test folder for associated tests.

A sample project would have a root directory like this:

$ ls -lt ./
total 62
drwx------+ 6   Andy.Glover usrs      0 Feb 19 21:37 test
drwx------+ 4   Andy.Glover usrs      0 Dec 27 21:20 src
-rwx------+  1  Andy.Glover usrs      32452 Nov 15 17:51 build.xml
-rwx------+  1  Andy.Glover usrs      260 Aug 30  2005 build.properties

The src directory would obviously contain directories which map to source code packages, man.

With the test directory, categorization is possible by creating three additional internal directories: unit, component, and system. For example, the directory listing would look like this:

$ ls -ltr ./test
total 0
drwx------+ 4 Andy.Glover usrs       0 Mar  1 22:09 unit
drwx------+ 2 Andy.Glover usrs       0 Mar  1 22:09 conf
drwx------+ 4 Andy.Glover usrs       0 Mar  1 22:11 component
drwx------+ 3 Andy.Glover usrs       0 Mar  1 22:12 system

There is a unit, component, and system directory listed above, which would then contain the associated tests for each category. Note, the conf directory would hold associated properties files, etc required for testing.

The unit directory, for example, would have a directory structure which maps to the unit tests’ package names (which usually maps to the corresponding class under tests’ packages) like this:

$ ls -ltr ./test/unit/test/com/van/sedna/frmwrk/filter/
total 12
-rwx------+ 1 Andy.Glover usrs  1190 Oct 25  2004  SimFilterTest.java
-rwx------+ 1 Andy.Glover usrs  2708 Oct 25  2004  RegexFilterTest.java
-rwx------+ 1 Andy.Glover usrs  1678 Nov 20 17:30  ClassFilterTest.java

Now that the tests are segregated into separate directories, your chosen build system needs a trippin’ update. In the case of Ant, three targets are created. One for running unit tests, another for running component tests and another for running those system tests. A fourth target could also be created which calls all three of the previous targets so as to run the entire test suite.

For example, for system tests, the all too familiar batchtest element of the JUnit task would look something like this:

 <batchtest todir="${testreportdir}">
    <fileset dir="test/system">
      <include name="**/*Test.*">
      </include>
    </fileset>
 </batchtest>

Note how there isn’t any special naming scheme going on here- tests are still appended with a Test regardless of granularity.

If the target for running the system tests was named test-system, then running it is as simple as typing ant test-system (don’t forget to set up the associated depends clause for compiling, deploying, etc).

Implementing a categorization strategy for developer tests is fairly easy, so long as the team commits to a common mechanism. Running these categories at various intervals within a CI strategy then becomes a simple matter of calling the proper build target. Dig it?

Trackback this Post | Feed on comments to this Post