The Disco Blog

Ever run a build that lasted 4 hours? Yeah, probably once, right? After that, you figured out how to merely compile your own hip stuff so you could actually get some work done and not have to wait around for the entire build saga to complete.

Then everyone else figured out how to do the same fab thing. Unfortunately, one day before the big release (and before you installed a CI system), someone actually ran the complete build and lo and behold, tests started failing (2 hours into the build, however).

Unless a build compiles millions (and millions) of files, the culprit of a prolonged build is usually the testing step (which could be a series of steps!). This aggregate time to run a series of tests has the additional tendency to become longer when there are extensive setup steps too, such as configuring a database, or deploying a .war file, to name a few.

For any nontrivial software project that wishes to keep build times bearable, it becomes paramount then to create an effective strategy for the categorization of tests. By segregating tests into categories and running the associated categories at prescribed intervals, build times can remain manageable for developers and CI systems, alike.

A true unit test should run to completion (successfully) in less than a few seconds. If a unit test takes longer, take a close look at it- it is either broken or really a component level test. Because unit tests run so quickly, they should be run anytime a build is run.

The whole groovy mantra of “code a little, test a little, code a little…” is predicated on the notion of rapid testing. If unit testing Bogarts enough time that someone can focus on something else, it’s taking too long! It will become a burden and soon forgotten.

In a CI environment, builds are run anytime someone changes a source repository; therefore, unit tests should be run anytime someone checks in code. There is no configuration cost and the resource consumption cost to run them is negligible.

Component tests, which usually have multiple dependencies, take a bit longer to run. As such, they should be run at regular intervals, but not necessarily every time a build is run, man. These tests have a cost to them- dependencies have to be put in place and configured; moreover, these tests alone may only take a few seconds, however, in the aggregate, this time adds up.

For example, the trippn’ component test below takes, on average, 4 seconds to run.

package test.org.aglover.words.dao.spring;

import java.util.Set;
import junit.framework.TestCase;
import org.aglover.words.bizobj.IWord;
import org.aglover.words.dao.WordDAO;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import com.vanward.resource.unittest.dbunit.hibernate.DefDBUnitTestCase;

public class SpringWordDAOImplTest extends DefDBUnitTestCase {

  public void testFindVerifyDefinitionsSize() throws Exception{
    ApplicationContext context =
        new ClassPathXmlApplicationContext("spring-config.xml");

    WordDAO dao = (WordDAO) context.getBean("wordDAO");
    IWord wrd = dao.findWord("pugnacious");
    Set defs = wrd.getDefinitions();
    TestCase.assertEquals("size should be one", 1, defs.size());
  }

  public SpringWordDAOImplTest(String arg0) {
    super(arg0);
  }

  protected String getDBUnitDataSetFileForSetUp() {
    return "words-seed.xml";
  }
}

This test does a number of things which add to the total test time and configuration complexity. First, this test seeds a database via DbUnit. In this case, DbUnit does an insert of the data found in the xml file words-seed.xml, which also implies that XML parsing is being done. This also assumes the test case can find the file easily.

Next, this test case configures Spring, which also configures Hibernate. Finally, a test is run and a word is retrieved from the database. Any wonder why this takes 4 seconds? Keep in mind, this is only one test case. Each additional test case in this class may not add another 4 seconds, however, it’ll probably add about 2 seconds. Do this about 10 more times and you have a minute. Get the picture?

While these types of tests shouldn’t be executed every time a build is run, it’s a good bet to run them before committing code into a repository. In a CI environment, it’s probably a good idea to run these at least a few times a day. Running them every time someone checks in code could cause issues; however, this takes good judgment. Some projects can get away with running component level tests in a CI environment anytime someone checks in code.

System and functional tests, which require a fully installed system, take the longest to run. Additionally, the complexity in configuring a fully functional system occasionally prohibits the full automation of these tests.

Ideally, developers can run these tests locally, when needed. In a CI environment, nightly (if they can be pulled off in an automated fashion) is a good bet with these tests. Running these tests frequently in a CI environment could a recipe for disaster and probably overkill if other tests are run often. Sometimes at the end of release cycles though, these types of tests can be run on a more frequent interval.

The next time you blindly add a test case to your build, consider the long term implications of running all of your tests and then start optimizing your build to categorize your tests so you can run them at varying frequencies. Everyone will thank you, especially the disco dancers in the crowd!

Trackback this Post | Feed on comments to this Post