The Disco Blog

I often find that people use the term “unit test” rather broadly. This can cause confusion, especially when people start claiming their unit tests “take too long to run”. Defining a common vocabulary for developer tests can assist in categorizing them into neat-o groups, which can make all the difference in creating an effective continuous integration process.

Unit Tests- Unit tests verify the behavior of small elements in a software system, which are most often hip single classes. Occasionally though, the one to one relationship between a unit test and a class is slightly augmented with additional classes because the classes under test are tightly coupled. Because of this small issue, it can be helpful to further segregate unit tests into two types: isolated unit tests and semi-isolated unit tests.

The key aspect, however, is that unit tests (regardless of isolation) do not rely on outside dependencies such as databases, which have the tendency to increase the amount of time it takes to set up and run tests.

Unit tests can be created and run early in the development cycle (like day one); furthermore, because of the rapid time between coding and testing the results, unit tests are an extremely efficient way of debugging.

Because it’s my bag, below is an example unit test written in Ruby, which verifies the behavior of a filtering type.

require "test/unit"
require "filters"

class FiltersTest < Test::Unit::TestCase

  def test_regex
    fltr = RegexFilter.new(/Google|Amazon/)
    assert(fltr.apply_filter("Google"))
  end

  def test_simple
    fltr = SimpleFilter.new("oo")
    assert(fltr.apply_filter("google"))
  end

  def test_filters
    fltrs = [SimpleFilter.new("oo"), RegexFilter.new(/Go+gle/)]
    fltrs.each{ | fltr |
      assert(fltr.apply_filter("I love to Goooogle"))
    }
  end
end

This test is extremely simple and runs in a flash. There is little to no set up and no outside dependencies either.

Component Tests- Component or subsystem tests test portions of a system and may require a fully installed system or a more limited set of external dependencies, such as databases, file systems, or network endpoints to name a few, man. These tests verify that components interact to produce the expected aggregate behavior.

A typical component test requires the underlying database to be running and may even cross architectural boundaries.

Because larger amounts of code are exercised by each tripping test case, more code coverage is obtained per test; however, these tests have the tendency to take longer to run than unit tests.

Below is an example of a component test which utilizes DbUnit to seed a database and then attempts to find data based upon the contents of the DB.

package test.org.aglover.words.dao;

import java.io.File;
import java.sql.Connection;
import java.sql.DriverManager;
import java.util.Iterator;

import junit.framework.TestCase;

import org.aglover.words.bizobj.IDefinition;
import org.aglover.words.bizobj.IWord;
import org.aglover.words.dao.impl.WordDAOImpl;
import org.dbunit.DatabaseTestCase;
import org.dbunit.database.DatabaseConnection;
import org.dbunit.database.IDatabaseConnection;
import org.dbunit.dataset.IDataSet;
import org.dbunit.dataset.xml.FlatXmlDataSet;

public class DefaultWordDAOImplTest extends DatabaseTestCase {

    protected IDataSet getDataSet() throws Exception {
        return new FlatXmlDataSet(new File("test/conf/wseed.xml"));
    }

    protected IDatabaseConnection getConnection() throws Exception {
        final Class driverClass = Class.forName("org.gjt.mm.mysql.Driver");
        final Connection jdbcConnection =
       	 DriverManager.getConnection(
           "jdbc:mysql://localhost/words",
       	   "words", "words");
         return new DatabaseConnection(jdbcConnection);
    }

    public void testFindVerifyDefinition() throws Exception{
        final WordDAOImpl dao = new WordDAOImpl();
        final IWord wrd = dao.findWord("pugnacious");
        for(Iterator iter = wrd.getDefinitions().iterator();iter.hasNext();){
            IDefinition def = (IDefinition)iter.next();
            TestCase.assertEquals(
                 "def should be Combative in nature; belligerent.",
                 "Combative in nature; belligerent.",
                 def.getDefinition());
        }
    }    

    public DefaultWordDAOImplTest(String arg0) {
        super(arg0);
    }
}

As you can see, this test case requires some set up! There has to be a database in place as well as a file (the xml one which contains all the data for DbUnit). This test case could take a few seconds to run- not a lot in isolation, but think about running this as part of a larger build with 100’s of tests. Those seconds can add up to hours.

One key difference between component level tests and higher level testing, like system tests (defined next) is that component level tests exercise code via an API, which may or may not be exposed to clients. In the code above, an object in a DAO layer is essentially tested via an exposed interface. Another example of a component test is exercising an Action class in a Struts architecture via the StrutsTestCase framework, which in this case, obviously requires a database to be running; however, the container is mocked out and the API exercised isn’t necessarily exposed to clients.

package test.com.acme.mein.web.prot.action;

import test.com.acme.mein.web.action.frmwrk.DefMeinMockStrutsTestCase;
import com.acme.mein.businessobject.impl.project.Project;

public class ProjectViewActionTest extends DeftMeinMockStrutsTestCase {
   public void testProjectViewAction() throws Exception{
	this.addRequestParameter("projectId", "100");
	this.setRequestPathInfo("/viewProjectHistory");
	this.actionPerform();
	this.verifyForward("success");

        Project project = (Project)this.getRequest()
          .getAttribute("project");
            assertNotNull(project);
            assertEquals(project.getName(), "DS");
  }

  protected String getDBUnitDataSetFileForSetUp() {
	return "dbunit-seed.xml";
  }

  public ProjectViewActionTest(String arg0) {
	super(arg0);
  }
}

This type of test is also commonly referred to as an Integration Test. The difference between this type of test and a system test is that Integration tests (or component tests or subsystem tests) don’t always exercise a publicly preferable API.

System Tests- These tests exercise a complete software system and therefore, require a fully installed system, such as a servlet container and associated database. These tests verify external interfaces, like web pages, web service end points, or GUIs, work end-to-end as designed.

Because these tests exercise an entire system, they are often created towards the latter cycles of development; furthermore, these tests have the tendency to have lengthy run times, in addition to prolonged set up times. What a trip!

Keep in mind that these tests are fundamentally different than Functional Tests, which test a system much like a client would use the system. For example, in the code below, this test mimics a browser by manipulating the site via HTTP; however, this test doesn’t use a browser. A framework like Selenium, which drives a browser, can be used to create Functional Tests.

The code below is an example of a JWebUnit test case, which attempts a website login and then verifies the attempt was successful.

package test.com.acme.web.cve;

import net.sourceforge.jwebunit.WebTestCase;

public class LoginTest extends WebTestCase {

  protected void setUp() throws Exception {
	getTestContext().
         setBaseUrl("http://pone.acme.com/meinst/");
  }

  public void testLogIn() {
	beginAt("/");
	setFormElement("j_username", "aader");
	setFormElement("j_password", "a1445");
	submit();
	assertTextPresent("Logged in as aader");
  }
}

While it may not be obvious in the code above, the entire system (a servlet container and a database) has to be installed and running for this test case to work. Note that the set up here isn’t in the test case, but part of a larger aspect of the build.

There are other types of developer tests which, in effect, cross these boundaries (such as performance tests); however, from a high level, these are my suggested terms. Dig it?

Trackback this Post | Feed on comments to this Post