The business model (or lack thereof) of selling unit testing

Ralph Waldo Emerson is often attributed to copasetically saying (or most likely writing) something like:
“build a better mousetrap and the world will make a beaten path to your door”
This Sirens call has lured many an individual to build a superiorly hip mousetrap. Yet, the truth of the matter is that the world has warehouses full of better mousetraps; unfortunately, the traps are gathering dust at the expense of the inventors (and perhaps distressed investors).
The basic mouse trap works, man. It is low cost and simple to use– put some cheese on it, place it in a strategic location where mice are known to be and bingo! soon enough you’ll have yourself a mouse (dead or alive). It’s not pretty, nor is it entirely effective when it comes to scalability; however, it’s hard to overlook the fact that a mouse trap runs you about $1 (plus the cost of a morsel of cheese).
Better mousetraps, while technically superior, are often more complicated and in many cases, because it’s their bag, more expensive. Some people will buy them and find them useful; however, the average disco dancer (otherwise known as “Joe”) doesn’t care so much about the fancy trap– he or she wants mice exterminated quickly. Accordingly, they still buy the basic mouse trap because they know how it works and it costs about a dollar. And in the end, it does basically the same thing as the snazzy “better mousetrap”– that is, removes mice, baby!
Better mousetraps will always be around so long as the cost to produce them is less than what is earned by selling them (the higher the difference has a direct relationship to how long that mousetrap will last on the market too– people like making money almost as much as they love disco music). People, from time to time, will buy the superior mousetrap– often times because they’ve done the research and realize they have a particularly bad mouse problem. This realization, though, takes a lot of self discipline and honesty that the average Joe lacks. Also too, it costs a lot of money to educate the market about the superior mousetrap– most hip people don’t care because they understand the essential premise of the basic mousetrap!
In essence, xUnit (and related frameworks) are the basic mousetrap. In fact, developer testing (in the form of JUnit or any xUnit framework) is free. You can download frameworks like JUnit and start coding with them in less than a minute– maybe 10 minutes if you decide to read some documentation.
They work (they catch mice)! They are effective, but maybe not entirely pretty. Scaling xUnit across a wide code base with no testing is, at best, a herculean effort. It is challenging to overlook, though, the basic cost– $0. People who download these frameworks are motivated by pain– in this case, defects, and they are looking for a solution (often times, though, they are looking for an easy solution to a difficult problem– i.e. lots of mice procreating at a rapid pace– but knowing that they have a deep problem takes a lot of honesty and discipline, remember?).

Companies have, indeed, baby, built better mousetraps for testing. For instance, Agitar’s product was phenomenal (and their team, outstanding); nevertheless, their superior mousetrap cost a heck of a lot of money. In the end, however, the Agitator produced JUnit tests. The same tests that one could have arguably written for free (yes, I know that in reality many of tests would have never been written by hand, however).
In essence, Agitar took a commodity (a mousetrap, but in this case, JUnit) and made it better– indeed, they have (or is it had now?) a good customer base (meaning they actually had customers, man). There were people that saw the benefit and had the discipline and honesty to admit they had a problem that the Agitar could solve in a superior manner.
Don’t forget basic economics though– the cost to produce their mousetrap was significant. There are only so many people who would buy a $500 mousetrap when sitting right next to it, on the same shelf, is the dollar variety (which, by they way, works fine). So too, there are only a few companies that would likely spend roughly $50,000 to $100,000 on a tool where any Google search would display a link for another tool that has zero cost (i.e. junit.org).
Indeed, there are other smokin’ companies selling superior unit testing tools that are still operating today; however, if you look closely at the big names, they have a diversified portfolio of products and it wouldn’t surprise me if their economics work out more favorably (but keep in mind the statement regarding profit and cost and the difference of the two related to length of presence in a market).
When it comes to better mousetraps, there is another well-aimed saying, which is, nonetheless, only partially true:
“Before you build a better mousetrap, it helps to know if there are any mice out there”
Indeed, this quote is half correct, for as everyone in this Age of Aquarius knows, taking the same hip comparison of mousetraps to testing tools and mice to defects, there are plenty of defects out there! The essential aspect this quote fails to mention is that:
“and you had better make sure there are plenty of well educated, motivated customers who will actually pay real money for the mousetrap (even though they could buy something similar on the cheap), baby!”
The business model (or lack thereof) of selling unit testing tools is difficult. Period. If you want to do it, you had better make sure you can build the better mousetrap on the cheap and that there are a lot of people who are educated, disciplined, and honest enough to pay for it.
I have mice (actually defects, but you get the point, man) and have, from time to time, considered buying the superior mousetrap; however, more often than not, I’ve ended up purchasing the dollar variety because it is quick and easy. Unit testing isn’t dead, baby– the inventors of better mousetraps (that have failed) often tout that mice will overrun the world. No, unit testing is a largely established practice that will gradually grow because the cost is free. Mice will always be around and people will always want to exterminate them– they just don’t always want to pay through the nose to do it, man. Can you dig it?
| Related odds and ends | ||
|---|---|---|
Thursday 29 May 2008 | Developer Testing, Software Development
As someone who has both defects and mice and was formerly part of the outstanding team at Agitar I hear what you say man and mostly agree.
A couple of quibbles:
1. It’s inaccurate to say that Agitar made an expensive version of the free JUnit. That’s like saying the Encyclopedia Britannica is an expensive version of pencils. Not saying the Encyclopedia Britannica is useful mind you…but you could write your own with pencils if you chose to.
2. There is a better mouse trap - http://www.raggedclown.com/tag/lolmouse/
But, companies die ( http://www.raggedclown.com/2008/05/14/lines-on-the-death-of-agitar/ ) and mice die ( http://www.raggedclown.com/2008/05/20/farewell-simon-thou-wert-a-valiant-foe/ ) and life moves on.
On the eradication of software defects…
I loved Andy Glover’s hip comparison of mousetraps to testing tools and mice to defects.
I live in a country that had no mammals, other than a few bats, and no software defects, until man arrived around 1000 years ago. The introduced mammals have ….
The cost of unit testing can hardly be considered free. While the tools are free the effort expended can be drastic. Projects looking to achieve complete automated testing (unit, functional, performance) can look to spend up to half their work days on the effort of building and maintaining the tests.
If a tool came out that could remove all that effort it would be an instant success. Agitar was more a sideways move to doing it differently with a moderate improvement in effort, but it didn’t suit the working styles of a developer. It aimed itself (unfortunately) at purchasers (managers) with monitoring tools and other non core functionality in an effort to get sales. Truth be told a drop in price, a free community edition and some real push to get the core testing framework out there would have worked better. Good idea but bad business plan considering most tools were headed to being free.
Agitar is tool for a code base that does not have unit testing. But junit is superior to agitar when combined with Test Driven Development. Agitar does not support that. This is why agitar is not a better mouse trap in my opinion, the greatest value is TDD.
[...] [...]
[...] goes on to highlight three recent events such as Agitar’s demise, Enerjy’s freebee spree and lastly, CodeGear’s mercy acquisition (one can argue [...]