There is not a lot of good information on the Intertubes about unit tests versus integration tests. They both server good purposes, but the key to using them well is to understand their limitations. The largest mis-perception is that 100% coverage with unit tests is all you need. This is not the case. In any system of any complexity, especially ones built on protocols or dynamic languages, this can lead to a false sense of security.
The reason for this is the misunderstanding around the limitations of unit tests. Although all that needs to be said about it is in its name "unit", some developers just don't really get the limitation. Let me illustrate...
Assume you have the following module that forms part of an integrated system:
function Greeting() {
this.greeting = null;
}
Greeting.prototype.getGreeting = function() {
return this.greeting;
};
Greeting.prototype.displayGreeting = function(el) {
el.innerHTML = this.greeting;
};
Greeting.prototype.setGreeting = function(str) {
this.greeting = str;
};
With a simple QUnit test case, we can validate that this works as expected (of course I have abbreviated what would be required for a full test for illustration purposes):
(function(jQuery) {
module( "core - Greeting" );
test( "Test Greeting", function () {
var g = new Greeting(),
fix = jQuery("#greeting");
expect(3);
g.setGreeting("Hello");
equal( g.getGreeting(), "Hello");
equal( fix.text(), "");
g.displayGreeting( fix[0] );
equal( fix.text(), "Hello");
});
}(jQuery));
And in another module, we can use this Greeting class successfully as follows...
var g = new Greeting();
...
g.setGreeting('Hello World');
...
g.displayGreeting(target);
...
Now this consumer module would have its own unit tests and (following unit testing best practices) would mock the Greeting class - perhaps as follows...
function Greeting() {
this.setGreeting = function(str) {
this.msg = str;
};
this.displayGreeting = function (el) {
el.innerHTML = this.msg;
}
}
All unit tests pass and the code coverage metrics exceed 95% - SUPER!!!
Now the Greetings development team decides that the module requires an update to simplify the API because all users of the module always call "new Greeting();" and then "setGreeting()", so setGreeting can be eliminated in favor of a constructor parameter and they refactor the code to:
function Greeting(msg) {
this.greeting = msg;
}
Greeting.prototype.getGreeting = function() {
return this.greeting;
};
Greeting.prototype.displayGreeting = function(el) {
el.innerHTML = this.greeting;
};
A huge improvement and a decrease in lines of code of almost 25%. This breaks the unit tests, so they fix these as follows...
(function(jQuery) {
module( "core - Greeting" );
test( "Test Greeting", function () {
var g = new Greeting("Hello"),
fix = jQuery("#greeting");
expect(3);
equal( g.getGreeting(), "Hello");
equal( fix.text(), "");
g.displayGreeting( fix[0] );
equal( fix.text(), "Hello");
});
}(jQuery));
The UNIT tests all pass and they commit the code. The CI server runs all the tests, they all pass. The code coverage metrics look great!! All is well... Well as you have probably already figured out, the integrated system just failed. This is the limitation of unit tests! They are blind to API changes.
Solutions
The only 100% effective solution is to make sure you have code coverage with your integration tests. Unit tests are great for libraries that are consumed by parties that are distantly related and who have an distinct distrust of one another.
An alternative approach is to build dependency tests. So in this example, if the consumer team had a set of unit tests for the Greeting class that it ran independently, then this issue would have been caught. But if you do this, what you are requiring is:
- That the producer of a module produces unit tests of their module
- That a consumer of a module writes a mock of the module they are consuming
- That the consumer of a module writes unit tests for the portions of a module they rely on
- That the consumer of the module write unit tests for their own module
You end up writing almost 4 times the code for every module.
Yet another approach is for API providers to publish unit tests separately, in a way that does not get updated automatically (e.g. a separate repository or directory for every release). This set of unit tests could then be run by consumers against the API and detect when they are consuming a version whose API has changed. This works well for internal teams, where the test execution framework is standardized, but is likely to place a large burden on a consuming team where the test frameworks differ.
Even with the above unit test approaches, data, environments and other factors will require a decent amount of integration testing anyway. Ultimately, in my opinion, for code your organization owns, you should always write integration tests before you write unit tests. Unit tests are optional, integration tests are not optional.
