Tight testing


Even though we do a fair bit of testing I notice I'm still struggling with parts of writing tests. I'm convinced that integration tests are vital for a product. I feel they are more important than any other test. Yet I can't deny that unit tests have their merit. The question I can't seem to answer is where to draw the line.

Say you have a function that only calls two other functions, nothing else. Do you test whether this function actually calls those functions? That almost feels like meta programming.

foo = {
A: function(){},
B: function(){},
AB: function(){

// test for AB
var ran = 0;
var placebo = {
A: function(){ ++ran; },
B: function(){ ++ran; },
assert(ran, 2);

The test is actually flawed in that it doesn't test whether A and B are called. It only checks if either of them is called exactly twice, or both are called exactly once. But that's not important to my point.

You're just checking if you are indeed calling those two functions. It is so trivial that it at least feels weird to do this. Am I supposed to second guess trivial coding? I'm not sure. It looks very trivial in the case presented, but what about a huge project with thousands of tests? Does it make sense in that regard?

Maybe. It could if it's actually vital to call A and B from AB. And of course, it usually is. I think the question is whether you would want to be notified of a problem when you are changing the name of either of those functions. And I'm not sure about this. In the end you're still just writing a test that says exactly what the function is doing; call two other functions.

Let's say we have a function that does a few things to an array:

function foo(arr){
var found = arr.filter(function(){ ... }).map(function(){ ... }).some(function(){ ... });
return found;

Would you write tests that check whether the array methods filter, map, and some are called? In that order? And if you would, would you check somehow whether the passed on parameter is indeed a function? And whether this function returns ... etc? No. I would extract the anonymous functions and put them in the class so you can test them individually (with integration, input/output, tests). So if you wouldn't test those method calls here, why would you test the calls in AB?

There might be another reason. One I don't agree with myself either, but I know people appreciate it. It's about putting comments and contracts in tests. In this case it would mean that the there would be a test for AB and aside from simply testing whether the two functions are called, it would get the rationale in the test comments/header. Why A is called first and such. If this is your case then fine. I feel these kinds of comments should stay in the actual code, and you should keep them up to date when you change things around.

The main two reasons, I guess, for not explicitly testing these trivial functions would be that 1: it adds a lot of overhead to writing code and 2: it makes refactoring even more of a pain. The overhead makes you less productive while, in the case of testing trivial code, the gains are doubtful at best. And refactoring any part of the code means you'll have to rewrite a part of your tests. This part grows quickly as the refactoring touches more and more code. At this point I hope that those tests have paid off somehow otherwise the overhead was certainly hardly worth it.

Don't get me wrong, I'm a proponent of testing. I'm just still wondering about the proper balance between what to test and what's too trivial to test. This is why I value integration tests more. They can quickly give you an overview of expected input/output. Even if it's on a (sub)module level. Integration tests aren't always viable though. Especially when visual systems are involved.

If you have undercover unit tests that are actually integration tests (say you're testing the outcome of a function that just applies a certain formula to the parameters), the tightly coupling problem comes back again. For me, that's very notable with string manipulations. If I have a function that constructs a string I'll almost always trigger a fail when I even change the smallest bit. But in this case the trivial argument doesn't really go. You need to test input/outputs and you need to update tests if you screw up. I suppose that this also goes for integration tests though.

While I can appreciate testing theory, it sometimes just clashes with the pragmatic approaches that I prefer to apply.