As far as Unit Testing goes, mock objects are a pretty useful addition to the toolkit. In a componentized system, by mocking out all the other components, you can isolate the one you're testing that much more effectively. Mock object frameworks exist to make the job a little easier: providing simple ways to provide substitute objects, and record and predict method calls.
Unfortunately, once you get in the mock object habit, it becomes too tempting to test things that you really shouldn't be testing. Because you can test to a granularity of which methods get called on which components, because it's easy, you find that you do it as a matter of course.
Often, you shouldn't be. Let's get down to what a unit test really should be. A unit test should predict some observeable effect, and test that it can make that effect occur. How the object makes the test pass is irrelevant, so long as the test passes. Mock object testing looks a little too closely at what lines of code are actually going to be written to make the test pass. Testing lines of code directly, rather than the effects of those lines of code, can lead to irrelevant and hard-to-maintain tests.
If you start writing mocks to predict (and test for) every call to the objects that are being mocked out, you end up with fragile tests that break trivially as the result of refactorings that change how the object being tested does something, without changing what the object does.
But that's not all.
Most of the systems I've seen over the last few years have been written in layers. Down the bottom, you have components that do the important stuff, but don't talk to other components much. The layer above coordinates between the lower-level components, but doesn't do much interesting itself apart from that delegation.
So what happens if you test those higher-level components with mock objects? Well, you sort of end up with tests that do absolutely nothing but test a series of method calls on mock objects. This is, I would suggest, worse than useless. What are you testing, really? You're not putting forward any kind of verifiable hypothesis about what the method you're testing should be producing: you're just writing it twice, once in the test and once in the object being tested.
If you get the series of calls wrong in the test, you'll get it wrong in the object too. You have exactly the same margin for error with and without the test, you've just moved the risk around a bit.
It pays with mocks, therefore, to always keep in mind the question: "what exactly am I testing?"
Yes! This is EXACTLY the problem with mock objects, and more specifically the 'generic' mock objects that have become so fashionable lately. Most of the code is spent setting up expectations and inputs, and if you strip away all the setup code, you end up testing some incredibely trivial method call that due to the nature of the test will never fail on the one hand, and a brittle test that will require more effort and time to maintain that the code it's attempting (and failing) to test.
It's a layered system, right? And each layer is separately unit tested, right? So when we test higher level stuff we really don't need to run all the way down, it's just easier to setup the test this way, because it's more isolated testing.
Mocks can of course be used in two different ways: testing whether method x is actually called with the correct parameter, and testing the output of the whole code but using mocks for easily testing the thing in isolation and with ease.
The first one is useful if you want to test whether during execution of the code "which leads to output y" method x is indeed called. Let's say for testing whether save() is actually called and somewhere in between if/else checks it's not missed, althought the output is what you're after. I believe this type of testing is rather less useful, though a good side effect of it is that it helps you write better code because you're more careful about "component boundaries".
The second use of mocks is much more valuable imho. We can easily test code that use HttpServletRequest outside of a container with mocks, but honestly testing whether getHeader() is actually called on the request object is imho a side effect of mocks and not the main goal.
And btw I don't believe in high test coverage numbers. If something just too trivial just don't test it.
Ara.
I found that the original Ruby Mock object suffered from over specified tests. It wanted to know exactly what methods would be called and in what order. For example, I just wanted to test that the correct data was passed with the "print_template" method of my web object. Other methods that were called were unimportant to that test case, but the mock object insisted I specify every little call. FlexMock (see http://onestepback.org/index.cgi/Tech/Ruby/MockTurtleSoup.rdoc) avoids that problem.
I have the same feeling about test code coverage reports. I don't necessarily care if every single line of code is executed in my tests, only that the right ones that produce the important observable/testable behaviour of my object are. Granted, I want a significant part of the code tested, and if there is not then either my tests are rubbish or there is a lot of crap in my object, but the holy grail of 100% coverage that some seem desperate to attain is not necessarily worth it. Its a bit like David Pinn's thoughts on when not to refactor (see: http://byandlarge.net/scuttlebutt/archives/2003_12.html#000097 )
Hi.
You raise an important alarm bell, but I think it is a problem with a more general solution. If you are getting a test that keeps failing for the wrong reason then refactor the test. I find that with TDD I introduce a lot of trivial tests whilst the code is still half formed. Mocks help a lot here, working as much as a debugging and prototyping tool rather than as a test tool. Once the code has settled I get a bit lazy to be honest. A few silly tests get left around that don't do much. As long as they are later dealt with on a case by case basis, I don't find them a great burden, though. The general solution of refactoring whenever you are repeating yourself kind of fixes these things for me.
Another issue touches upon this. What are we doing when unit testing? I am usually debugging, testing, designing, documenting, getting confused and clearing my head again, all at the same time. Are testing divisions a matter of style? It's a new area after all.
I usually mock nearly everything I am not actually testing. It documents all the interactions of that class group very clearly. This pushes a burden onto your "integration" tests. I use the word "integration" loosely here because, to be frank, I have no idea what it means. We tend to have additional tests beyond acceptance/functional/end to end tests. We call these integration tests.
We don't attempt 100% test coverage though. We are too lazy.
yours, Marcus