Today, I like to argue with people who value code coverage as a good indicator of quality of the testing.
In my opinion, code coverage has nothing to do with quality of testing. You cannot associate these metrics to testing quality. I'll address the benefit of code coverage at the end of this blog.
1. You are saying "this man is tall therefore this man must be good at playing basketball".
If you are saying "testing is done well because of high percentage of code coverage", then you're saying the same thing as "This man is tall therefore he is good at basketball." Code coverage percentage is an illusion. What illusion am I talking about? You are assuming that dev code that you're testing is perfect. Dev code is never perfect. You can have a high percentage of code coverage but application can be still really bad. Let's say you have 95% of code coverage from your testing. If some simple input from a user make the application crash, is that a good testing result? I don't think so. Testing should be judged by users/clients and customers satisfaction not by code coverage percentage.
2. Bug or problem comes from mistake or wrong logic in the code. How the he?? code coverage metrics find that?
This might goes with my first point. Testing is a process of examine the code based on purpose or business needs. This intellectual activities are too huge and great to even compare the value of code coverage. My critical thinking, my creative thinking, diligence, effort, cognitive decision making, experience, and product knowledge add the quality of the testing. Not code coverage percentage.
3. Code coverage tool can never understand the business value of the application.
I don't 100% believe in 80/20 rules, but most of valuable business cases are in small portion of the application (I could say 20%). What does this mean? This means your testing should be focused on important/critical/money making features of the application. We humans(thinker and tester) know what those are but code coverage tool does not know. We're making conscious decision (not to test or test lightly) on those minor/ very little critical features. We have priorities but code coverage tools does not.
4. You can easily increase the code coverage percentage by mistake or poor testing.
Let's say the code that you're testing has very complicate decision making code. Lot of if-statements and switch statements (these are basically bad coding practice but let's assume we have). How can you possibly make code coverage high with poor testing? If each if-statements calls some sort of methods to get true or false value, all you can do is put one parameter or input to go all the way down to the last conditional (if, else if ...... esle). You can do it by accident or by poor testing. And now you cover all the method execution used previous if and lots of else if statements. You might have one test case but that accidentally cover all the code path. Wow, you have one test case but you've got a good code coverage. You're good to go. Why don't you ship it without testing?
5. What about integration testing?
If you are in the middle of the stack like in SOA world. You need to examine how your application plays in the entire stack. Code coverage percentage tell you that? Integration testing is one the most important part of the software testing. It works fine individually. Oh well, it can easily break things when these individual components are working together. Can you really tell code coverage percentage is integration testing requirements? Seriously?
6. Your code coverage percentage goal make you waste your time.
Some test team has a goal of code coverage greater than 90% as part of "EXIT CRITERIA." I don't know what exit criteria means but this code coverage percentage make me very sick. You've done your best to say I'm done testing. You carefully consider time and budget and get to the point where you're confident about the application you're testing. Now you code coverage tells you that you're covered 75%. What do you do now? Well the "Exit Criteria" said that I have to cover 90% code coverage, if I don't do that it will affect my reputation and review score. I'm getting on to my desk and write meaningless automation just to get more coverage. Hold on. This is not all that bad. This point leads to some interesting value of code coverage. Let me mention some good part of code coverage.
Benefits of code coverage.
I find code coverage valuable when I understand which area of code were not covered and why. Again not the percentage metrics. I might miss something important. Oh good, This nice tool reports the classes and codes that I did not cover. Let's sit down with dev and discuss about this. "Hm... I did not know that. Hmm. This code need to be verified since it is important." OK.. now I see the value of code coverage tool. Percentage is meaningless. Finding out place where I have not thought of. THIS IS THE VALUE OF CODE COVERAGE!
