Test coverage does not equal quality

You follow test driven development. Code coverage is excellent. Yet everything is buggy. Quality of the code is terrible. What is wrong?

Chosen the wrong indicator, the wrong measurement, we optimize for the wrong things. Like code coverage. It is an indicator, or not, depends on situations.

There are coverages you can’t measure. Like how many times your team actually clicks through your app. It is still a testing.

You have high coverage and neither your business or design colleagues ever touched the app? Going to have bad surprises.

It is not even that the app will be buggy or crashing. It may just simply not do how and what the business expected. And they will perceive it as low quality.

So many dimensions could be tackled here. Let’s keep to coverage. Have you considered the value of each test?

High value tests are the happy path and the major failure paths. Only on the application level.

And use Pareto’s principle. 20% of your tests will catch 80% of your problems. If you take care of those tests and not turn them off the first time something seems off.

Besides, your user does not care if it took 3 server calls or 5, until it was fast enough. They are not the ones paying your infrastructure bills.

They care if the app is quality or not. How do you know quality? If it solves my problems without introducing new ones, I consider that quality.

Move bottom menu up to the top for tablet users. Who uses it upside down while scrolling? It may work technically perfectly. User-wise it is annoying. Would a test catch that?

Nobody can automate you out of your own business as a leader, founder, creator. If someone could, may not be a good business to be in.

Stay in the loop. Use your own applications, services. Not just say it’s QA’s problem. It’s all of our problem. And our responsibility to make it better.

And not the coverage number on the dashboard. It does not tell the whole story.

How did you argue for or against code coverage?