The basics of: QA metrics part 1 - What is quality?

Metrics can be a tricky territory to navigate. It’s not so much about the data itself, but figuring out what’s actually important and what that looks like in the shape of numbers.

Yes, metrics can be a deceiving business. That’s why when it comes to QA and testing, you’ll have to get them right – before you even get started.

But it doesn’t start with metrics. Because before you define your metrics, you have to define what quality really means.

That’s why in this blog post, we take a closer look at what quality means and what stands in the way of getting it right.

Let’s dive in.

What is quality?

The late Gerald M. “Jerry” Weinberg taught the software world a lot about quality. Weinberg, who started his career at IBM in 1956 as an applied science representative, had by the time of his death in July 2018 published about 100 books.

But perhaps the most important part of his legacy was this: “Quality is value to some person(s).”

Later, James Bach and Michael Bolton, added “…who matter” to Weinberg’s initial statement, completing the definition of quality as:

Quality is value to some person(s) who matter(s).

Meaning, decisions about quality start with decisions about whose values matter, what they value, and how that value might be threatened.

Defining value

Jerry’s definition of quality is used frequently. But his definition of value isn’t mentioned all that often. It goes:

What are people willing to pay (do) to have their requirements met.

Anyone who’s involved in making quality related decisions, such as product owners, programmers, testers etc, have to invest their efforts in meeting end-user requirements. That starts with defining what those requirements are and what stands in the way of delivering those requirements.

For example, your app might meet the requirements of a potential user, but in order to register the user gets redirected or has to pay x dollar amount. In certain cases and for certain users, that will instantly turn into negative value.

It’s important to understand that a product is much more than its components. And as such, quality is far more than the lack of defects. So what does that mean for testing? Well, testing then, becomes far more than “simply” determining that the product–or aspects of the product–are “correct.”

Quality is political (and emotional)

In his book Agile Impressions (originally in Quality Software Management Volume 1: Systems Thinking – as far as I know), Jerry Weinberg talks about how our decisions about quality are political, but also emotional.

Why? Well, we all have a personal relationship with software and products. That means value is going to vary. It’s unlikely that we’ll all value the same aspects of a product or have identical requirements.

That’s why our personal preferences affect our decisions about quality. Because we’re all biased. We all make decisions (knowingly or unknowingly) that drive the product closer to something we’d personally like to use. Perhaps we want to report bugs that simply bug us, but don’t really have any effect on the experience or requirements for the end user.

Decisions about quality are usually made by people in power. Often, that decision is based on how those people in power feel. When making those decisions, they may try to cater to stakeholders other than the end user. And any decision that doesn’t prioritize the end user or create value for them, is ultimately a bad decision.

Emotions aren’t all bad

Should our personal preferences affect our decisions? Well, it’s impossible to avoid. It’s the human factor. But emotions and politics aren’t all bad. Our own feelings can (and should) act as triggers for further investigation. But one’s subjective angle shouldn’t stand in the way of value output when it comes to end user requirements. We just have to be aware of the fact that it’s happening and see if those preferences are aligned with the who, what and how’s.

Testing doesn’t assure quality

Okay, let’s say you have a clear definition of what quality means for your product and/or service and you know what value looks like to your end user.

The next step is assessment.

We assess quality and determine value by testing the product or service. So, what does testing mean? Here are a few of the most notable definitions:

“Questioning a product in order to evaluate it.”– James Bach

“Gathering information with the intention of informing a decision.”
– Jerry Weinberg

“A technical, empirical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek.”– Cem Kaner

Testing is about questioning, investigating and searching for information. But note that none of the above definitions actually refer to assuring quality. Because the process of testing doesn’t do that. It’s the response to the data you’ve gathered along with all other information available to you – including your own awareness – that assure quality. Yet testing, is what makes informed decisions about quality possible.

Figuring out who the product is important to and what value means to them aren’t easy tasks. But there’s an additional challenge that poses a threat to both quality and value: Our needs and values aren’t stagnant. They change over time and sometimes for reasons unknown. That means that a product can fit certain requirements at a very specific moment in time, or perhaps users get used to a product and start expecting more from it. Or perhaps it’s just being used for a one-off purpose. Because our needs aren’t static, neither is quality.

As a result of our tendency to change behavior, Weinberg’s definition was extended again by Michael Bolton. The temporarily finalized definition of quality reads:

Quality is value to some person, at some time, who matters.

Final thoughts

The meaning of quality and the meaning of value is what will have the biggest impact on the work as a tester. We can never lose sight of who matters, why, what they need–when. It’s a never ending process of investigation. As such, our jobs become more than just following scripts that pass or fail.

That’s why learning to think correctly about problems, solutions – and quality itself – is where the work begins. Because if you don’t understand what quality and value mean for your end users, you simply can’t measure it.

Don’t want to miss the next part of this series? Make sure to subscribe to our blog below to make sure the next post lands in your inbox.

>>Try Waldo now