What you should know about analyzing the value and economic feasibility of Value-Driven Testing

What you should know about analyzing the value and economic feasibility of Value-Driven Testing

As a financial analyst, you keep calculating hundreds of indexes, compare the periods and draw conclusions about past, present, and future. Altman, Taffler, Beaver models, retrospective and investment analysis, and financial stability index give a clear-cut answer to what we shall do.

They indicate what we'd better reduce or increase, what is profitable and what is not, where this notorious breakeven point is, and what the project's payback period is. Economic universities don't teach you how to code, but from year one, they show how to figure out the feasibility of any activity, calculate the product's added value, and spot when the game is not worth the candle.

"But what has testing to do with it?" you may wonder. My answer is "Quite a bit." Testing and software development as a whole, just like any business activity, can be analyzed.

It is easier to analyze an entire IT project than one of its individual processes; besides, testing as it is does not earn money. Does it mean the company cannot analyze a separate workshop or the cost of a specific production phase? Of course, it can. Testing analysis from a feasibility perspective is not only possible but also necessary.

Beekeeper’s view on the bee-hive

Thinking over the labor process, Carl Marx would indicate: "What distinguishes the worst architect from the best of bees is this, that the architect raises his structure in imagination before he erects it in reality." According to Capital’s author, the bees are deprived of the ability to think systematically, analyze, and rationalize their labor. There is why we use concepts of an architect and bees in the context of this article. In layman's terms, the IT industry is sort of a warm, blooming orchid that attracts people, like bees, with the aroma of coffee and pastries. The number of bees (a.k.a. regular implementers) keeps growing, and there is an increasingly pressing need for architects to optimize the working process and boost profitability. It is as much an urgent agenda as thorough hand-washing was in 2020.


In 2020, the Best movie award went to the South Korean film "Parasites," focusing on one family, where its members one after another creep into the house, make a fool of the host, and parasitize him, enjoying board and lodging. Any similarities with “entering IT industry” stories are coincidental, of course. However, hypothetically, imagine that a team acts just like those parasites: they seemingly do what they are supposed to and take no effort to optimize the processes. What is more, a better share of their activities is loss-making for the project.

It often includes extensive testing, meaningless or non-informative reports, the stranglehold of manual audits, metrics for metrics, toothless test cases, and loss-making automation. Thanks to the artful steps of business analysis and PMs, the project costs for the customer keep increasing, so do the content and amount of the additional charges; the team is getting overstaffed.  The customer once decided on cooperation, basing on the good price-quality ratio. But after such a "grant-eating," they realize that they could as well use the staff in the country of their residence. But it's too late to make any changes; parasites have hooked the customer.

No money, no honey

To get an idea about how testing is going on in the project, you must analyze its efficiency regarding the quality of the product and processes. One can calculate defect density, memory holes and leaks, the efficiency of test cases, RC, FDP, DDP, PTC, MTTD, TDE, and dozens of other testing metrics. However, to determine the profitability of such testing, one must count money. The money and its increasing flow is the ultimate goal of any customer when it comes to software development.

To make correct management decisions, a test manager should skillfully navigate testing activity costs, detect development zones, and ways to optimize processes. Customers need to understand what they pay for and why, what causes losses, and what brings profit. No man is an island, and the task of a so-called architect is to do their best to make bees comprehend the reality of how much money they earn for the customer and how much they helped to save. The "leftover" funds (not necessarily, but still) could help fund those bees' pay raise.

Price and value

Any quality has its cost: Cost of Quality = Cost of Poor Quality + Cost of Good Quality:

The total testing cost is quite extensive, but it looks pretty affordable once you estimate the cost of poor-quality testing. Warren Buffet once said that price is what you pay, and value is what you get. They do not always coincide. Quality does not necessarily signify value. Try to sell a high-quality typewriter or persuade the customer that it will be more valuable for them to release a feature not in two days but a year because, during this time, you can test it even better and improve the quality. It won’t work. A spoon is dear when lunchtime is near and "time to market" is still in place.

The task is to reach a price-quality balance for the customer. What is "a balance"? Because with increasing costs of searching for and eliminating the defects, the price of this problem will decrease to the optimal point, after which any further boosting of testing activities becomes counterproductive.

Let's call this approach value-based or value-driven testing and consider the following example.

A QA team comprises three Manual QA. It is not a fixed-price project, and the price-setting is Time&Materials. We have 826 manual tests, a shortage of time, and loads of quality issues. Our task is to improve and optimize the testing process.

The Inspector is coming to us!

Let’s start with a large-scale cost revision. Without recourse to Accounting Standards or IFRS, the cots can be divided into Capital or CAPEX (purchase of servers, licenses for soft, virtuals, etc.), operational (running workload and auto testing, server operation, reporting, setting of the environment), direct (wages, training costs), and indirect (debugging, retesting, update and improvement of test cases). What we also keep in mind are fixed and variable costs. And we don't have to follow Boehm's COCOMO here - everything is far more down-to-earth.

The first thing to be done is to determine how much this or the other testing activity takes in hours and then analyze how to cut this time. It may include a cut in labor intensity, relative labor saving, and the fight against implicit absenteeism of QA team members.

W = I*T, where W — work effort, I — labor intensity or your performance, and T — time of QA work. A concise design, clear prioritizing, decreased number of step repetitions, and other test case optimization helped reduce their number from 826 to 611. In its turn, it reduced time expenditure threefold. This activity and established game rules, applicable for the entire team, save time, assigned on test engineering for the future. An example of test optimization:

Similar measures were used for document drafting (content for Wiki-project), reporting, and internal communication. Despite work efforts (time expenditure) decreasing, they reached their limit (see the graph). Regression scope would increase, and, despite vacant hours giving time cushion, something more was necessary.

The economy of scale.

Left shit testing is the foundation of testing theory. In my opinion, it is similar to the time value of money. A dollar today is worth more than tomorrow, as you can invest it today. And a bug you found this afternoon is more valuable today as you can fix it earlier.

You benefit from the speed and cutting off idle time. The faster we get money and fix defects, the better. A higher number of checks, along with a decrease in the production cost of one audit, inevitably leads to the economy of scale. God save Her Majesty Automation! Let's have a look at this process through the lens of the economy:

where C is the cost of one test, TFC is the total value of fixed charge, Q is the quantity of the tests launched, AVC is average variable charges.

Therefore, a larger amount of inspection per time unit and lower average variable charges are two keys that open the door to cut in production cost of spotting one defect.

Adding QA Auto to the team raised the QA Team fixed charges; however, it could be balanced against performance gain in the long term.

An automation breakeven point wasn't reached until most of the tests ran automatically. The critical volume of tests for automation can be calculated by applying mathematical analysis:

Critical volume = Total automation fixed charge/ (production cost of one manual test - one autotest running variable charges).

The effect of operational leverage started going down, and so were testing variable charges, while QA Manual gained more time for experience-based testing. It enabled the team to promote turnover of regression runs and increase the sprint scope dramatically.

This positive trend indicated that it was beneficial to increase the automation rate. Still, an additional QA Auto did not improve ROI because of associated direct fixed charges (remuneration of labor) that kept growing. The solution was fast and straightforward - to onboard a QA Auto Trainee with a three-month probation period. With no additional direct charges (remuneration of the second QA Auto labor), onboarding cost slightly decreased the productivity of the first QA Auto, but the general upward trend remained unchanged.

Ahead of buffalo

A herd of buffalo can only move as fast as the slowest buffalo. If your automation engineers work quickly but have to wait for manual tests, all their promptness makes no sense. It is the project's bottleneck, and it needs to be urgently fixed.

If you enhance control over manual test writing and standardize their format, QA Auto's time on their examination and conversion will drop dramatically. It doesn't mean you have to dedicate all your time and energy to emergency drafting of meaningless automation tests "for show." The tests should be a strong net for bug-catching, and the test flow should cover the Automation Team performance with a safety margin.

Otherwise, the accumulation of tests will make no sense, while time loss from their delay will be ungrounded. It relates to the entire testing process - from planning to summary reports. No bottlenecks.

It's tough to adjust the conveyor line, but the streamlined manufacturing line is more valuable than hundreds of individual machines. If you dive deeper, many activities can be eliminated. It is essential for automation processes. Else, the introduction of automation will only boost the productivity of this Sisyphean labor.

It takes more than doing your best. As Deming once said, you must know what to do and then do your best while improving processes. Among all improvement models, I like TMMi best - Level 5, which, as they say, has much in common with a perfect man: many have heard of it, few have seen one!

But it is something to strive for, and if we mean to deliver value to the customer, it's high time to determine where we are now and where we're heading. The introduction of test improvement models is a worthwhile part of your value-driven testing.

Risk-taking or Hard Math

Sometimes, despite the bugs you found, you decide to release the version. Such news always discourages testers. They start saying, "well, blame yourself, do what your want," "how can you release like that," "why did we even check it then" and so on. There is a reverse situation when lots of tests are performed and almost no bugs are found. It is a simple truth that if software, just like a human, is absolutely healthy, it means it is underexamined. It triggers questions like, "Why are there so few bugs?" "How exactly did you test it?" But if you look at this situation in terms of value, it will click.

Let’s have a look at the example. You are going to release a feature that costs, let’s say, $75K. There is a 40% chance that this version has a critical bug, and if this defect leaks down to the users, the associated costs will hit $150K. You can play safe and not release it, but there will be no profit either.

If we release it at once, given the probability of critical bug emerging, the expected net profit is: $75K — ($150K * 40%) = $15K

We can spend money on testing - let's say those $15K. Testing will either spot the bug or not - it’s 50/50%, so we divide 40% by 2. In this case, the chance the bug slips into the release drops from 40% to 20%. Now let’s do the math.

Multiplication of the event probability by the cost of the outcome shows a more profitable decision.

  1. We don't release it at all: $0K
  2. We release immediately without testing: $15K (as calculated above)
  3. We test and release it only if a defect is fixed: −15K * 60% + (-15K * 20%) + 60K * 20% = $0K
  4. We test and release it anyway: 60K * 60% + (-90K * 20%) + 60K * 20%= 30K

Interestingly, we did not consider fixed charges; otherwise, the third strategy would have been even less attractive. It doesn't mean you should not fix bugs. It's just illustrating one fact: the most profitable strategy has proved to be the fourth one where you test the product and release it anyway. It's hard to believe, but for this specific case, it is true. Even if the tests performed found no bugs (given that they could), they bring actual economic value by decreasing the chance of defect leakage.

Therefore, the range of test coverage (a probability of detecting a defect) is much more significant than an actual number of detected bugs. It’s like a Battleship game. The winner is someone who reaches the most extensive coverage of the opposing player's field. It's more important than the number of destroyed ships.

Testing activities will be even more valuable if any Risk-Based Testing is applied, be it FMEA, PRA, or PRISMA. Risk-based testing consistently prioritizes your tests, teaches your team and customer to look for and evaluate them correctly, and, as a result, will safeguard your future releases. The exposure to risk is estimated by multiplying the probability of functional usage by fail probability and the cost of its outcome. Implementing such an approach is cost-consuming, but the product quality and peaceful sleep are well worth it.

Now, when our risks are apparent, estimated, and translated into relevant test priorities, you will explore the riskiest features more thoroughly and in the first place, and the example with the Battleship will look like that:

I recommend using the results of financial analysis when making managerial decisions in most various situations.

Let’s have a look at the example. Your project has a 20% chance of data loss on the testing server. Should it happen, the cost of such loss (terms and price of restored data) will make $20K.

Let’s evaluate risks: 20% * $20K = $4K

One can avoid risks, accept, reduce or transfer them. But what shall you opt for? There is an option to introduce a backup mechanism and cut the risks by 5%. The influence will remain unchanged since should the server fail, we'll lose the same amount of data while the backup operation will cost $2K. Now:

Probability - 5% (after the measures being taken)

Losses - $20K (unchanged)

Risk evaluation after 5%*$20K = $1K

Cost of risk reduction - $2K

Let’s calculate the leverage of risk reduction: ($4K — $1K) / $2K = 1.5

This index obtained (>1) indicates that, in general, such measures are appropriate.

We can consider an option of transferring risks by outsourcing. If the second option proves to be more profitable, but the team decides to introduce a backup mechanism themselves, it will bring out opportunity costs.

You should perform a similar comparison with ROI, choosing a better option.

There are many details to be considered when calculating ROI, depending on the project and testing type. The most challenging thing is to estimate the profit correctly. It would be very problematic to do that without analyzing charges and the production cost of testing activities. The automation costs can be cut mainly at the expense of parallel test launching, code reuse, reduced cost of result analysis, automated drafting of testing reports, and setting up the test environment.

Problem of choice

New Year is coming, and so is the budgeting of testing activities. Imagine you have two projects and two separate testing teams - A and Z.

Where shall you allocate the money for development, and how do you make your estimation correctly? Project A's discounting rate is 10%, and Project Z's - 12% due to the longer term of its implementation.

Other things being equal, it seems it is more profitable to provide the budget for Project Z’s development as it asks for smaller investment and will pay back better.

But a dollar tomorrow is worth more than a dollar today, so you should estimate the net present value for each project:

Here, Fn is a volume of a cash flow during an N period, and r is a discounting rate.

The calculation of the final cost of net cash flow with the distraction of projected investments produces the following results:

NPV (A) = 54545 + 33057 — 70000 = 17603

NPV (Z) = 17860 + 23910 + 21360 + 19080 — 68000 = 14210

Even before the Internal Rate of Return (IRR) is calculated, it is apparent that it is profitable to budget the development of Project A.

These examples are not only about finances but rather about mindset in testing and awareness of the concept of their work. No matter what testing activity the team launches, it would be helpful to think of the value it can deliver to the customer and end-users. This approach is equally instrumental for both Testing Director and a freshly minted Junior. Sometimes the stereotypes that the developer is the only creator interfere with the tester’s drive to love and pamper the product. They feel like a wicked fairy who wasn't invited to a princess' christening. Instead of safeguarding the baby, she stays out fray until the kid touches the spindle and then comments spitefully: "I've told you - it has nothing but bugs." But we do not destroy; we create. One can do their job well, spend the budget, and do as much as he or she was instructed. But it will generate minimal value. To meet expectations does not mean to exceed them.

It doesn't matter whether you want to get rid of parasites or teach the bees, boost performance or rationalize your actions to the management, one thing is beyond doubt: value-driven testing, adopted as a rule by the entire team, will undoubtedly help you.