Software defects continue to make their way into production, despite efforts to detect and resolve them before they are released. Unfortunately, bugs in production software can have a negative impact on the business, regardless of severity.
Even low-impact defects can have a cumulative effect on your brand. But according to a new Harris Poll study of 200 developers and managers in large software suppliers, organizations that use agile and DevOps methodologies have reduced the number of production defects, and the defects that do get through are less serious.
Here are several key takeaways from the report, as well as practical tips to help you reduce both the number and impact of production defects.
Agile/DevOps is better than waterfall
Agile software development grew out of frustration with software development methods that had a significant time lag between defining the software's objectives and delivering the final product.
Developers were looking for a way to build working software quickly, and get feedback from users that could help improve the product. This was win-win, because users saw value early on, and developers received faster feedback, which helped them refine their focus.
In agile and DevOps, testing happens earlier, so defects are identified and resolved sooner. While some bugs will continue to escape into production, there are fewer than in waterfall software projects. Furthermore, if you happen to discover a defect in production, you can fix it faster when it's an agile or DevOps project.
Of course, a more definitive study would be obtained by repeating the same project in both agile and waterfall, and comparing the results. But that doesn’t happen, certainly not in industry. Therefore, it's not really possible to state conclusively that there would be fewer production defects in a specific project if the project was done with agile as opposed to be waterfall.
However, it is possible to look at the impact that production defects can have, which is what the Harris poll did.
Three different levels of defects
The survey categorized defect impacts into three levels: low-impact, defects with little or no impact on users or revenue; medium-impact, which affect some users or activities; and high-impact, which have a widespread impact on users, affect key activities, or cause loss of revenue.
Respondents were asked to estimate the distribution of defects found in production across these three impact categories, in waterfall and agile/DevOps projects.
In waterfall, there was an approximately even distribution across the different categories. On average, 38% of defects had a low impact, 32% had a medium impact, and 30% had a high impact. Agile/DevOps projects have a similar proportion of medium-impact defects (33%), but there is a clear tendency toward more low-impact defects (40%), and just 27% of production defects have a high impact.
(Authors' note: While the sample size here is small, and the analysis is representative only of those who completed the survey, these results should still be of interest. We plan to follow up with a more comprehensive study in the future, with a much larger sample size.)
While agile and DevOps do not eliminate defects from appearing in production, any defects that do make it into production are more likely to have a low impact.
No defect is insignificant
Realistically speaking, there will always be defects that escape into production. But is it worth addressing low-impact defects, given that by definition, they have little or no impact on users or revenue?
The survey presented several statements to respondents, and asked them to rate their level of agreement with each. Some 84% respondents agreed with the statement that "a single low-impact defect usually has a low effect on brand reputation, but many low-impact defects may have a significant effect." At the same time, 47% of respondents considered a defect worth fixing only if it caused an outage or data loss.
The lesson here is that an isolated low-impact defect might not be worth fixing. But if several defects are discovered in production, they may have a cumulative, detrimental effect on your brand's reputation even if they don't disrupt business continuity.
Security risks contribute most to the cost
People often talk about the cost of defects, but rarely do they quantify that cost. The survey listed factors that might contribute to that cost and asked the respondents to rate them. These included:
- Security risks
- Severity level
- Loss of business
- Damage to reputation
- Resources required to fix
While each factor contributed in some way to the overall cost, 61% of respondents said that security risks contributed the most.
The survey categorized defects into three areas: functional, performance, and security. Respondents were asked to rate the contribution of a few elements on the business impact, for each of the defect categories. These elements included:
- Revenue
- Brand reputation
- Regulatory fines
- Disruption to the team
- PR and marketing costs
In waterfall, functional defects mainly affect revenue, while performance defects can result in regulatory fines. Security defects, however, affect both almost equally.
In agile/DevOps, security defects affect revenue and brand reputation more than anything, and these have more of an effect than do functional and performance defects.
The survey examined these three categories of defect in detail, as well as the impact of defects discovered at different stages of the lifecycle, for both waterfall and agile/DevOps scenarios.
Go agile and shift left
There is a widely held belief that defects discovered in production are more costly to fix than if found earlier in the process. The data in this study supports this, and while agile/DevOps does not prevent defects from reaching production, those that do get through tend to be lower-impact defects than those produced when following a waterfall development methodology.
Also, it takes less time to fix defects in agile/DevOps than it does with waterfall.
Here are the key takeaways:
- Fix known low-impact defects before release. Don't ignore them. They may have a cumulative effect on your brand's reputation.
- Go agile/DevOps as soon as possible. You'll have fewer production defects, and the ones that do make it out there will have a lower impact. In addition, you will be able to fix them faster.
- Shift testing left. The earlier you test, the sooner you find defects, and the quicker you can fix them.
- Prioritize security testing. Security defects have the biggest effect and the biggest impact on revenue.
Follow these recommendations and you'll help your organization reduce the number of defects that escape into production, and mitigate the negative affect of defects on your brand. To learn more, download the study here (registration required).
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
