ProcessOptimisationQualityEngineering
The terms quality engineering (QE), quality assurance (QA), and testing are often used interchangeably within the IT industry. Whilst there is a direct relationship between the terms, they are three distinct concepts, each offering an increasing range of benefits.
The aim of this paper is to explore the relationship between them, and to clarify:
Let’s start with clarifying the relationship between QE, QA, and testing.
Testing is the process of evaluating a system against its requirements. It is used to determine whether requirements have been met, to detect defects, and to improve confidence in release readiness. It includes processes for test planning, test design, test execution, and reporting.
Quality assurance is the process of preventing and detecting defects to ensure a system meets its requirements and to review and improve lifecycle processes. QA includes testing (defect detection), and a range of defect prevention and process improvement activities.
Quality engineering is the process of designing a quality-optimised development lifecycle that builds quality into each system, and that determines whether quality is present, as early and frequently as possible. QE includes and goes well beyond QA and testing, incorporating a range of requirement, design, architecture, development, test, release management, operations management, production management, and maintenance activities that shift quality to the left and to the right, across the full software lifecycle. QE uplifts quality across the lifecycle, and utilises a range of tools, techniques and metrics to optimise and continually improve the process.
Figure 1: Relationship between QE, QA, and Testing
Another way to illustrate the differences is to consider the practices that can be utilised within QE, QA, and testing. As this diagram shows, testing is one of many activities that can be applied to improve software quality.
Figure 2: Relationship between QE, QA, and Testing
How does transitioning from testing to QE benefit our industry? Put simply, you can’t test quality into a system – you can only build it in!
If companies focus their quality budget on testing after requirements have been written, and as their only mechanism for improving software quality after development has commenced as their only mechanism for improving software quality:
In short, it places too much emphasis on testing, when software quality improvement comes from creating better quality requirements, design, architecture, code, configuration, release management, and production management practices.
Through adoption of QE practices, our investment in quality shifts to the left and to the right across the full lifecycle of each system. Software development teams can “get it right first time” by engineering products that meet their requirements faster and with less cost. The goal is to:
QE distributes the process and the responsibility for quality across the full lifecycle, truly making quality everyone’s job. QE It is also about planning for and evaluating quality at every step of the lifecycle.
QE includes testers, business analysts, developers, architects, product owners, project managers, release, operations, the business, and end-users in the process of evaluating quality throughout the lifecycle. In doing so, it optimises feedback loops, reduces rework, reduces development costs, and accelerates product delivery.
A critically important aspect of shifting left is engaging quality-minded employees like testers early in the lifecycle, including in requirement reviews, design, and architecture reviews, and code reviews; quality coaching for the team; product quality risk assessments; and earlier planning and strategising of QE activities. Engagement should be early and upfront, while requirements are still being written, and ideally while business cases and vendor contracts are still being written. This enables an up-front investment and improvement in quality, and ensures the budget, schedule, resources, and delivery strategies include QE practices that allow the organisation to build quality in and achieve their desired quality outcomes.
It is also important to adopt shift-right activities for QE, such as dress rehearsals, dry runs, release readiness reviews, and the uplift of release and operations management and maintenance processes.
Testers possess inquisitive, questioning minds. They spend their working lives considering how systems might fail, and working backwards from that, to understand how software and artefacts can be reviewed and tested to improve quality. They also often have skills in writing requirements, doing system design and coding, and are often skilled at writing documentation.
Testers are also skilled at discussing quality with stakeholders from across the full lifecycle, from business analysts to architects, developers, product owners, project managers, business representatives, operations and beyond. They often already have a holistic quality mindset. Testers are therefore well-placed to transition into roles in quality engineering, that support and enable the enhancement of software quality across the full lifecycle.
Not only does transitioning from testing to QE support each organisation in achieving its software quality goals, it also enriches the skillset of each tester, which can be very motivational and uplifting. This makes each tester more valuable to their teams, organisation, and industry.
Transitioning from testing to QE requires a change in the language and the mindset of our IT industry.
Currently, when a new system is built, it is common for testers to be asked to write a test plan and strategy that describes how they will test the system. Such artefacts usually have very limited coverage on how the quality of requirements, design, or architecture will be reviewed or improved. Typically, there will often only be a very brief mention of unit testing being the responsibility of the developers, with little to no responsibility for quality or testing spelled out for any other phase or role across the lifecycle.
In today’s fast-moving digital world that embraces multiple platforms and Software-as-a-Service (SaaS) products, testers are often brought in to plan and strategise the testing after most requirements have been written, after a new platform has already been purchased, and just prior to configuration and customisation. Even in Agile environments, by the time testers join the team, it is common for much of the product backlog to already be built into the chosen system, and for the product to have already been purchased. By then it is too late to identify and resolve early quality issues in the requirements, design, and proposed solution, shifting the investment in quality to too late in the lifecycle.
A much more optimal way to build software is to assign quality engineers to facilitate an early whole-team approach to strategising and planning quality and testing across the lifecycle, by creating a “product quality plan” (rather than a test plan). In addition, the organisation itself should adopt a “quality policy” (rather than a test policy) and an “organisational quality strategy” (rather than a test strategy). This enables the full team to take responsibility for the quality and testing of each system, and allows everyone to actively contribute to the discussion on how to ensure quality is built into each product.
Throughout our industry, we should create artefacts that focus on quality, rather than testing alone.
Figure 3: Changing mindset and language
Key drivers for transitioning from testing to QE are the time, cost, and risks saved, and the improvement in software quality that can be achieved. A range of metrics illustrate this.
The Standish Group have been reporting on project success and failure rates since 1994, for over 50,000 projects in total, and an average of 5,000 per year. Throughout this time, the number of projects that have succeeded (on time, on budget, meeting requirements), been challenged (over budget, over schedule, and/or not delivering required features), and failed outright have not changed significantly.
Figure 4: Chaos Report Project Resolution Rates (Standish Group, 1994 to 2020)
Two of the top reasons reported by the Standish Group for project failure are incomplete requirements and a lack of user involvement in IT projects.
A separate review of over 5,400 projects by McKinsey and Oxford University found that within their study:
According to the Project Management Institute, 39% of companies that experience project failure cite inaccurate requirement gathering as primary cause of failure. In addition, a recent study by the Consortium for Information and Software Quality (CISQ) found that in 2020, the total cost of poor quality in the USA alone was $2.08 trillion, on top of another US$1.31 trillion in technical debt residing in severe defects yet to be corrected.
Add to this the oft-quoted statistic that a defect found in production is 30 times (and up to 100+ times) more expensive to identify and fix than a defect found in the requirements phase. The goal of QE is to invest more in finding defects much earlier in the lifecycle, so that the overall cost of quality and delivery is far less overall.
Figure 5: Cost of defect repair (NIST), with the comparative goal of reducing the overall cost of quality via QE
Another way to look at the economics of quality is to consider the effectiveness of early lifecycle quality improvement practices such as requirement, design, and architecture inspections, compared to later-stage approaches such as testing.
According to Capers Jones, later-stage approaches like testing are effective at identifying and supporting the removal of around 30% to 40% of defects respectively. Compare this to early-stage approaches like requirement, design, and architecture inspections and pair programming, which have defect removal efficiencies of 75% to 87%.
This doesn’t mean that all we need to achieve quality are early-stage QE approaches. What we do need are a range of QE approaches across the full software development and delivery lifecycle, that will help reduce defects, reduce, and rework, and accelerate delivery to production.
Figure 6: Defect Removal Efficiency of a variety of software engineering best practices (Capers Jones)
Engineering high-quality systems starts well before software product selection, design, development and testing starts. Ideally, it will begin at the ideation stage, when the business case and requirements are being specified, well before software product selection, design, development and testing starts.
As soon as the requirements start being specified, a quality engineer can be used to facilitate whole-team discussions on:
Balance the wish list against the available budget, schedule, and resourcing, by discussing the risk of not implementing each QE practice. Any time a recommended QE practice is deemed not affordable, ask your development and/or operations teams, steering committee, and/or business and executives what the cost of not addressing each risk is. Consider the potential impact each risk could have on your company, employees, customers, end-users, shareholders, industry, Government, and the general public.
Also consider the potential human, business, reputational, and cascading impacts. Ensure risks and their treatments are reviewed by senior stakeholders, managers, and executives in your organisation, and check whether they are willing to accept any risk that is not satisfactorily treated/mitigated.
Rather than saying, “cost, schedule, or quality – only pick two”, companies should seek a balance between all three, with the material risk and impact of not delivering quality being the deciding factor on how to achieve that balance.
You can’t test quality into a system – you can only build it in. To that end, there are a range of QE practices you can utilise across the lifecycle to improve the quality of requirements, design, architecture, code, testing, release, and operational management of software systems. These can be utilised via a “whole-team” approach to quality.
QE practices can determine whether quality is present earlier and more frequently across the lifecycle, and ensure that quality truly does becomes everyone’s responsibility. Therefore, investing earlier in system quality will not only help to reduces the overall cost of delivery, and it will also enables systems to be released to production sooner, with much greater confidence, and with far fewer production defects and failures.
Practice Director - Quality Engineering and Assurance
We use cookies to optimise our site and deliver the best experience. By continuing to use this site, you agree to our use of cookies. Please read our Cookie Policy for more information or to update your cookie settings.
