Organisations that succeed in business transformation via the adoption of Agile and DevOps best practices exhibit a similar pattern – they achieve speed with embedded quality, increase the frequency of their production deployments, and build fast feedback loops that allow them to rapidly react to shifting client and market demands.
Underpinning their business goals is the ability to produce quality products at speed. As such, Continuous Testing and Automation practices are key ingredients to successful business transformation.
This is supported by the latest Accelerate: State of DevOps Report 2019, in which quality metrics around speed to market and operational reliability clearly illustrate that automation provides a foundation for successful business transformation.
As examples, high performing organisations deploy code between once per hour and once per day, compared to poor performing organisations that release every one to six months at a minimum. In addition, it is the change failure rate (% of time a change to your system produces an issue) that provides a strong indicator of product quality, with high performers reporting 0-15% change failure rate, compared to 46-60% for low performers.
For these reasons, speed and quality are the two most important attributes of Continuous Testing.
In this article, we will highlight what is continuous testing and why it accelerates delivery, why technical leadership is so important, common challenges in continuous testing, and some techniques to help.
Continuous Testing as an accelerator
Continuous testing enables more successful business transformation because it is one of the most effective and rapid ways of ensuring quality is built into each release. This is demonstrated in a strong and continuing uptake of continuous testing practices, including in organisations undergoing transformation.
Consider that 92% of organisations report that continuous testing helped them achieve fast and continuous feedback on defects, which is needed for accelerating testing and delivery to production. Also, 58% utilise continuous testing to detect defects earlier in the lifecycle, while 80% report it has led to fewer reported defects (Soliman, 2018), key ingredients for shifting quality to the left and reducing the cost of delivery.
Continuous testing is also delivering tangible results to those who do it properly, with 84% of organisations reporting that continuous testing contributed to reduced release cycle times and increased test coverage (Soliman, 2018).
There are many viewpoints on what continuous testing is, with many organisations treating it as only continuous regression test automation. It's better to view it as a process that embeds testing and quality across the entire delivery lifecycle, utilising tools to streamline testing and building quality and speed into each product.
It is also about achieving continuous test automation in not just functional testing, but non-functional testing as well – in critically important business areas, such as security and data privacy, performance and accessibility.
Technical Test Leadership
Modern development uses a multitude of software architectures from monolithic to microservices to event-driven and nowadays serverless architecture. Coupled with the varied levels of software models and platforms such as SaaS, Commercial Off the Shelf (COTS), legacy applications, middleware, IoT, the landscape for designing an effective test strategy that can stand the test of time is challenging.
Organisations will typically invest significant time and money to design roadmaps and portfolios for system and technology architecture, but will often not consider investing the same amount of time to design the right test architecture. With different development strategies at play, this invariably requires a smarter approach to automation and tooling solutions in order to achieve efficient and effective continuous testing solutions.
Evidence of this is that many organisations still focus much of their automation testing effort on functional regression testing, typically with the majority of tests at the UI level. Supporting this view is several industry reports, which have found that as many as half of organisations have an inability to apply tests automation at the right level.
To resolve this, organisations need to appoint strong leadership in test automation (and testing in general!) to ensure the right strategy and plan is in place to produce the desired level of quality, while at the same time reduce testing cost and speed to market. Ensuring your organisation has technical strategic oversight in quality and testing will enable you to:
- Achieve test design at the right level.
The well-known testing pyramid is a good guideline for achieving the right balance in test architecture. By designing tests based on your entire application architecture and not just the UI layer, you can get the most out of your automated testing approach.
For example, API testing is quicker, easier and less brittle and costs less to build and maintain.
- Ensure that automated testing is not limited to functional regression.
Automated performance, security and accessibility testing are common non-functional test types that, when implemented earlier in the lifecycle and integrated into CI/CD frameworks, further accelerate time to market and improve quality of delivery.
However, many organisations do not focus on automated non-function testing, with surveys finding that only a quarter of organisations use automation for performance testing. Businesses often lack a clear understanding of the level and load their systems may experience in production, leaving them vulnerable to failure or severe degradation, which are significant contributors to a poor customer experience.
Continuous Testing is as much about process and culture as technology
As much as technology is a key factor in successful Continuous Testing, organisations must also commit to cultural and process changes that enable it. Simply adopting tools to automate test cases created under a manual test approach will often lead to technical debt and inefficiencies down the line.
There is clearly a need to provide technical leadership and embed the right skills in teams. Thus, it is important to recognise Continuous Testing as an ecosystem of tools and processes that are designed to fit a goal of releasing software quickly and maintaining a risk profile that is acceptable to your organisation.
Common challenges in continuous testing
Two of the most common challenges that occur in product delivery that impede an organisation’s ability to achieve continuous automation testing are:
- Lack of skilled automation staff.
This is a key prohibitor to achieving continuous test automation, with as many as half of organisations listing this as their biggest challenge. This often leads to a culture of individual heroics, reliant on one or a few individuals to complete the work, resulting in excessive wait time and slowing down delivery.
Engaging automation specialists also often occurs too late in development, with 78% of organisations reporting that testing starts too late in their lifecycle (Panaya, 2017).
- Inadequate test environments and test data.
Research shows that over a third (36%) of developers/testers find themselves waiting a week or more for access to development/test environments, while up to half of organisations report challenges with test data/environment availability.
The demands on both test environments and a reusable data source are very high when tests are run frequently and often. Organisations must move away from adopting the same strategies for data and environments if they are to create the foundations for continuous tests.
When these challenges exist and are not adequately planned for and mitigated, the effect is a lack of in-sprint automation. Teams that cannot complete their automation in-sprint often find they do not have enough time for testing, with applications changing too fast for testers to keep up.
Industry research often shows that more than half of organisations find it difficult to automate because applications change too frequently with every release.
However, having an ability to respond rapidly to change is often a key goal of business transformation. In order to support this businesses transformation goal, these common automation challenges must be overcome.
Achieving Continuous Testing
Automation is an enabler of Continuous Testing, but it comes with certain challenges that need to be overcome in order to ensure automation can support Continuous Testing across an organisation. To tackle these challenges, here are some best practices:
- Develop a Continuous Testing Roadmap.
Create a plan with clear goals and metrics to baseline and measure success of activities required to introduce or change tools and processes to better support automation.
Functional test automation is a great first step on the Continuous Testing journey, but it’s not the end state. Organisations must look at automating other types of testing, such as security, performance and accessibility, to further accelerate time-to-market.
- Build a capability and skills matrix.
The right level of competency across each of the activities is required for effective Continuous Testing. This aids with identifying gaps and mapping out a plan to drive capability uplift across the entire Continuous Testing pipeline.
- Design a test architecture.
Testing needs to be extensible for each application within your organisation, with the understanding that there will be changes to tools and processes to cater for differences in projects and technologies. Ensure good test design principles are adopted and tests are created and executed at the right architectural levels.
- Invest the time to plan out your test environment.
This also needs to include test data creation and management strategies. Identify opportunities for Service Virtualisation where data can be reliable, repeatable, and exhaustive.
As your organisation matures, optimise continuous testing by investing in the DevOps toolchain and creating infrastructure as code to further support test and developments teams.
- Consider the overall delivery process.
Look at how quality is continuously tested. Make changes to traditional test strategies to cater for automation.
Prepare for your transformation
Whether you are at the start of your digital transformation journey, or in the process of undergoing one, the above insights should provide a high level view of the needs, challenges and activities for well-executed lifecycle practices of agility, quality engineering, automation and continuous improvement as an outcome of business transformation.
If you are facing some of the challenges mentioned above, contact us to discuss a delivery optimisation review and see how we can help streamline your continuous testing pipeline.