The fourth annual State of Testing survey was conducted by Dimensional Research and sponsored by Sauce labs. A total of 1,091 technology professionals responsible for testing web and mobile applications participated in the survey, comprising a wide range of company sizes, roles, vertical industries, and geographies.
Cross-browser testing
Around 90% of respondents mentioned that there is an increase in demand to test on more than one browser as compared to a single browser. This is a slight increase from 88% in 2017.
There was also a 35% in increase of teams testing on five or more browsers.
Even though there is a significant shift in the overall number of browsers tested, the less comforting fact is that only the latest versions of browsers are tested. This can be a risky approach, as even auto-updating browsers may not have the latest versions due to organisational security policies not allowing the browsers to update. For example, more than half (53%) of Chrome users were still on older versions.
Real mobile devices, emulators or simulators
77% of organisations are presently using a mix of real devices and emulators or simulators for mobile device testing, which is a 34% increase from 2015.
The study suggested that smaller companies (15%) still depend on emulators or simulators to test applications rather than large organisations (8%) due to the fact that maintaining an on-premise mobile device lab is both expensive and a resource intensive.
Room for improvement in automation
With the adoption of Agile practices, organisations are not waiting till the end of the development cycle to start testing, as testing at the end of code development presents significant drawbacks in terms of test coverage.
The Shift Left testing approach of Agile is popular and one to be recommended. Early automation is critical in testing continuously, early, and often to provide immediate feedback to improve quality.
Even though 87% of organisations say management is on board with automation, with 45% of management planning to increase the budget for automation testing, there is still a small dip in automation, from 60% to 58%, compared to the last 2 years (as shown).
There is still scope for automated testing to increase. After all, it’s almost impossible to complete testing, including regression and build testing, within the iteration if you are not using automation. Both the Testing Quadrant and Test Automation Pyramid recommend automation testing, with guidelines to which degree it should be applied.
Agile adoption on rise
The survey reveals that 91 % of organisations have adopted Agile practices, in some form, in their organisations. This equates to a 2% rise compared to the previous survey. It has been observed that there is a steady growth in Agile Transformation over a period of four years.
The reason for the increase in Agile adoption is attributed to the complexity of software, large scope of requirements and to remain competitive in the market by reducing time to release stable working product.
Presently, Agile trends indicate that the percentage of "All Agile teams" is more prevalent in smaller (<100 employees) companies than larger organisations (>1000 employees). Typically, large organisations take more time to transform, and Agile is no exception.
As it can be deduced from above chart, once organisations where “most of the teams are Agile” - i.e. 26% (100 – 1000 employees) and 30% (More than 1,000 employees) - become “all teams are Agile”, it is clear that there will be a rise in large organisations adopting various Agile methodologies
Embracing DevOps
The implementation of DevOps practices is also on rise, with 17% respondents reporting that their organisation has fully embraced DevOps.
The increase in adopting Agile practices by organisations is also instrumental in the increase of DevOps adoption, up by 7% from previous year. Agile methodology helps organisations to release their products quickly and more frequently in the market. To support this requirement, release management needs to be modified in such a way so as to provide Continuous Integration/Continuous Deployment (CI/CD).
Defect Aging
One of the important aspects of developing a quality product is the fixing of defects as they are found.
The survey depicts that there is an increase in overall defect aging - i.e. defects are not getting fixed as quickly as compared to earlier years, a drop from 23% to 21%.
The main reason is an increase in time for defects to be fixed by developers immediately or within few working days. There is an upswing in adding the defects and prioritising them rather than fixing them as soon as they are identified.
Decline in hourly/daily build deployments
The survey’s most surprising trend is the sharp decline in organisations wanting to go-to-market quickly. Organisations are now deploying builds on weekly, biweekly, monthly and above timeframes rather than hourly or daily deploys as shown in the chart.
As we saw earlier, there are a couple of reasons for this decline in deployment, including room for improvement in automation and defect aging.
Conclusion
Based on the trends observed in this report, we can infer that:
- There is a significant increase to test applications on multiple browsers with focus on the most recent version of browser. This is a risk, as most organisations do not have the latest versions of browsers available to them. In fact, more than half (53%) of Chrome users were still on older versions.
- Testing mobile apps on a mix of real devices and emulator/simulator has increased by 77% from last year.
- Test automation has stalled, even though 45% of organisations expect to increase spending on test automation in 2018.
- There is rise in Agile and DevOps adoption from the previous year, but there is a decline in fixing defects as soon as they are reported.
- Organisations are more focused on the quality of the product rather than deploying the products early to market
Planit can partner with your organisation to help you achieve success in the above mentioned trends.
We have developed an Agile QA framework that can be tailored to meet the needs of a specific organisational or even project requirements. This framework, along with Planit’s expertise in DevOps implementation and Digital QA in-house real devices lab, will help organisations to adopt Agile and DevOps successfully. This in turn will enable high quality products to be deployed early to market.
Planit’s testers are Agile certified practitioners that have been trained in test techniques like risk-based and session-based testing. This training includes more versions of browsers in test coverage and hence, reduces the risk related to only testing recent versions for compatibility, and increases the quality of the product.
Only a third of Australian organisations typically have integrated Test Automation as a key part of their SDLC because it's hard to get right. Planit’s test automation specialists are skilled in a wide range of advanced tool sets prevalent in the industry and have helped organisations in areas like Continuous Integration, Virtualization, Parallel Execution and Continuous Delivery.
Whether it’s Agile or Automation, our experts can support your team in building effective processes. If you are starting on your journey or feel that you are not getting the value from implementing that you thought, contact us today to find out how we can help.