ABSTRACT: The purpose of this whitepaper is to answer the question “Will automating regression suites save us time?”, by providing examples of success and failures with projects that I have worked on. It will also aim to provide the pain and benefits of automating regression suites.
Starting Point
This question is often tossed around from project to project, and most of the time we find ourselves asking “can this be done faster and more efficiently?” Most of the time the answers we get are “it’s too hard”, “there are so many factors that are in play”, “it’s too expensive” or “we need to upskill and train personnel”.
So is having a repeatable automation suite the answer? Or can it be that having repeatable manual test scenarios that are high level enough to go through business functions more acceptable?
The solution chosen must take into consideration the risk attached to the business function and the level of coverage the function must have. The question of what to automate or what to place in a regression suite needs to be discussed with both the business and project teams so that technical areas that are at risk are also covered.
Automation can provide immense benefits such as time reduction in test execution, costs in running tests (once the suite has been setup), improvements to quality due to exact repeatability, little human interaction once running, reusability for future projects, ability to run at any time on multiple environments and the most common benefit removing the mundane and lengthy process of performing manual regression suites. These automated regression suites can provide companies and project teams a cost effective and fast way to run the suite after each enhancement or change. It is also important to note that the cost of repeating this automated suite is relatively constant.
However, providing repeatable automated regression suites is harder than it initially seems. Yes the benefits are great but the outlay and continual maintenance and/or tweaking may prove to take longer than running the manual suite with a team who has experience in the functionality of the system, and in some cases it may prove to be more costly than running the regression suite manually.
There are many frameworks, models, and software packages that can aid the analysis and design of automated test cases. However these solutions almost always require in-depth knowledge of the software being used to create the automation suites. So, when designing the automation suite, it has to be flexible enough to adapt to changes, remove existing functionality when needed (i.e. functionality that is no longer suitable for regression) and add new functionality.
There is no wrong or right answer, and the question to automate must be made on a case by case basis. This decision must take in all factors that can and affect the application, it will also depend on the test environment and many other variables such as the automation package, training, data, and the skill of personnel to name a few.
Maybe the solution is to meet halfway with both automation and manual approaches to regression suites? Can this be done?
To better understand the complexities behind the question to automate or not to automate, I have given you some examples of some projects that I have worked on and what we have done. The following projects will describe the issues and/or whether the automation was successful.
Case Study 1: BankOne
BankOne is an Online Banking system that allows customers to perform day to day transactions (such as transfer funds / pay anyone / international money transfers etc). BankOne had major and minor releases to enhance the application; these releases included mobile banking, unified login and exposure of services.
The regression suite consisted of 400+ test cases that covered core application functionality. The project team initially automated the regression suite for the first release, however found that during subsequent releases the regression suite took longer to update and run, when compared to having a team of 4 executing the 400+ test cases in the 1 week cycle.
As the releases progressed the automated regression suite was abandoned and the manual run was favoured. This was widely accepted by all project teams as the best solution in the ever changing nature of the application. So why did automation fail here? Let’s look at a few factors:
- Application was in constant state of change
- Automated regression suite designed around the first release
- New backend procedures
- The manual team possessed sound knowledge of the core functions
- Data issues
The defects raised by the automation suite were mainly centred on data and missing objects variables. The turnaround time in investigating, analysing, creating new data and finally updating test automation scripts was often longer than running the manual suite.
With each release containing new core functions such as unified login, it became harder and harder to implement an automated regression suite.
Case Study 2: BankTwo
BankTwo is an Online Banking system that allows customers to perform day to day transactions. BankTwo had minor releases to enhance the application; these releases included new products, changes to backend and communication protocol changes.
BankTwo’s online system is being replaced with a new platform and backend in the next couple of years, so the current online banking system was stable with only minor GUI and backend changes being made. With each backend change the regression suite needed to be run.
BankTwo has been able to successfully implement a repeatable automation suite using two products: Quick Test Pro (QTP) and Selenium. The QTP suite was used when minor changes were made such as new products and/or GUI changes; this suite contained 60 test cases covering core payment functionality (accounting for 30% coverage). There was an overall 95% pass rate for QTP runs; the 5% failure was due to data and changes in the test environment.
The Selenium suite was used for backend changes and when deemed necessary to regression test all core functionality. This suite comprised 500 test cases covering payments, two factor authentication (accounting for 50% coverage of core functionality), changes in personal details and different user profiles. The pass rate varied per run for Selenium from 70% – 85% depending on the environment. In most instances the failed test cases were rerun manually by the functional testing team. The automation suite also reported non data / non environmental defects.
With every Selenium run, the team needed to tweak and maintain the test cases, as well as manually run the tests that failed. This combination of manual and automation worked, as it enabled the Selenium team to update and maintain the scripts as well as provide a suite of test cases that could be run on any environment on any code base.
The time taken to run the Selenium suite was roughly 3.5 to 4 hours, whereas to execute the suite manually it would have taken a team of 6 to execute 12 test cases each day for 7 days. The following table below describes the time taken for the various success rates to manually test the failed and no run automated scripts:
Swipe to see more
% Run
|
Total Executed and Passed
|
Total Remaining
|
Time Taken to Manually Execute
|
Number of Team Members
|
0%
|
0
|
500
|
7 days
|
6
|
50%
|
250
|
250
|
3.5 days
|
6
|
70%
|
350
|
150
|
2.5 days
|
6
|
75%
|
375
|
125
|
2 days
|
6
|
80%
|
400
|
100
|
1.5 days
|
6
|
85%
|
425
|
75
|
1 day
|
6
|
90%
|
450
|
50
|
< 1 day
|
5
|
95%
|
475
|
25
|
<0.5 days
|
2
|
Table 1: Percentages of tests run by the automated suite vs. the time taken to manually execute
It is also a good point to note that not all automated products take the same time to run and create test automated cases.
The above case studies refer to medium and large businesses with little or constant change. The next questions that can be asked are “What about small businesses? Should we automate?”
Small businesses are in a unique situation, if the application that they have is a COTS solution then, regression testing updates will be performed by the vendor. However if there is an in-house built application, there may be scope to create an automated regression suite. For example, if the small business application was an interest payable calculator on funds borrowed, it may make sense to automate the tests as the variables (interest rate, amount borrowed, and length of term) are the only variables to change in the calculation, thus an automated suite can be run to ensure that a subset of combinations to calculate interest payable is correct.
However not all applications will be as straight-forward as an interest payable calculator. The following case study below outlines how using some concepts from the agile methodology can also produce repeatable testing.
Case Study 3: eTimeSheets
eTimesheets is an application that allows employees to enter their hours booked on projects or bench time online. It allows different codes to be allocated to the time being booked, such as annual leave and sick leave.
eTimesheets goes through minor updates a couple of times in a year. When these updates occur, regression must be run to ensure that the system is still available and not impacted by the update. The teams were only given 3 days till deployment to have the application tested.
The testing conducted utilised the agile methodology, where the user stories provided a basis for creating the high level session sheets. These user stories contained the business rules and actions for the particular function. The session sheets provided a high level scenario (charter) and minimal steps (test ideas) to complete the scenario. Regression session sheets created in an earlier release were used to test the update.
Test execution was conducted fairly quickly with the test cycle spanning only two days to cover 20 stories. The testing was conducted by staff that understood the main functionality of the system, and thus they were able to follow the high level session sheets without the need for detailed test cases.
Pain and Benefits of Automated Test Suites
The following table describes the pain and benefits of automated test suites:
Swipe to see more
Pain |
- Data setup
- Environment variables
- Complexity of code
- Constant changes to code base
- Training required to upskill users
- Complexity of automation packages
- What to automate?
- Troubleshooting issues that arise
|
Benefit |
- Repeatable
- Easy to run
- Run at any time
- Cost effective
- Discovering defects or issues before code is placed in production
- Suite can be extensive
- Ability to update
|
One of the biggest issues facing automation teams is the question of what to automate. Selecting scenarios and/or functionality to be automated is a huge task as the functionality selected may not be suitable to be automated. For example, the add new client function for a doctors surgery application, requires the user to fill in multiple screens such as contact details, medical history and health fund details. Automating this process may be hard as the automation script needs to go through many screens which contain many combination of variables.
The case studies provided above depict the different types of results from using automated or manual regression suites. If the application in question is one that is undergoing constant change, as the BankOne case study, having an automated regression suite may not be the answer. The time taken to maintain, analyse, update and resolve issues could be more than having a team manually executing the regression suite. However, if the application in question is not undergoing constant and major changes, BankTwo’s approach may be appropriate. BankTwo utilised two automation packages that enabled them to cater to the change, and thus appropriately regression test the change at hand.
If automation is not appropriate for the application in question, maybe using the concepts of session sheets and high level test scenarios is the way to go in creating a reusable and repeatable regression suite.
To answer the question “Will automating regression suites save us time?” the answer is yes and no. It depends on many factors such as the level of maturity of the application, the rate of change, return on investment, and the skillset required to create and maintain the automated regression suite.