by Alex Chaves – GlobalNow QA Team Lead
This is a follow on to my previous discussion on Regression Testing – Types and Considerations.
A critical first step for any automation initiative is to create test cases, ideally using a test management system. Those test cases should be organized by system function in a fashion that will allow users to test specific areas of the application as needed. There are many test management tools, both open source and commercial,
that can be selected based on your requirements and budget.
Picking a suitable testing automation tool is the next very important consideration. Along with available budget, the technology used by the targeted systems and the endpoint devices of the supporting interfaces are the most important consideration when selecting the automation tool. There are a variety of commercial and open source tools that can be leveraged. Keep in mind, however, that open source is not always the cheapest total cost of ownership. As the application changes, the tests themselves change and therefore automation must keep pace. The frequency and volume of application changes is a key component to the total cost of ownership equation and should heavily influence your choice of tool.
For example, as system functionality changes, Selenium (an open source tool) requires test engineers to update each element in multiple locations since every test case instance is hard coded. Commercial tools such as Ranorex record the case objects/scripts, assigns a unique ID in a repository for each instance, which in turn allows the test engineer to update the repository once so all the impacted test cases can use the same object.
Another major requirement for automation is having a dedicated environment for the automation to run against. You should guarantee that all automated tests will run in an environment consisting of production-like components (browsers, operating system performance, without antivirus, internet connection speed, etc), which avoids generating inconclusive test results or false positives. Having an environment that reflects an appropriately scaled version of production is always the best approach.
Testing Automation generates additional responsibilities for the QA team. These include implementation of the automation scripts, triggering the execution of the regression (if it is not automated by a continuous integration tool) and checking and analyzing regression results in order to find actual bugs. Analyzing results is one of the most important tasks. Determination must be made on whether the error(s) reported is a real error or a false positive. This requires intimate knowledge of the system and supporting business process. Of course, if it is real, a ticket must be created and development should be notified. Otherwise, if it is a false positive or if the test case is out of date, updates must be made to avoid future failures. This is part of the maintenance cost and effort associated with automation.
The initial test automation project can sometimes be a significant challenge because the quality assurance team will most likely be maintaining ongoing manual testing to service the software delivery process while creating the overall automation environment. Over the long term, there are typically two main benefits. The first is labor savings (both developers and QA personnel) gained from the efficiency of automation over manual testing. The second is the reduction of damage from costly production defects and associated impacts to customer service. The ROI calculation should consider the expanded costs in the early stages against the realized cost reduction and the very real cost avoidance on defects and reputation in the longer term.
In conclusion, if you are considering testing automation for regression, keep the following in mind:
Regression
- Select a test management tool based on your specific requirements.
- Document, organize and prioritize your test cases within the testing management tool.
- Group test cases by smoke, sanity and complete regressions base on priorities.
- Understand when to execute smoke, sanity and complete regression depending on circumstances of the project.
Automation
- Determine if the software and your organization has enough maturity to merit automation.
- Estimate the ROI of applying automation in your project/company.
- Evaluate the most appropriate automation tools for your application and select one.
- Setup a clean and robust environment for automation execution.
- Ensure that your test environment mimics the intended production environment.
- Clearly Define automation responsibilities among your team.
- Measure impact from automation – and adjust as needed.
SIgn up