Regression Testing – Types and Considerations

By: Alex Chaves – GlobalNow QA Team Lead

Sign Up Below to Receive More Tips on Regression Testing

automated-testingAs a QA Team Leader who helps our DevOps automation solutions, I’m often asked by various stakeholders to explain the nature and purpose of regression testing. There is much discussion these days about leaning left with continuous integration to improve system delivery throughput and quality and regression testing is a major component of these solutions. Below is a basic description of how I view the various types of regression testing and their primary purpose.   Next week, we’ll post a brief summary explaining the typical steps used to implement these regression solutions.

So, let’s start by asking a basic question: “What is the nature and purpose of regression testing?”

A regression test is a set of test cases that are executed periodically or before a code release to validate that the latest software changes did not break any of the existing, working functionality. Consequently, if implemented correctly, regression testing will:

  • Reduce Quality issues that impact production and customer service
  • Improve the delivery speed of new functionality
  • Reduce programming rework and associated cost

Now let’s talk about the different kinds of regression testing that can deliver on the above objectives. We focus on 3 kinds: smoke, sanity, and complete regression. The first two are subsets of the complete regression. To clarify the differences, let’s use an example of a user’s module on a mobile bank application. Let’s assume we are adding new functionality specifically related to user permissions.

A smoke test is a set of test cases that exercise the critical areas of the application to guarantee that the core of the application is working. In other words, if any of the smoke tests fail, it is not worth testing the rest of the application since the “core” is not working. Think of it this way, smoke tests were often used by plumbers to test the integrity of plumbing infrastructure. Pushing smoke through pipes has a much lower potential to cause damages to surrounding areas than pushing water, so it’s a safe way to look for leaks. Related to our mobile bank example, a smoke test might include things like: create a new client, login, and check your account balance; the minimum functions that must be available for the application. This kind of regression could and should be executed automatically after every new push of code in the source repository as part of the continuous integration environment (link here). An automated smoke test provides instant visibility to core functionality problems introduced by a developer. The developer is then required to resolve the issue before being allowed to proceed with the repository update.

A sanity test is a set of tests that provide exhaustive functional testing on the application area where new features have been introduced. Sanity testing should be executed only after a successful smoke test. A Sanity test for the mobile bank application example would likely target only the specific “user module”. In our test case arrangement, there would likely be a test suite that includes a set of test cases, such as creating a user with invalid fields, changing user permissions, login with valid and invalid credentials, changing user information with valid and invalid information, and verifying app behavior. After this set of test cases is applied, you will know if the new changes have affected the existing functionality of the “user module” application area where new features have been introduced.

Finally, complete regression applies all the active test cases in our repository. Whereas Sanity focuses on targeted areas of the application, complete regression covers all areas of the application. While the targeted sanity test may prove successful, the exercising of full regression can uncover issues only detected as the components of the application interact. Complete regression provides a complete report for the application, not only specific modules.

Automating all or portion of the above regression testing can lead to significant efficiency and quality gains.  However, you should consider the sophistication level of your delivery environment before implementing automation solutions. Considerations include:

  • Are the test cases organized in a fashion that that allows targeted testing?
  • Is the application stable enough to allow targeted testing?
  • What is the rate of change, and can the regression test suite be maintained at the level necessary to support this change?
  • Are my testing tools adequate – do I have a test automation tool  that can cover the scope of the system?

If the answer to these questions are mostly “yes”, implementation of automated regression for the 3 regression types can result in significant dividends.  You can use this tool to calculate your expected ROI.

Next week, Ill cover some of the key steps for implementation of the above.

Sign Up to Receive More Tips on Regression Testing