Agile Regression Testing: How to Do It Well

Have you ever wondered what agile regression testing means and how to do it well in the IT industry? Whenever we introduce new functionality or an improvement, we need to make sure that existing functionality is not broken, which is something we achieve by regression testing. In agile, code updates happen more often so regression testing is of big importance, it must be effective, have good coverage, and prevent shipping any critical bugs to production. Today, our colleague Dalila Terzić, who is in the position of Technical Test Lead and has been in our team for more than four years, will talk about the significance of regression, which is a very important part of the testing process in order to improve your knowledge about the system under the test and have a more effective business in the QA world. We can proudly say that Dalila started her career path in System Verification in 2018 as an intern, where from the first day she showed a great desire to learn, and effectively work in teams on new projects. Dalila was assigned to one of the major projects within the company and started as a manual test engineer where, together with her great team, she was testing an eCommerce web application for their Swedish client. Their application was connected to a couple of other applications and altogether were building an interconnected system. After detailed monitoring of the development of her career path, her desire for constant learning and advancement opened the opportunity to start with test automation, where her current role is a test team lead. This time, she would like to highlight the term which is very familiar to everyone working as a test engineer, which is "regression," a very important part of the testing process for all those interested in the QA industry. This time she wants to share her knowledge and the term ''regression'' with all the younger generations and those interested in the QA world because it is a very important part of the testing process. Dalila's quality blog is an excellent representation of her work and efforts in the Bosnia team, which, thanks to the various real-life projects and experiences she gained with us, she now wants to share with a wider audience. When we had a quality conversation with her about what advice she would give to all of you and what her experience is with agile regression testing and how to do it well, she said:

''A simple definition of regression would be: a regression is a functional test performed in order to make sure that your application still works as expected after any code changes. The first step of regression testing is to decide what to take into the scope, which of your existing test cases are the best choice to run, and whether will it make a reliable regression test suite. While selecting test cases for regression, you need to consider code changes and bug fixes you recently had and include test cases related to those so that the regression scope covers possibly affected areas of your application under the test. Testing everything is almost impossible, especially if you are working with a bigger application with many functionalities. Trying to have 100% coverage in each regression would result in work that takes time, effort, and money but does not have to bring value to the team. When deciding on what to include in the regression, it is very important to be able to pull out a group of your test cases based on different criteria. A simple example would be to group your test cases based on the importance of the functionality they cover. With this, you can have a group of test cases that would make a high-priority suite that can be part of each regression. Then you can have a lower level, for example, a medium priority suite, and finally, a low priority suite, or you can create as many levels as you think are applicable for your application. You can also have a group of test cases called the Smoke suite, which would consist of a smaller number of test cases checking the overall functionality of the application, ensuring that the most basic things at a higher level work. Another example of how you can group your test cases can be an area of your application that the test case is related to. Let’s say you have an application for online shopping. One group of test cases can be related only to the cart, while another group can have test cases related to, e.g., product details, etc. From my experience, it’s better if you from the beginning include different ways of grouping your test cases, because depending on the change, you might need to pull out test cases from each part of the application, where you can use grouping by criticality, while in some other cases, you might have a change related only to e.g. the cart, where it is enough to take a group of test cases related only to the cart functionality as only that one can be affected. Grouping can be achieved by adding specific tags to your test cases, using some naming conventions, or any other field applicable to it, depending on what tool you use for your test management. In QA today, there are so many tools making the world of regression much easier. If you can easily pull out sets of test cases based on the specific criteria you need, half the job is done. You should also periodically review your test suites and remove those test cases that are no longer valid or applicable for any reason, so there is no loss of time on executing something that no longer brings value.

The second step of the regression is the test execution. Manual test execution is time-consuming. It can often appear as a bottleneck to frequent releases and slow down customer feedback. This is something you definitely don’t want, as frequent delivery and fast customer feedback are key things in the agile way of working. In my experience, the key to having fast and effective regression testing is in test automation. Automate as much as possible, starting with a suite containing the most important functionalities of your application that are going to be executed the most or in each regression, assuring that the critical path always works. You can also have the above-mentioned group of smoke test cases integrated into the deployment process so it is automatically triggered with each package deployment. This will give you an insight into application health in an early phase. However, the maintenance of automated test cases has a very important role, otherwise, it can become a source of technical debt. It is not enough to automate the test case and consider it done. Your application changes, new technologies are coming, ways of working are changing, and there are many things that can affect your test automation solution, so it is very important to have regular maintenance of your test automation solution as one of your regular sprint activities. Depending on your application and the frequency of new changes, an unmaintained solution can become obsolete in a short period of time.

As previously mentioned, it is usually not possible to have 100% automation coverage. Some test cases are not suitable for automation at all, while some can be edge scenarios, and automating those would take a long time, so the value-time ratio is not good enough, so you leave those as manual. There are always some test cases left for manual execution, and we should never underestimate the power of manual testing either. Test automation is powerful, fast, and time-saving, but it only does pre-defined checks. It always repeats exactly the same actions, while with manual testing there can be some randomization, and usually, you are able to notice if there is some strange behavior besides what is written in the test steps. If you have good test automation coverage of your application, I would suggest from time to time including some manual testing too. For example, some quick exploratory testing at least, but this, of course, depends on what is under the test and your general experience with your application.

The final step of regression would be the analysis of your test results and the creation of a test report if needed. For easier and faster analysis of automated test cases, make sure you have valuable logs added, or even screenshots included, depending on the test framework you use. Use anything that will help you get the reason for the failure quickly. You can use your test results in many ways. You can see if there are some flaky test cases or some that are not so stable and you should try to stabilize those for your next regression. You can also see if there are some parts of the application that are more vulnerable and prone to bugs, so you can increase coverage there, etc. Regression results can also help you to see if changing one part of the application affected some unexpected part, so you are aware of it for the next time you make changes in the same area. Appropriate regression results analysis can help a lot in building a more reliable, good, and stable regression process. ''

In conclusion, Dalila has tried to summarize everything said above in a couple of sentences. In summary, based on her experience and the general principles of the agile way of working, in order to have a good regression process, you should follow the steps below:

1. Make sure to prioritize your test cases appropriately and create reliable test suites;

2. Keep your test suites clean, make sure there are no obsolete, duplicate, or in any way invalid test cases so suites are ready to go at any time;

3. Choose the right automated testing tool and automate time-consuming repetitive tasks as much as possible;

4. Make sure you have a proper regression result analysis included as a step for each regression; and

5. Take feedback from your regression results and make the required adjustments whenever needed.

We want to share Dalila's success story with all of you in order to demonstrate that you can make efficient work right away and that success is guaranteed if you have a fantastic team and a strong work ethic. We are sure that Dalila's professional career path will undoubtedly advance further and bring new benefits and solutions to the QA industry. We suggest getting in touch with Dalila if you have any questions about a certain blog topic or are just starting out and want to map out your professional path since we are convinced she will be able to give you great advice.