How to get the best ROI from test automation?

When we are working on a software development project, we have different actors that are involved on the process, such as the developers, the testers, the project managers, etc. With agile projects we usually deliver new product versions every 15 days, in what we call Sprints, in which we prioritize and estimate the features that we are going to deliver in that particular sprint.

As we know, each feature must be tested before we deliver it for U.A.T, so testing must be part of the estimation when we are doing the sprint planning meeting at the beginning

As we know, when a project is starting and we are in our first sprints, the amount of test cases that we need to execute to make sure that we have a good testing coverage is not impacting the estimation very high, but as time passes, and we have more and more features that are already developed, the amount of tests tends to increase as we need to make sure we have a good regression and integration set of test cases apart of the tests we created for the features of that particular sprint.

Having that in mind, we know that in every new sprint, the testing effort will be bigger and that the estimations to complete each new user story that we have in the backlog will be bigger as we need more time for testing.

The following is a common scenario that we can find on every project that is not taking care of this:

  • The amount of stories that the team delivers is lower sprint to sprint
  • As the team is having more bugs in the backlog, developers spends more time in bug fixing.
  • Team start to have a carry over of bugs.
  • The known issues backlog start to increase.
  • Developers start to deliver new features at later stages of the sprint
  • Testers doesn’t have time to properly test new features
  • Team doesn’t have a properly regression test coverage.

When projects reach this point, they have two possible exits:

  1. They hire more manual functional testers
  2. They start to automate the test case backlog

The first option can be a fast solution for the problem, and it can be a good choice for projects that are going to end soon, but if not, they will reach the same point they had in the past.

The second option will be much slower at the beginning, but it will give you a better R.O.I in the near future, as it allows you to have complete automated test suites that can be executed on demand, in parallel, and that are much more reliable than human beings for repeatable tasks, allowing human testers to focus on the execution of the test cases related to features that are still in development, for those that are more suitable to be tested by human beings (visual validations, colors, non functional, etc) and also on the execution of other types of testing approaches, such as exploratory testing.

Let’s talk about the second choice

When we decide that we need to automate our backlog, we tend to think on the test automation approach as a “Automate everything” approach, and that is an incorrect decision.

We need to know that automating a set of test cases for a feature is more expensive than manually testing it, so we need to make sure that what we are doing will give us the best R.O.I we can get, for this, we must assure that we choose the best set of features to automate:

  • Features that are stable and will not need mantainance
  • Features that will persist in time and will not be deprecated
  • Features that are really important
  • Features that needs to be tested several times

Nobody wants to expend money automating features that will need to be refactored in a few weeks, or that will not need to be tested several times in the future.

But companies tends to follow the “Automate everything” approach we mentioned above, and that is because how managers are used to measure the success of things, for example, when they ask every time: “Hey, how much automation code coverage do we have right now?” or “We need to automate the 90% of our regression test suite”.

That is a terrible idea, as we spend time and money in automating features that are deprecated, that are not adding a real value to what we are doing, instead of focussing on adding a real value to what we are, as a team, are building.

Some advice for choosing the right set of test cases to automate:

  • Make sure that the feature under test will not be deprecated soon
  • Make sure that the feature really needs to be tested several times in the future
  • Make sure that the feature is really important, prioritize your backlog
  • Amount of test cases is not important, 50 selected test cases is better than 200 random selected test.
  • Focus on end to end, do you really need to have test cases checking single UI elements presence?
  • Gain coverage, a test running on a single browser is not good.
  • Run your test cases on the cloud, avoid buying phones or hardware as it will be costly.
  • Start automating from the very begining

We hope that this article helped you if you are thinking about test automation for your project, also, in case you need help or advice, don’t hesitate contacting our team, we will be very happy to help you.

Sorry, the comment form is closed at this time.