Each stage of implementation is hard in its own way. When we start out with a new effort or process, we think of it as: Sure, it will be hard in the beginning, but itâs going to become easier as we continue. Itâs true, and then after a while a new challenge appears.
A team Iâm working with has developed quite an automated suite of unit, integration (logic integration and mocked databases), and some end-to-end test (including deployments). In addition, they are also employing manual (scripted) effort by testers.
The painful part, which originated the talk, was the continuous effort in manual regression testing. They feel they are not getting the most from their process: As the applications continue to grow, the manual efforts do not scale. They feel that in their current pace, they cannot test the applications at the quality they had before.
The other issue is that while the automated suites are doing a lot of the work, during manual testing, they find most of their actual bugs when testers step off the script. Thatâs not the problem in itself, but it raises concerns regarding their effort in writing manual test cases.
We performed an analysis on their inventory of tests.  Automated unit and integration tests run on every build, and the automated E2E tests complete in a few hours, so the automation suite returns the needed feedback in a reasonable time.
When we drilled down deeper, we saw that the automated suite can be expanded. The test plan for each feature covers both integration, E2E and manual test cases. Currently the guidance for automation of high level flows are either happy paths and easy-to-write negative paths. We concluded we can automate more test cases and reduce the number of manual test cases. This will require more automation effort, but in the long run frees time for manual testing.
However, letâs not forget that the manual tests were not as effective as-is. The bugs that were discovered were not thanks to the detailed, prescriptive steps of the tests.
We started discussing exploratory testing to increase our chance of uncovering more of these bugs. If we currently find these âsurpriseâ bugs, there are probably a lot more hiding. Only, we canât find them by thinking up front of where to look for them.
In fact, itâs what the testers did in the first place. But thereâs a lot to learn about doing exploratory testing continuously, even methodically. Many testers arenât even aware of exploratory testing as methodology. While itâs still new for most testers, what I really find puzzling, is that even when the solution is popping up naturally, we tend to stick with the tried-and-true scripted tests.
Regardless of process, thereâs always room for improvement. We need to analyze the big pain points and start solving them in a holistic manner. If you havenât already, you should start now.
Quality is too important to just leave to emerge.
| Reference: | Testing Strategy Analysis – An Example from our NCG partner Gil Zilberfeld at the Everyday Unit Testing blog. |
