Technology fragmentation imposes testing burden on banks

As banks increasingly digitalise and provide mobile services to user’s mobile devices they face the double-whammy of having to test software that has to run on multiple environments and doing it in an increasingly short time-scales as continuous release development cycles become the norm.

Some of the problems that UK banks faced in getting ready for the launch of Apple pay were caused at least in part by this issue.

TestPlant, whose eggPlant software test automation tools are used by Banco Sabadel, Citi and Nationwide, among others, says that its data indicates software quality departments in the banking sector are facing scalability challenges when testing applications due to the ever-growing range of devices and browsers. Conflicts between the need to roll out new applications and updates quickly, and the need to adequately test these applications, present an insurmountable challenge if manual testing practices continue.

Antony Edwards, chief technology officer at TestPlant, said the problem is getting worse as more and more banks adopt Agile and DevOps development cycles that require continuous release of production products in a “fast-fail” culture. “Two or three years ago, I’d have said fragmentation was peaking, but now we have new versions coming along all the time.”

Putting the scale of the problem in context he cites a report on Android devices from OpenSignal, which reported last year that it had recorded more than 18,000 distinct devices running the Android platform.

The company recently investigated the testing challenges faced by companies in the financial services sector, concluding that technology fragmentation is one the biggest software testing issues in banking today as companies’ online and mobile interactions with customers are increasingly seen as a way to differentiate competitively.

TestPlant’s data indicates that a typical banking application (web or mobile) today will require some 200 test scripts to be created and run for each release. In today’s highly fragmented consumer technology landscape, banks’ quality assurance teams are generally choosing to test 15-30 of the most popular device/OS/browser combinations – for example, Internet Explorer on Windows 8, or Safari on an iPhone 6. Assuming just 15 combinations are tested, the time taken to manually test a typical release is more than 380 tester-days.

Subsequent releases typically add 10% more functionality, but in a purely manual testing environment these have to be fully tested again. This increases the time required to manually test each new release to 422 tester-days.

“In the past, companies were only testing on very small numbers of device and browser combinations, which would take a much smaller number of tester-days for each release,” said Edwards. “This meant that banks could reasonably expect that a fully manual testing approach was enough, but that is no longer the case.”

As fragmentation increases, all of these numbers will also increase and it doesn’t take long to realise that testing departments will find it impossible to scale up manual testing processes. In five years’ time, TestPlant estimates the number of testing combinations is currently approaching 60. This would see the manual testing effort for an initial release rise to more than 750 tester-days.

“The problem is linear – for every hour it takes to manually test an application on one phone or browser, then testing more phones/browsers multiplies the time required,” said Edwards. “Banks IT teams are struggling with this because they can’t just keep hiring more people to absorb the impact of fragmentation – but their marketing and customer service departments can’t afford delays.”

To effectively test a banking application, whether it be a mobile app or web-based interface, banks ideally need to test on as many devices and browsers as possible, but typically they only choose the 15-30 most widely used technologies. In the world of manual testing, where tests are conducted by human beings, even this fairly small cross-section presents a huge testing challenge.

“The scalability question relating to the impact of technology fragmentation on software testing is undoubtedly addressed by test automation,” said Edwards. “Manual testing has its place, but banks who want to stay ahead of the competition when it comes to the quality of their apps need to consider the right mix of manual and automation testing.”

TestPlant’s data shows, for instance that in the previous example where 15 combinations are tested, using automation for the first release cuts the testing effort by almost 75% and by around 95% when testing subsequent releases. As fragmentation continues to grow, the case for test automation increases.

“Test automation used to be all about saving time when testing new releases, but browser and mobile fragmentation add a whole new dimension,” said Edwards.