Banking sector faces overwhelming testing burden due to technology fragmentation

  • Posted by JB Brockman
  • On September 30, 2015

TestPlant analysis shows that traditional software testing practices are becoming unsustainable as device and browser numbers multiply

 London, UK – September 30, 2015 – TestPlant, the maker of the eggPlant range of software test automation tools, today announced that its data indicates software quality departments in the banking sector are facing scalability challenges when testing applications due to the ever-growing range of devices and browsers. Conflicts between the need to roll out new applications and updates quickly, and the need to adequately test these applications, present an insurmountable challenge if manual testing practices continue.

TestPlant counts many large financial services institutions among its customers, and recently investigated the testing challenges faced by companies in the sector. Technology fragmentation is one the biggest software testing issues in banking today as companies’ online and mobile interactions with customers are increasingly seen as a way to differentiate competitively.

TestPlant’s data indicates that a typical banking application (web or mobile) today will require approximately 200 test scripts to be created and run for each release. In today’s highly fragmented consumer technology landscape, banks’ quality assurance teams are generally choosing to test 15-30 of the most popular device/OS/browser combinations – for example, Internet Explorer on Windows 8, or Safari on an iPhone 6. Assuming just 15 combinations are tested, the time taken to manually test a typical release is more than 380 tester-days.

Subsequent releases typically add 10% more functionality, but in a purely manual testing environment these have to be fully tested again. This increases the time required to manually test each new release to 422 tester-days.

“In the past, companies were only testing on very small numbers of device and browser combinations, which would take a much smaller number of tester-days for each release,” said Antony Edwards, CTO, TestPlant. “This meant that banks could reasonably expect that a fully manual testing approach was enough, but that is no longer the case.”

As fragmentation increases, all of these numbers will also increase and it doesn’t take long to realize that testing departments will find it impossible to scale up manual testing processes. In five years’ time, TestPlant estimates the number of testing combinations to approach 60. This would see the manual testing effort for an initial release rise to more than 750 tester-days.

“The problem is linear – for every hour it takes to manually test an application on one phone or browser, then testing more phones/browsers multiplies the time required,” said Edwards. “Banks IT teams are struggling with this because they can’t just keep hiring more people to absorb the impact of fragmentation – but their marketing and customer service departments can’t afford delays.”

About technology fragmentation

Technology fragmentation refers to the large and increasing number of different devices, operating systems and browsers being used around the world. For mobile devices this means multiple iPhone and iPad versions, many Samsung Galaxy models, Nexus, etc. For desktop applications, fragmentation refers to the many browsers and browser versions in existence, including all the variants of IE, Chrome, Firefox, Safari, etc. That’s not forgetting the plethora of new point-of-sale devices and smart TV browsers.

To help provide some context to the scale of the challenge, consider Android devices alone. OpenSignal reported last year that it had recorded more than 18,000 distinct devices running the Android platform, and fragmentation is only increasing.

To effectively test a banking application, whether it be a mobile app or web-based interface, banks ideally need to test on as many devices and browsers as possible, but typically they only choose the 15-30 most widely used technologies. In the world of manual testing, where tests are conducted by human beings, even this fairly small cross-section presents a huge testing challenge.

Maintaining a manual testing approach means having to adopt one or more of the following unappealing options:

  • Pay for more manual testers, which has a significant cost implication.
  • Increase time-to-market (i.e. take longer to do the testing), which has a significant impact on competitiveness.
  • Test less, which will ultimately mean more errors in the app and customer service failings.

Automation is key

Automating software testing is the key to dealing with fragmentation. While there is an upfront cost, the costs scale much more favorablyfter that.

“The scalability question relating to the impact of technology fragmentation on software testing is undoubtedly addressed by test automation,” said Edwards. “Manual testing has its place, but banks who want to stay ahead of the competition when it comes to the quality of their apps need to consider the right mix of manual and automation testing.”

TestPlant’s data demonstrates the benefit of using automated testing to combat the impact of fragmentation. For instance, in the previous example where 15 combinations are tested, using eggPlant Functional for the first release cuts the testing effort by almost 75%. When testing subsequent releases, using eggPlant Functional cuts testing effort by around 95%. As fragmentation continues to grow, the case for test automation increases. For a real-life example of how eggPlant tools give banks a testing advantage, read TestPlant’s Banco Sabadell case study.

“Test automation used to be all about saving time when testing new releases, but browser and mobile fragmentation add a whole new dimension,” said Edwards. “Now people are effectively trying to test 30+ combinations in a single project they MUST use test automation. If they don’t, then either their costs will get out of control, their quality and customer satisfaction will fall, or their time-to-market will be behind their competitors. The imminent arrival of more payment devices and the Internet-of-Things will simply overwhelm any organization without test automation.”

Media contacts:

Milli d’Hangest d’Yvoy
Marketing Manager, TestPlant
Tel: +44 207 002 7888

Danny Sullivan
Tel: +44 772 497 4255