When Slack experienced a four-hour outage in January 2021, millions of remote workers were suddenly cut off from their primary communication tool. Slack was already important prior to the pandemic, but it become the sole communication tool during it for millions.
The incident cost the company an estimated $500,000 per hour in lost productivity across their user base, and this doesn’t count the reputational damage and customer churn that followed. Forget cyberattack or infrastructure failure, this was simply a software bug that could have been caught with more comprehensive application testing.
Application testing isn’t just about finding bugs before users do, despite that being how many treat it. It’s a practice that validates whether software behaves as intended under real-world conditions, scales under load, provides a secure experience across different environments, and many other user scenarios. For help with application testing to avoid its false economy of compounding costs, contact our team at Qalified for comprehensive application testing services.
The economics of poor testing
Consider the real costs when applications fail in production. Of course it depends on scale, but these figures are significant for firms of any size. Knight Capital Group lost $440 million in under an hour due to a glitch in their trading algorithm. It was a failure that essentially bankrupted the company. This cost wasn’t so hidden, but some are.
Customer support costs are an obvious one that escalates exponentially when users encounter bugs. Each critical bug may generate several hundred support tickets, and each one may cost a few hundred dollars to solve depending on its nature. To invest an extra $5,000 in user testing to avoid several bugs a year could easily be cheaper than paying for the extra support costs alone. Here, we begin to see why it’s an investment, not to mention the bug fixing debt which is compounded over time. Each bug can take over 20 times longer to fix when in production vs development.
Performance testing to prevent the Netflix Christmas crisis
When Netflix’s streaming service crashed on Christmas Eve 2012, it wasn’t due to a coding error, but a capacity planning failure. Amazon Web Services, their cloud provider, experienced an outage that exposed Netflix’s insufficient redundancy planning. The incident affected millions, all during the most important day of the year when they’re together, watching something, during peak holiday viewing. There were thousands of angry social media posts flooding through and news coverage that cost far more than any testing investment would have.
Security testing and the lesson of Equifax
Equifax’s data breach back in 2017 exposed the personal information of 147 million people due to a known vulnerability in Apache Struts – it remained unpatched for months. The total cost exceeded $4 billion in settlements, legal fees, security improvements, and general lost business value. The vulnerability had a published fix available, but Equifax’s application security testing processes failed to identify the exposure.
Security testing isn’t just pen testing or vulnerability scans either, but validating authentication flows, testing authorization boundaries, verifying data encryption, and so on. Modern applications integrate dozens of third-party libraries and services, each potentially introducing security vulnerabilities.
User experience testing
When Apple released iOS 8 in 2014, users immediately complained about keyboard lag and connectivity issues. While the software functioned correctly in Apple’s testing labs, real-world usage showed that crashes were common and many users didn’t upgrade or even remain within the ecosystem. The lesson was that functional correctness doesn’t guarantee user satisfaction.
User experience testing is what verifies that applications work intuitively for actual humans under these realistic conditions.
Automated testing: Compounded quality
The most compelling ROI can come from automated testing that provides continuous value throughout an application’s lifecycle. Like compound interest, automated tests generate returns that accelerate over time. A comprehensive automated test suite, like offered by Qalified among others, prevents regressions, enables confident refactoring, and generally allows teams to deploy more frequently with a reduced risk.
Spotify deploys code to production thousands of times per day across their services, and this is enabled by automated testing that provides rapid feedback on each and every change. This deployment frequency isn’t reckless like it may sound – it’s risk management through small, incremental changes that can be reverted when issues arise.
The initial investment in automated testing can seem substantial, as a typical enterprise application might require 200-500 hours to achieve solid test coverage. But when using offshoring companies, this can be less than expected, and more importantly, the tests execute thousands of times throughout the application’s lifetime to prevent bugs that would each cost more to fix in production.
If you’re wondering why many of the above examples date back to the early 2010s, this is because after a handful of mega-disasters, it became clear and obvious that there was a positive ROI to be had on more thorough application testing. While these crises still occur, ROI is more accurately measurd today and is taken more seriously. As a result, higher levels of investment is taking place.
