Living with bugs in released software products

Bugs are the reality of software. We hate them, we do everything we can to get rid of them but they are always there and we have to learn to live with them - that is we have to manage the bugs and fix them in a sane way. This is important. Actually trillion dollar important. in study done by the Austrian software testing firm Tricentis, software failures cost the worldwide economy over $1 trillion annually!

The same report also found that software failures have caused more than 315 years of lost time and have affected approximately 4.4 billion customers. Software failures also have a massive negative impact on the reputation of companies. The companies surveyed by Tricentis lost an average of $2.3 billion of shareholder value just on the first day after announcing a software failure. No wonder that so many companies keep quiet about bugs. Yet that is probably the biggest mistake. The trick is to learn live with the bugs as a reality and have a practical and realistic process for managing bugs in live products. Today’s post is an attempt to summarize some of the strategies we found as the best for managing bugs on live products.

Accept bugs

This may sound obvious, but you’d be surprised how many companies treat bugs as something totally unacceptable and creates all kinds of penalties and punishments when bugs start surfacing on a product. This behavior comes from not understanding the nature of software products. Yes, the goal is always to have a bug free software release, but the reality is that there will always be some. So the goal should be:

“Release with as few deadly bugs as possible”

Accepting bugs opens up the possibility of a realistic software development process with healthy bug identification and resolution strategy. So this is the first step - for the management and for the technical team leadership.

Make finding and reporting bugs easy

Real life story:

A group of security researchers were prepping for a major reveal in 2013: They planned to disclose at a D.C. cybersecurity conference how a security flaw in luxury vehicles could let bad guys break in without keys and start the cars.

But Volkswagen stopped them, winning an injunction in a British court after arguing that publishing a paper detailing the problem would "allow someone, especially a sophisticated criminal gang with the right tools, to break the security and steal a car,"

https://www.theguardian.com/technology/2013/jul/26/scientist-banned-revealing-codes-cars

After you’ve accepted that there will always be bugs the released products your next sane thing is to setup a process and workflow for identifying and resolving bugs. It’s just crazy how many software companies out there try to hide their bugs or try their best to make it difficult for people to report them. There are literally hundreds of known cases where large companies tried to stop reporting bugs in their software - just making their users suffer even more.

This line of thinking is stupidity defined. You just cannot stop people from finding bugs. What you should do is make it easy for people to find them for you. Have easy ways for reporting such bugs - such as customer support emails that can easily turn into bug reports, a bug reporting page like Facebook’s Report a Bug feature.

On your development team’s side also have the tools for keeping track of all bugs that are being reported - both by customers and your QA teams. Tools such as Jira are absolutely essential, just as important as the software development tools or issue trackers when the actual software was being developed.

Involve customers in the bug finding journey

This needs to be a management and marketing level decision - involve your customers in the bug finding process. Ask them for feedback, have formal ways of checking back for issues, reward them with freebies when they find bugs. Customers are the ideal QA of a software, they use it everyday and they use in ways that your dev team would not even think about using a software (e.g using MS Word for image editing or Power point to make web pages!).

Microsoft has a thing called bug bounty which literally inspires users to spend time to find bugs and be rewarded for such efforts. Google has it’s various reward programs for finding defects, these are all examples of far sighted companies who have not only accepted bugs in live products and made it easy to report - they are actively involving the customers in their journey to find and ultimately fix those bugs.

Have workflow for updating users about bugs

As you accept bugs as reality, your wording and communication will let your users also accept them and make them expect you to come up with fixes as soon as they are found. They also expect you to be mature about bugs found in the current system and have a place to know about such bugs and the status of those fixes. User forums are great for such things and modern ones like Zendesk or Uservoice and many others like them do a great job of updating users about bugs, getting their feedback and letting them know when things are fixed. They have also the feel of user communities and can lead to bigger and better things such as users helping each other out and users forming their own groups to teach each other.

Create a culture for celebrating finding and fixing bugs

Finally, the most important thing of all - within your company and your development team create a culture of celebrating the finding of each bug and fixing them. Some companies make the fatal mistake of making the dev team feel bad each time a bug is found - this creates the negative emotion that takes away the enthusiasm and spirit the team needs to polish off a released product over time. To a dev team the release product is their own creation - any bug would sound bad to them anyway, it defeats the purpose if the company culture is to make them feel worse. The culture should be that bugs are unfortunate yet inevitable - but once found how fast they are resolved and an updated software is released is the true sign of a great software company.

Trust me on this, after more than two decades in this bug creation, I mean software development business, this culture fix is the biggest thing that differentiates a great software company from the others.