Why do software projects fail and how to save them
/A recent post about making software project plans that work started a thread of conversation with a fellow software guy about how true it is that software projects are prone to fail and we should just accept that fact and plan around that. His point was that we should actually take a more positive attitude that software projects work out at the end and work from that positive space managing the risks. He does have a point, any work starting out with a negative thought actually leads to bad things - your thoughts are a thing as Napoleon Hill says in a book about wealth creation. And it’s a good question to ask - what is a better approach for software projects - assume failure and plan to manage it or assume success and avoid traps?
Facts & Data
Let’s look at the facts again. Research after research shows that software projects fail most of the time. A great source of software project health in general are the CHAOS reports published by the Standish Group. The CHAOS Reports have been published every year since 1994 and are great indicators of the state of the software space. The 2015 report studied more than 50,000 projects around the world, ranging from small few man week software projects to huge thousand man year IT projects and system implementations. The results show clearly the trend that software projects are failure prone. Take the summary pie charts in the report (shown below). The red is ominously present in all the three aspects that were judged - budget overflows, timely delivery and feature achievement. I agree that the green is also strong (note the error on the first pie though, where the colors are flipped!) but shouldn’t you the red to be much smaller on all those pies? With such large amounts of money spent and with companies betting their existence these days on their IT systems having that high percentage of red is not a good sign.
And don’t think this is a recent trend. Actually things have been getting better over the years. Software projects have always failed but the rate of failure is reducing (at least according to these guys). The stats show the success percentages are going up over the years, so that’s definitely good!
Why do they fail? And, how to save them?
This is obviously the million dollar question (I guess a billions of dollar question in this case). Every software project is different, be it with the requirements, the nature of the product, the nature of the customer and the nature and composition of the team that worked on it. So it’s impossible to generalized. But overall some common risks are seen in all software projects that can if left unmanaged lead to be one of the factors of failure. Note that I said “one of the factors” as most software projects fail not because of one single cause but a whole bunch of them, each stacked on the other. So instead of talking about generalized causes of failure, let me just claim that most software projects have the following risks that should be managed. It would not gurantee the success of the project but it would certainly make the project safer.
Bad project planning
In our experience, bad project planning is the biggest cause of software development failure. This is one area of engineering where project planning makes it or breaks it. You can have the best developers in the world, unlimited budget, time, documentation yet if you don’t plan things right, if you don’t plan things realistically you’ll just burn that team, eat up that money and time and end up with a bad software or no software. To take a recent software project that failed despite having everything ($1.7 billion budget, 1K+ strong team, etc.) is the US’ healthcare.gov platform. It was way over budget (original budgeting was $93.7 million!), severely delayed and very buggy even when delivered. Problems started as soon as it was somehow launched. Although it was known from day 1 in the planning that there will be huge demand, the huge hit caused the site to go down within the first 2 hours. While website user hit initially cited as the main cause (inexcusable as this was one of the given parameters) other problems showed up because the UI was not complete or confusing. Issues such as drop down menus not being complete and insurance companies information data not being correct or complete was a common problem. By some estimates, only 1% of people managed to successfully use the site on the first week. On October 20, 2013, President Barack Obama remarked, "There's no sugar coating: the website has been too slow, people have been getting stuck during the application process and I think it's fair to say that nobody's more frustrated by that than I am." It was all down to the poor project planning.
Work on that project plan. Read our recent post about project plans that work. And most importantly work with skilled project planners who has experience to understand the problems of a software project.
Lack of communications or excess of communications
Software is always a collaborative exercise. You need open channels of communications that are fast and reaches a resolutions quickly. The the quick resolution is also super important that is whey the excess of communication is something to worry about. You have to monitor that “analysis paralysis” doesn’t creep in just because there is a lot of communications possible.
Too many targets & unrealistic goals
Often the only things that pushes a efficient and streamlined software project towards disaster is the multiplicity of goals. A team can only do one thing at a time with a good quality. If you try to push them to do too many things even with the best of planning they will burn out over time and reach a point where it is only the team’s speed that hinders the success of the project.
Many great companies fail to understand this - they push their wonderful software teams to create great software one after another - pushing new features in, pushing new versions and upgrades in - at one point the team just breaks down. No matter how good your plans are or how good your communications channels are a burnt out team will always lead to failure.
Too large a team and too large a project!
Software project size and the team that works on them have an optimal size. Beyond that optimal size things don’t work too well. Communication becomes too noisy, mistakes start happening, people tend to lose focus and the wonder project plan starts to break down. Over and over research has shown that larger projects fail more than smaller ones. Here’s a table from that 2015 report:
If you have a large project or a large team - break it down to smaller projects with smaller teams that interact at set points to build the large project by parts. That is the only safe way of doing things - we know it from experience and data proves it over and over. I’ll finish with this quote from the report … says it all.
“... that size was the single most important factor in
the resolution of project outcome ... It is clear that the larger the project, the less valuable the return rate. In many cases larger projects never return
value to an organization. The faster the projects go into production the quicker the payback starts to accumulate. ”
