Nightmare in code street - a software project horror story with a happy ending

I recently came across a documentary, spaghetti code, that talks about how a multi-million Euro software outsourcing project became a disaster. The film was in Dutch but given the topic, which is the focus of most of my energy these days at work, I persevered to understand. And with the help of my Dutch friends got to the gist of the story, which is sadly a very common one – big company outsources software development to another big outsourcing company, there are many layers of management and not everyone is sure what is being done, the project fails and developers far away gets an unfair proportion of the blame (although to be fair, the documentary blames the management too and also there additional twists of greed and corruption).

I smiled and then raved a bit with righteous rage (being a techie in the inside meant the slur of “spaghetti code” and blaming it all on the developers touches a nerve or two). Then I cooled down and decided to write my “ultimate guide to safer outsourcing” or something along those lines. Which, by the way, is what I was planning anyway for my series on software project outsourcing – the first of which appeared last week: deciding whether to outsource or not? But a little more reflection led me to ask:

 Was I also not hiding behind righteous indignation and the comfort of the “blame them” myself? Weren’t there a story or two of my own where things went wrong?

Let’s face the facts; a software project is difficult to pull off smoothly even if the team is in-house. Add to that difficulty a team that is miles (sometimes thousands of miles) away, different culture, different language and different time-zone – you have a really risky venture. There are of course ways of managing that risk. They are time honored, field tested and pretty fool-proof – but they are to be the subject of a different article soon. Today, I’m going to face my nightmare and tell you the story of how everything went wrong in one of our projects, and (happily) how we survived and saved that project. I can’t go into too much details, of course, but will stick more to an overview and on what we learnt from the experience which I think will be of more value to my readers.

The project

It was an angel funded project trying out a very innovative solution for a common problem that happens in large companies. There were three people on the client’s side with none of them with any prior experience in software. They had an existing codebase – a left over from a different company that they had used and did not like.

The spec

We received a large amount of documents at varying degrees of up-to-dateness (and yes, that is an acceptable noun). Then we did several sessions of going through the concepts. We found that the clients have actually moved away from many of the features as described in the specs. This is something very common with start-up, actually it was surprising that they had specs at all, so we were not worried.

The process

We are an agile shop when it comes to process. For startups we choose a Kanban model, where pretty much everything thing can change at the last minute (within reason). This works best for start-ups because most of their priorities and product requirements move with time.

We chose trello for issue management, google docs for sharing documents, github for source control. We put a tried and tested scheme of weekly meetings and daily builds. Since we saw a lot of instability in the feature requirements, we added a short but daily feature discussion session with the clients and our designer and  project manager.

The nightmare

Almost from the start things started going wrong. We kept getting feedback that the product features were not what they were expected to be, this led to piling up of tasks that were adjustments to features we’ve already done. This backlog led to delays in delivery of major releases planned – which led to our clients losing important business. Something was very wrong, but we didn’t know what it was. We had put in all our standard safety measures in. We were communicating (maybe too much) on a daily basis. It just didn’t make sense.

The realization

After several missed deadlines and lost weekends (which we avoid the like the plague because it’s the worst thing you can do the developers’ productivity) we decided we need a proper retrospective. We pinned the problem down to the following:

1.       Clients could not visualize the features that were to be built based solely on the wireframes we were using. We had opted to do quick wireframes to speed up the process to meet the delivery schedules of the client (typical for start-ups that need to demo to prospective investors).

2.       The feature priority that the clients were setting was not taking into consideration the relative level of effort for the features. So we were not doing the good practice of picking up easy to do features first even though they might be slightly lower in priority than other harder features.

3.       Unrealistic expectation about deliveries, and correspondingly ambitious estimates from the team trying to appease those expectations.

The happy ending

We spent days thinking up of new and quite frankly convoluted fixes. But finally we boiled them all down to an absurdly simple solution – there needs to be a techie on our client’s side.

We managed to convince our clients (somehow, don’t ask me how!). They hired a consultant who would work with them 3 days a week in a role of a product manager. And things changed for the better almost immediately. The fix really felt like a miracle cure.

The moral of the story

… is that: there is no single formula for making your outsourced software project safe from disasters. There are prescribed best practices that you must follow, but stay alert for signs of failures and jump in with remedies when you see them.

Well that’s the horror story. You can never really depend upon a foolproof process in this world. But there is one thing you can certainly depend upon – there will always be an appropriate dilbert strip that you can steal, for anything to do with life in engineering. Here is my loot for today’s article.