Software is the engine of our economy, at the root of almost everything we do. Our success rate has been creeping up in recent years, but one thing is sure: there has been no major breakthrough. What will it take?

Is our software becoming more or less expensive to develop?

This is a difficult question. The applications we develop continue to become richer and more sophisticated in their scope and functionality. The software projects we now undertake are much more complex than software projects were 10 or 20 years ago. There is no doubt that we expect more from software today than ever before.

In a sense, we have a "chicken and egg" situation. Are we doing more daunting things today because we can? That is, are we attempting more ambitious projects because we have better methodologies and tools that enable their conception and execution? Or, have the methodologies and tools just kept pace with the increasing challenges and degree of difficulty demanded by the external world?

When projects are successful, we have productivity levels that are much better than ever before. The mountains we are climbing are higher, and we are climbing them with proportionately fewer resources than in the past. That is the bright side of software development: better systems and higher productivity.

The dark side of the equation is project success rates. Why does the needle on success rates move so slowly? It is as though all the advances in methodologies and tools have helped us build more ambitious systems, yet there is some other factor that is keeping us from being successful more of the time. Perhaps there is something in the product development approach of the past several decades that is missing the point.

A Look at the Data
The Gartner Group reported that there were $3.7 billion spent worldwide in 2004 on application development tools. This was up over 5% from that spent in 2003; it is therefore not unreasonable to assume that in 2006 somewhere in the neighborhood of $4 billion will be spent. Note that this does not include money spent on methodology, consulting, or other development expenses.

But what about the software development project success rate? Dr. Philippe Kruchten, professor of software engineering at the University of British Columbia, has provided me with the following data from the Standish Group's CHAOS reports (see Figure 1).

The data show that software developers are making progress on the success rate front, but slowly. The improvement is about 1.7 percentage points a year, and appears to be linear based on this small sample of data. If the current improvement rate continues, we won't achieve a 50% success rate until the year 2014.

Software is the engine of our economy, at the root of almost everything we do. Our success rate has been creeping up in recent years, but one thing is sure: there has really been no breakthrough. Unless you consider almost "doubling' the success rate in 9 years a "breakthrough"; this effect is the result of a very low baseline for comparison. We may still be in the business of harvesting low-hanging fruit.

Maybe we are not succeeding because we are attacking the wrong set of problems. The current approaches are still squeezing out failures, but we may soon run into the law of diminishing returns. We haven't yet seen the success rate curve flatten out, but there is no reason to expect that the linear progress we have seen over the last decade will continue indefinitely. In fact, we can be sure that at some point we will hit the wall. We just don't know when. Prudence dictates that we investigate alternatives before we are faced with even more unpleasant realities than we have today.

Where Has All the Money Gone?
Most of the money over the past 25 years has been spent in two areas: methodologies and tools. Let's look at each of these. Methodology has improved. We have seen a steady progression from functional decomposition to structured analysis, which in turn was replaced by object-oriented analysis, and then by modeling languages and notations. By the turn of the century, we saw acceptance of standards, among them the UML (Unified Modeling Language), adopted by the Object Management Group, and, later, the Rational Unified Process. All these things have helped. In particular, the notion of a solid architecture as the backbone of software systems has finally taken hold.

More projects now use iterative development in one form or another. Recently, agile methods have added more evidence that we can tune our processes to be more productive and avoid the catastrophic outcomes that overtake many projects. Concomitant with these improved methodologies are tools that really help. Enterprise software developers have seen configuration management and version control systems mature, and have also seen software development or programming environments become almost commodities. Automated testing methods have greatly improved. There is a higher degree of integration among these tools than ever before, and, compared to their ancestors, many of these tools truly save time and reduce errors. And it cannot be disputed that increasingly sophisticated languages have emerged. The libraries that come with each of these languages provide more and more opportunities for large scale reuse, something that has a very positive impact on overall software development productivity.

Productivity vs. Success Rate
Besides the success-rate metric, we also need to consider productivity. Is our software becoming more or less expensive to develop? Frankly, I don't think there is a clear and unambiguous answer to this question. What we can say is that when projects are successful, we have productivity levels that are much higher than heretofore. We are completing larger and more complex projects with proportionately fewer resources than in the past.

But what about pure "success." Are the software projects helping us achieve expected business goals? Why does the needle on success rates move so slowly? It is almost as though all the advances in methodologies and tools have helped us build more ambitious systems, yet there is some other factor that is keeping us from being truly successful more of the time. Perhaps there is something in the approach of the past several decades that is missing the point.

The CEO's problem
Let me put this another way. Assume for the moment that 75% of the $4 billion software tools money spent in 2006 is disbursed by the Fortune 1000 companies. Simple division then tells us that:

On average, each Fortune 1000 company spends $3 million on software development tools each year and sees less than a two percentage point improvement in project success rates.

The $3 million must be spent just to keep up! While that money helps increase productivity and allows the creation of more competitive applications, it is doing very little to raise the chances of success on the next set of projects. Today we can say that roughly one project in three will be successful; 35% is a good estimate. If all goes well, it will take us another eight years to get to 50%. Most CEOs are either concerned about this or they should be.

Where's the Problem?
Maybe the problem is not where we first think it is. When I talk with my most knowledgeable colleagues in the field, I continue to get the same answer over and over again. We are getting better at producing software, but problems persist at the front end. That is, we are still having difficulties getting the requirements right. It is challenging to correctly specify what it is that we want built. Requirements are often ambiguous, unclear, incomplete, or contradictory.

As a result, developers often guess at what is desired, only to come back later in the cycle and rework their software once discrepancies have been discovered. We discover the problems too late, after much time and money has been spent building the wrong thing. Rework decreases overall productivity, and sometimes the rework is so overwhelming that the project fails. Previous software development spending was not for naught, but there is now a wiser way to spend at least a portion of it. In effect, we are spending our money solving the wrong problem. No matter how good our "production tools" are in developing software, they cannot overcome a mangled or misdirected specification. Project success-rate improvement will almost certainly hit the wall unless we do a better job of specifying our requirements. Figure 2 illustrates the danger.

The floor at 50% may be lower; we don't know. But if requirements errors are indeed the root cause of a large percentage of our failures, not addressing this issue is going to cause us to cease to improve. Conversely, if we did make progress in this arena, we might see a dramatic improvement over the linear progress to date.