One might think that as the IT industry gets more mature , the "estimation problem" will be solved sometime in the future. However, the outlooks are not promising at all: according to The Standish CHAOS Report, the accuracy of estimates has not improved over the last 10-20 years. As a matter of fact, the problem seems to be inherently unsolvable. In 1968 Alfred M. Pietrasanta (IBM Systems Research Institute) had already explored the root causes: "Anyone who expects a quick and easy solution to the multifaceted problem of resource estimation is going to be disappointed. The reason is clear: computer program system development is a complex process; the process itself is poorly understood by its practitioners; the phases and functions which comprise the process are influenced by dozens of ill-defined variables; most of the activities within the process are still primarily human rather than mechanical, and therefore prone to all subjective factors which affect human performance". After forty years his assessments still remain valid...

First of all we have to clarify the goal of software estimation. The estimation serves as the basis for project planning and budgeting, and more importantly, it attempts to indicate how realistic the project goals are at all. Over-optimistic and not-thought-through estimates result in unrealistic schedules and deadlines, which will certainly be missed. This is desired by neither the customer nor the developer.

The estimate

What shall be the output of an estimation? It is pretty common that a single number is presented as an output of an estimation: the estimated effort is X man-days. But what exactly does this statement mean? Does, the estimator deem that the project will take X days, no more, no less, with 100% probability? Certainly not.

Such single-point estimates are both unusable and meaningless. The estimate should reflect the intrinsic uncertainty of software projects, which indicates that each estimated effort should have its own probability. Therefore, ideally, the estimate is a probability function of project outcomes. This function can be (partially) given by specifying the probabilities of certain project outcomes. For example, the project will be completed within 100 days with 0% probability, within 120 days with 20% probability, within 150 days with 75% probability, and within 210 days with 90% probability.

In practice, a prediction interval and a median outcome can also be a useful output of an estimation. In this case the estimator asserts that the effort will fall into a given interval with a specified probability, and also specifies the project outcome with 50% probability. For example, the project outcome will be between 120 and 210 days with 70% probability, the median outcome is 135 days.

Estimation by developers

Maybe the most commonly used estimation technique relies on the "gut feeling" of developers, i.e. developers are asked about how many hours the project or a task will take. This is an inherently intuition-based approach which is also prone to all subjective factors which affect human performance. One such factor is the preconception of or even the unintended pressure of the project stakeholders on the estimator. For example, a CEO may not only ask the project lead to estimate the development effort but also defines certain non-technical constraints which already determine the outcome of the estimation.

Another factor is the unfounded optimism of developers. Even without any pressure from their bosses, developers tend to be over-confident in their estimations and their accuracy as well. Tom DeMarco's common definition of an "estimate" reflects this observation: “An estimate is the most optimistic prediction that has a non-zero probability of coming true…"

Good news about over-confident developers is that, according to Mange Jørgensen, regardless of their experience they remain over-confident even if they get repeated and timely outcome feedback. Therefore, the inaccuracy of the developer estimation can be compensated in the same manner for a long period of time (e.g. by multiplying it by a constant).

Besides using an "adjustment factor", other techniques can also improve the accuracy of estimations. For example, if we decompose the project into many small tasks we can exploit the inaccuracy reducing effects of the law of large numbers. In this case the developers have to estimate the completion time of the identified tasks and the project estimate is calculated by aggregating the task estimates. The smaller the individual tasks are the better the accuracy of the aggregated project estimate will be since, according to the law of large numbers, the small under- and overestimation estimation errors will eliminate one another. Even better results can be achieved if the developers do not give single-point estimates for the individual tasks and the aggregated estimate is calculated accordingly.

Analogy-based software estimation

Estimation by developers is an intuition-based, basically subjective process, which is prone to human error and misjudgment. The analogy-based software estimation method reduces the influence of human factor. This estimation technique reuses the development experience of past projects stored in a project repository, and derives the estimate from the metrics of previously completed "similar" projects.

The analogy-based estimation consists of the following major steps. First, the estimator measures the feature metrics of the project (e.g. counts the number of screens). Then the project repository is searched for similar/analogous completed projects. The effort value of an analogous project is the initial estimate, which obviously has to be adjusted in accordance with the feature metrics of the current project. The adjusted estimate is calculated by comparing the known metric values (e.g. number of screens) of the current and the selected analogous project(s). For example, if the current project has 15 screens and the analogous project had 30 screens and the relating effort value then was 100 days, then the adjusted effort estimate will be 200 days (using linear adjustment).

As we can see, estimation by analogy follows the commonsense reasoning process we use on a daily basis. However, it is not so easy to use in practice. A major obstacle to its use is that many organizations do not record data from completed projects. Besides, a project database must be maintained as well. For example, if we want to distinguish between static and dynamic web pages using AJAX by introducing new project attributes, we need to update any existing database entries.

Analogy-based estimation is supported by various tools. However, tools should never give a final answer. According to studies people are better than tools at selecting an analogue. People can distinguish between relevant and irrelevant project attributes (e.g. the customer of the product may be a more relevant "attribute" than the client technology) and therefore can identify such analogues that result in more accurate estimates. Consequently, individual judgment and experience cannot and should not be eliminated from the process entirely.

Finally, we have to note that analogy-based estimation techniques, or, as a matter of fact, all estimation techniques which heavily rely on past experience, have a major weakness: they are prone to nasty "estimation surprises" (large estimation errors). In the summer of this year (2008) oil prices set a new record at 140 U.S. dollars a barrel. Based on their previous experience from past weeks/months/years, experts predicted the prices would hit 200 dollars a barrel by the end of the year, and prices could possibly surge to 200, 300, 400 dollars. They could not be more wrong! Their lack of experience from the past about a major investment bank going into bankruptcy did not allow them to forecast the global recession and oil prices tumbling to under 50 dollars late November.

In his book, The Black Swan, Nassim Nicholas Taleb explores the problem of induction-based reasoning more deeply: regardless of the number of events observed in the past we cannot fully predict the future. Humans rely on their experience almost uncritically, which, in most cases helps people finding their way in the world. But we also have to be aware that the problem of induction limits our ability to foresee future events.

Conclusions

Given the complexity of the problem and uncertainty in software projects it is unlikely that we will ever find one universal estimation strategy that would outperform all others regardless of the estimation environment. We can either select the most suitable method, or combine the results of different estimations. M. Jørgensen found that the selection of effort estimation strategies is not an entirely objective process. According to his research, there are large individual variations in the selection of estimation strategy, which are partially due to subjective circumstances. Therefore, the combination of results provided by different estimation strategies seems to be preferable as opposed to the actual selection of strategies.

Standardized processes and well-controlled projects also improve estimation accuracy. A good standardized estimation procedure adopted by an organization eliminates the subjective individual preferences that may bias the selection of estimation methods and the estimate as well, and leads to future improvements by learning from prior lessons. Project control can also aid the accuracy of estimation: according to Capers Jones (Estimating Software Costs), in the case of well-controlled projects, accuracy of +/-10% is possible.

Chaotic projects, however, are dominated by uncertainty to such an extent that it is impossible to achieve such levels of accuracy. We have to bear in mind that "There is no point in being exact about something if you don't even know what you are talking about." (John von Neumann). By leaving important details of the project (such as requirements, architecture, etc.) undefined for long, predictability will be jeopardized. Although, even if you deem that you have clarified all significant details and you have dealt with all important issues, as Taleb pointed out, you are still subject to unexpected surprises which have extreme impact on the project.