Estimating Software Development Projects is Hard

As a software developer, working with non-technical management and end users to define a deadline for when a project will be ready to use or how long it will take from start to end of a project is an extremely complex task. Unfortunately, the non-technical audience thinks that it should be simple to give some sort of estimation on the fly without detailed analysis of the project and what it involves, as it seems simple for them to estimate tasks that are like ones they have done before.

Unfortunately, a good software developer will do anything possible to avoid having to repeatedly do the same task. They will instead automate the repetition of the task so that it will automatically happen when needed. When tasked with estimating a new project, the standard practice for a civil engineer building a bridge or a homebuilder building a home is to break the major task into the largest possible tasks that can be estimated to the precision necessary to the project. Many times, these tasks have been completed previously, with the amount of time and effort to complete being well known.

When compared with the projects that software developers face, however, it is quite a different story. Anything that a developer has done before, they should be able to reuse in the new project, making the time to implement known functionality almost 0. Once you start looking at the remaining functionality, there are a great number of unknowns that must be accounted for, assuming you have a good understanding of the problem that the project will be solving. However, I have seen that many times, the initial problem definition for the project will be lacking details, resulting in a project that lacks functionality that is required by the customer. As a result, partway through the development of the project, the missing functionality is discovered, added to the project definition, frequently without updating deadlines or time estimates.

Any time you have someone estimate how long it will take them to complete a task they have never completed previously, the estimate will vary widely from the actual time needed to complete the task. This is precisely what we ask developers to do when we want them to estimate the time it would take to complete a software development project within a reasonable 10% margin of error. The best that the developer can do is get a rough comparison to a similar project they had worked on before, but really has some fundamentally different requirements, making it too dissimilar to be an accurate estimation aid.

In the end, if management decides to judge software developers on the accuracy of their development estimates, they are judging the wrong output of the development team. Instead, they should be judging the business value delivered when a project is completed compared to the cost required to deliver the value, and how accurately and defect-free the end result is.