Beware the false analogy
Reasoning by analogy refers to the process of drawing inferences based on similarities between different objects or concepts. For example, the electronic-hydraulic analogy is based on similarities between the flow of water in pipes and the flow of electricity in circuits. Closer home, project teams often use analogies when they estimate tasks based on similar work that they did in the past. Analogical reasoning is powerful because it enables us to leverage existing knowledge in one area to solve problems in other, possibly unfamiliar, areas. However, such reasoning can also mislead. This post looks at the problem of false analogies in project estimation.
I’ll begin with a story…
Some years ago, I was in a discussion with a client, talking about costs and timelines for an application that the client needed. The application was a sales bonus calculator for front-line sales staff. The client needed an app that would calculate bonuses for each salesperson (based on some reasonably involved calculations) and display them via a web front-end. Nothing fancy really, just a run-of-the-mill corporate web-database application. The discussion was proceeding quite nicely until a manager from the client’s side felt obliged to make a remark based on a false analogy. I can’t recall the conversation word-for-word, but it went something like this:
“It can’t be that hard, he said. ” You guys have built a similar application before, your promotional literature says so”. He knew from our brochure that we had built a bonus calculator before; problem was he didn’t know the details.
There was a brief silence until my boss said, “Umm…yes we have done a superficially similar project before, but the details were very different from this one.”
“How different can it be?” retorted the manager, “bonuses are based on sales data. You process the data based on rules and display it. Yes, the rules are different, but the concept’s the same. You should be able to do it in half the time you’ve quoted.”
My boss countered the argument politely, but the manager would not let it go. They went back and forth a number of times until the sponsor stepped in and asked my boss to ensure that the manager’s concerns were addressed. The issue was resolved later, after my boss stepped the manager through the application, showing him just how different it was from the one they had requested.
The technical manager based his estimate on a superficial similarity between the app we were building for him and one that we had done earlier. Analogies almost always break down when examined in detail. For example, the electronic-hydraulic analogy mentioned in the first paragraph has several limitations. The same is true when comparing two projects or tasks.
An insidious (and dare I say, more common) occurence of such reasoning is when team members themselves draw false analogies. This happens when they make seemingly harmless (and often tacit) assumptions regarding similarities between tasks that are actually dissimilar in ways that are important. See my post on the reference class problem for a discussion of an estimation technique that is prone to incorrect analogical reasoning.
Estimates based on false analogies are a reflection of poorly understood requirements. This begs the question: why are requirements misunderstood when most projects involve countless meetings to discuss scope and requirements? In my opinion this happens because talking about requirements doesn’t mean that everyone understands them in the same way. In fact, in most cases different stakeholders walk away from such meetings with their own version of what needs to be done and how to do it. The first step towards curing the problem of false analogies is to ensure that all stakeholders have a shared understanding of the requirements. This applies particularly to those who will create the product and those who will use it. Dialogue mapping, which I’ve discussed at length in several posts on this blog, offers one way to achieve this shared understanding.
Of course, a deep understanding of the requirements does not by itself cure the problem of false analogies. However, it does make estimators aware of what makes a particular project different from all the other ones they’ve done before. This makes it unlikely that they’ll use a false analogy when making their estimates.