The myth of the lonely project

Introduction

Much of project management theory and practice is based on the premise that projects that are lonely – that is, they are unique undertakings,  largely independent of their organizational environment and history.  In an important paper published in 2003 entitled, No project is an island:  linking projects to history and context, Mats Engwalls argues that such a view is limited and misleading. This post presents a summary and some reflections on  the paper.

Background

It is easy to understand why the lonely project perspective dominates much of project management practice and research. A project manager’s focus is on his or her current project; history,  environment and context are therefore seen as irrelevancies.   Perhaps as a consequence, this view also permeates much of mainstream research in the field. There are exceptions, of course, such as the paper I  have reviewed in my post on the relationship between projects and organisations.

As Engwall states:

With few exceptions, research has been dominated by a perspective based on “the lonely project”. The primary interest has been in the structures and dynamics of individual projects, typically discussed from the individual project manager’s (PM’s) perspective. As an outcome, the project has been conceptualized as a lonely phenomenon, independent of history, contemporary context and future. Earlier experience, simultaneous events, and future intentions are seldom included in the analysis. In this dominating ontology, procedures employed in one project are considered to be unique and factors determining project success are considered to be due to the individual project in question

In contrast, organizational theory has long recognised that (permanent) organisations are influenced by their environment and history. As he states:

In organizational theory, the environmental impact on organizations is a classical issue. There are probably few organizational theorists today who would challenge the idea that external factors strongly influence the inner life of an organization.

Engwall’s contention is that projects, like permanent organisations, are best viewed as open systems whose structures and processes are influenced by their environment and the history/culture of the organisations in which they are embedded. He builds his argument through a detailed analysis of  two major engineering projects undertaken within one  organisation. We’ll take a brief look at the two projects first and then discuss his analysis in some detail .

The case studies and the approach

The two projects were carried out within a major Scandinavian utility company. Both projects were multidisciplinary, involving teams with expertise in construction, process and electrical engineering. The first project – referred to as the Hydropower project– was a major upgrade of a hydroelectric plant and the second – called the Transmission project – involved the design and construction of a high voltage direct current (HVDC) power link across the Baltic Sea.

The two projects had several features in common. These include:

1. Both were internal projects – i.e. the deliverables were intended for use by the power company.
2. The projects were carried out in roughly the same period (between 1985 and 1992).
3. They were the two biggest projects carried out by the company during the period.
4. Both projects were organized on a matrix principle. Work carried out in individual departments was coordinated by the project manager.
5. The project managers on both projects were of a similar age. Both were seasoned engineers with plenty of experience of working on major projects.

As we’ll see, the two projects took very different trajectories despite these similarities.

Engwall gathered data on the two projects through a variety of sources – from archives, through interviews and even by direct observations in project meetings. He  then analysed the data from two perspectives: first, from the view of a near-autonomous project (the “lonely project” view) and  then from a viewpoint that takes into account historical and contextual linkages with the parent organisation.

The lonely project view

The hydropower project

This project was implemented by the hydropower division, one of seven engineering and business divisions within the utility. The project was run by a project manager who coordinated the engineering work done by the different divisions. Most of the work was done by internal staff, supplemented by externals where necessary.  Most important, though, is the fact that the project was run as a traditional “textbook” project. As Engwall puts it:

The PM was well aware of the message in project management literature and had the explicit intention of employing its concepts and techniques in his project. He put a strong emphasis on structuring, planning, scheduling, and cost control. He produced a project management handbook, defining guidelines and checklists for the project. He formally defined the roles and procedures of the project organization. He initiated start-up meetings, workshops, and seminars for key engineers, and assembled his project team for coordination meetings once a month. He felt responsible for all kinds of issues concerning the design and engineering of the hydropower plant…In many ways, his management approach resembled the role model of the project management textbooks.

The project manager felt that he was ultimately responsible for delivery of the design and engineering work, and his approach reflected this. In short: the project was run pretty much as recommended by project management orthodoxy.

The transmission project

This project was run by the transmission division of the utility. The project was managed by a team leader from one of the engineering departments within the division. The core HVDC system was procured from external vendor, but most of the other engineering work was done by different divisions within the company. The project manager’s main job was to coordinate the work done by the different divisions and the external party. He did not have as much positional or expert authority as his counterpart on the hydropower project. Further, he kept a much lower profile, often deferring to experts to make decisions. Significantly, neither the manager nor his deputy had any formal knowledge of project management principles. As Engwall puts it:

In relation to the Hydropower Project, this PM had a lower formal rank within the company’s hierarchy. In his position as PM, he had no formal authority at all. He held a degree in electrical engineering from a junior college. He was very humble, had a very low personal profile and was often silent during meetings. He, and his deputy, coordinated the project without an explicit management approach. Neither of them had any formalized training in project management and they had limited theoretical knowledge of project management methods and techniques. During the interviews, they often excused themselves for their lack of knowledge in project management theory.

Moreover, the project had no formal organizational structure. Project meetings were optional – it was up to the departments to decide whether or not send a representative. Most of the coordination work was done through direct (but informal) contact between the project manager and the engineers doing the work. The responsibility for scheduling and timely delivery of different modules was delegated to the engineering department doing the work. In short: the project was run without formal project management processes and controls.

Analysis

Surprisingly, the manager of the hydropower project had many more problems than his counterpart who ran the transmission project. As Engwall states:

The PM of the Hydropower Project encountered several difficulties when he tried to coordinate the activities of his project. Many of the participating engineers did not follow his instructions and decisions. The project was constantly delayed because the engineering departments were not paying enough attention to it. Furthermore, many of the engineering departments tried to perform their parts of the project independently of the other departments, and without the involvement of the PM. On many occasions, the PM found it impossible to gain control of the project, which he was formally in charge of. Time schedules and milestones were not taken seriously, costs were recorded without consideration to budget, and different departments were planning and engineering their technical power plant subsystems without sufficient coordination with the other players…

The project manager’s response to the above was pure traditional PM: he attempted to assert control by pushing the formal processes even further on an already mutinous team. Yet, despite his efforts, the project was judged a failure because of delays and budget blowouts.

In contrast, the transmission project ran smoothly, and was deemed a great success. The specifications were met within the stipulated budget. Further, the transmission link was operational several days ahead of schedule.

From the lonely project perspective, the difference between the two projects is paradoxical. As Engwall writes:

The PM with his formal authority, explicit management approach, and who deliberately applied state-of-the-art methods of project management was having significant problems, while the PM who lacked both formal authority and an explicit management approach, and who deviated substantially from the textbooks, was being significantly successful. How come?

As an answer to the question posed above, Engwall suggests that classical project management theory (as described in books and BOKs) is incomplete – i.e. there are factors that the theory does not take into account. To illustrate this point, he takes a broader perspective of the projects, one which includes the organisational environment (context) and history.  We’ll look at this organisational view next.

The organisational view – linking projects to environment and history

The hydropower project

On the technical side, one of the key requirements of the hydropower project was that many features and aspects of the existing plant had to be preserved for their historical value. It was also stipulated that the plant had to continue operating uninterrupted whilst the upgrade was in progress. These requirements greatly increased the technical complexity of the project.  Finally, the project was truly unique because it was the first time the organisation was carrying out a major upgrade within a single project; all prior upgrades had been carried out as a series of minor, ongoing extensions.

On the management side, the project was used as a testing ground for a new project management approach within the organisation. The new approach conflicted with the existing one in many ways.   For one, the project manager took on responsibility for coordinating work between different engineering divisions – something that, in the old approach, was done through direct contact between those responsible for the work. Consequently, many project personnel saw the new approach as a deliberate move by management to micro-manage their work and reduce autonomy.

As Engwall puts it:

For the first time, engineers and department heads, who were used to working almost autonomously, encountered a PM who was deliberately trying to control the project process by actively getting involved in their day-to-day work. The management problems of the project were, in this sense, the result of a collision between two philosophies: … traditional procedures…and [a] “modern” management style…The hidden agenda behind the client’s appointment of this specific PM for this specific project was as follows: to force the Hydropower Division to change its management procedures from within.

In short: there was a clear lack of trust and open communication between the project manager and the team.

The transmission project

The transmission project, because of its strategic importance, was strongly supported by top management. The project was therefore given high priority. Further, the project was considered interesting by engineers because of its technical novelty and complexity. Yet, despite the perceived novelty, the organisation had carried out many similar projects involving HVDC technology. Further, the company already had a longstanding relationship with the external contractor who was responsible for the core HVDC component: the players involved in the project were known to each other and, more importantly, trusted each other. The key team members – many of whom had worked on the earlier HVDC projects – had the same responsibilities as they had on those prior endeavours. The project was therefore not so unique.

Another important point is that the low-key management style of the project manager fit perfectly with the ethos and existing procedures of the organisation. Quoting from the paper:

The PM had been working at the division in over 30 years; he was a well-respected engineer and had an in-depth understanding of the inner life of the division. He manifested trust in his engineering colleagues and had no interest in challenging the traditions of his division. He did not fight for any formal power or official recognition as a PM. Once the HVDC contract was signed, he left most of the responsibility for its execution to the informal team of key engineers….the PM did not challenge the permanent engineering departments by fighting for more formalized procedures or methods in accordance with project management textbooks. On the contrary, his humble and informal way of coordination harmonized the existing norms and structures perfectly.

In short: the PM did his job in a way that was sensitive to the expectations of those who worked with him. He did not violate the (explicit or implicit) rules and norms of the organisation.

Analysis

When viewed through the lens of context and history, the following differences between the projects become apparent:

1. The transmission project, because of its strategic importance, had greater visibility and prestige in the organisation. The fact that it was well supported by management and had interesting technical challenges made the project attractive to engineers within the organisation. In contrast, the hydropower project was low key and had little support from top management. Consequently, there was little incentive for engineers to devote time to it.
2. The projects differed in degree of uniqueness: the organisation had more experience in HVDC transmission projects than it had in performing upgrades on functioning hydropower plants. The fact that the organisation had successfully carried out projects similar to the former reduced uncertainty associated with that project.
3. The projects were managed using very different approaches. The manager of the hydropower project used an approach that did not fit with the existing culture and ethos of the organisation whereas the transmission project was managed in a way that was sensitive to institutional norms and practices. It isn’t surprising, therefore, that the manager of the hydropower project had more difficulty in getting the job done.

The paradox is thus resolved: a full understanding of a project must take into account the context of the project, and  the culture/environment of the host organisation(s).

Discussion

The author’s analysis suggests that project success factors are context dependent: a methodology or approach that works in your organisation may not work in mine.   An implication of this is that prescriptive project management methodologies are unlikely to work when they are applied without due regard to organisational quirks and circumstances. Several other studies support this conclusion – see my posts on going beyond best practices and project management in post-bureaucratic organisations for examples.

Another important implication of this work is that intrinsic properties of projects (such as size, scope, technology etc.) are perhaps less significant than extrinsic ones (such as organisational culture). As Engwall puts it:

… these [latter] factors have little to do with the technical content of a project per se, but rather with how different stakeholders interpret a project in relation to the procedures and traditions of its surrounding context. The findings suggest, for instance, that important aspects of a project’s inner life are dependent on the level of deviation between the practices applied within the project and the knowledge base and institutional structure of its organizational context.

The research also calls into question the notion of a project as a unique undertaking. Engwall’s contention, supported by the case studies, is that projects typically consist of a mix of unique and repetitive processes. We never begin with a completely blank slate, nor do we ever repeat exactly the same project.

Endnote

The obvious takeaway from the paper is that project managers need to be aware of the history, culture, strategic imperatives and social dynamics of the organisations within which their projects are embedded. They should then tailor their  approach based on their knowledge of these factors. As the case studies illustrate, project management best practices may count for little if historical and contextual factors are ignored.

At a more philosophical level, the paper illustrates that project success is contingent on a host of factors that at first sight have nothing to do with projects.  Often these factors will become evident only in hindsight,  after the project has run its course.   Managers can therefore never be absolutely certain that they have considered all the key variables on their projects. Consequently they should view any actions they take as being provisional,  subject to change as their knowledge  of the project evolves.

What should I do now? A bedtime story about dialogue mapping

It was about half past eight in the evening a couple of weeks ago; I was sitting at my computer at home, writing up some notes for a blog post on issue mapping.

“What are you drawing?” asked my eight year old, Rohan. I hadn’t noticed him. He had snuck up behind me quietly, and was watching me draw an IBIS map. (Note: see my post entitled, the what and whence of issue-based information systems for a quick introduction to IBIS)

“Go to bed,” I said, still looking at the screen. It was past his bedtime.

“…but what are you drawing. What are those questions and arrows and stuff?”

A few minutes won’t hurt, I thought. I turned to him and explained the basics of the notation and how it worked.

“But what good is it,” he asked.

“Good question,” I said. “It has many uses, but one of the most important ones is that it can help people make good decisions.”

“Decisions about what?”

“Anything, I said, “for example: you may want to decide what you should do right now. Well, IBIS can help you make that decision.”

“How?”

“I’ll have to show you,” I said, “and I can’t because you have to go to bed now.”   What a cop out, I thought to myself, as I said those words.

“Come on, dad – just a few minutes. I really want to know how it can help me make a decision about what I should do now.”

“You should go to bed.”

“How do I know that’s a good decision? Let’s see what IBIS says,” said the boy.

Brilliant! It was checkmate. I relented.

———

“OK, “ I said, opening a new map in Compendium and drawing a question node. “Every IBIS map begins with a question – we call it the root question. Our root question is: What should I do now?”

I typed in the root question and asked him: “So, tell me: what are the different things you could do now.”

He thought for a bit and said, “I could go to sleep but that’s boring.”

“Good. There are actually two things you’ve said there – an idea (go to sleep) and an argument against it  (its boring). Let’s put that down in the map. In an IBIS map, an idea is shown as a light-bulb (as in a comic) and an argument against it by a  minus sign.”

The map with the root question along with Rohan’s  first response and argument is shown in Figure 1.

Figure 1

He looked at the map and said, “There’s another minus I can think of – it is hard to sleep so early.”

I put that point in and said, “I’m sure you could also think of some plus points for sleeping early.”

“Yes,” he said, “I can get up early and do stuff.”

“What stuff?”

“I can play Wii before I go to school.”

“OK let’s put all that into the map,” I said.  “See, an argument supporting an idea is shown as a plus sign. Then, I asked you to explain a bit more about why getting up early is a good thing. Your answer goes into  the map as another  idea. Notice, also, that the map develops from right to left, starting from the root question.”

The map at this point of the discussion is shown in Figure 2.

Figure 2

“What else can you do now?”

“I can talk to you,” said Rohan.

“And what are the plus points of that?” I asked.

“It is interesting to talk to you.” Ah, the boy has the makings of a diplomat…

“The minus points?”

“You are tired and crabby”

OK, may be he isn’t a diplomat…

The map at this point is shown in Figure 3.

Figure 3

“What else can you do,” I asked, as I cleaned up the map a bit.

“I could spend some time with Vikram.” (Vikram is Rohan’s 4 month old brother).

“What are the plus points of doing that?”

“He does funny things and he’s cute.”

“That’s two points, ” I said, adding them to the map. Then I asked, “What kinds of funny things?”

“He gurgles, smiles and blows spit bubbles.”

“Great,” I said, adding those points as elaborations of “does funny things”.

Rohan said, “I forgot. Vik is asleep so I can’t play with him.”

“OK, so that’s a minus point that rules out the choice,” I said, adding it as an  argument against the idea. The map at this point is shown in Figure 4.

Figure 4

“Can you think of anything else you can do?” I asked.

He thought for a while and replied, “I could read.”

“OK,” I said. “What are the plus and minus points of that.”

“It’s interesting,” he said, and then in the same breath added, “but I have nothing to new to read.”

I put these points in as arguments for and against reading. The map at this point is shown in Figure 5.

Figure 5

After finishing I asked, “Anything else you want to add.”

“I could just stay up and watch a movie,”  he said.

I stopped myself from vetoing that outright. Instead, I put the point in and asked,” Why do you want to stay up and watch a movie?”

“It’s fun,” he said.

“May be so, but a movie would take too long and you have school tomorrow.”

“School’s boring.”

“I’ll note your point,” I said, “but I’m afraid I have to veto that option.”

“I was just trying it out, dad.”

“I know,” I said,  as I updated the  map (see Figure 6).

Figure 6

“Can you think of anything else you could do?”

“No.”

“OK, let’s look at where we are. Have a look at the map and tell me what you think.”

Rohan looked at the map for a bit and said, “It shows me all my choices and gives me reasons to choose or not to choose them.”

“Does that help you decide what you should do now?”

“Sort of,” he said, “I know I can’t spend time with Vik because he’s asleep. I can’t talk to you because you’re tired and might get crabby. I can’t stay up and watch a movie because you won’t let me.”

“So what can you do?”

“I can read or go to sleep”

“But you have nothing new to read.,” I pointed out.

“Yes, but I think I could find something that I would like to read again…Yes, I know what I will do –  I’ll read  for a while and then go to sleep.”

“Sounds like a good idea – that way you get to do two of the things on the list.” I said.

“This IBIS stuff is cool. I think I’ll talk about it at my news this Thursday. It is free choice.”  (News is a 2-3 minute presentation that all kids in class get to do once a week. Most often the topic is assigned beforehand, but there’s one free-choice session per term where the kids can talk about anything they want to)

“Great idea,” I said, “I’ll help you make some notes and map images tomorrow. Now you’d really better go off to bed before your mum comes in and gets upset at us both.”

”Good night, dad”

“’night, Rohan”

Six ways in which project estimates go wrong

Despite the increasing focus on  project estimation, the activity still remains  more guesswork than art or science.  In his book on the fallacies of software engineering,  Robert Glass has this to say about it:

Estimation, as you might imagine, is the process by which we determine how long a project will take and how much it will cost. We do estimation very badly in the software field. Most of our estimates are more like wishes than realistic targets. To make matters worse, we seem to have no idea how to improve on those very bad practices. And the result is, as everyone tries to meet an impossible estimation target, shortcuts are taken, good practices are skipped, and the inevitable schedule runaway becomes a technology runaway as well…

Moreover, he suggests that poor estimation is one of the top two causes of project failure.

Now, there are a number of reasons why project estimates go wrong,  but in my experience there are half-dozen standouts. Here they are, in no particular order:

1.  False analogies: Project estimates based on historical data are generally considered to be more reliable than those developed using other methods such as expert judgement (see this article, from the MS Project support site for example). This is fine and good as long as one uses data from historical projects that are identical to the one at hand in relevant ways. Problem is, one rarely knows what is relevant and what isn’t.  It is all too easy too select a project that is superficially similar to the one at hand, but actually differs in critical ways.  See my posts on false analogies and the reference class problem for more on this point.

2.  False precision: Project estimates are often quoted as single numbers rather than ranges. Such estimates are incorrect because they ignore the fact that uncertain quantities should be quantified by a range of numbers (or more accurately, a distribution) rather than  point values. As Dr. Sam Savage emphasises in his book, The Flaw of Averages: an uncertain quantity is a shape, not a number (see my review of the book for more on this point). In short, an estimate quoted as a single number is almost guaranteed to be incorrect.

3.  Estimation by decree: It should be obvious that estimation must be done by those who will do the work. Unfortunately this principle is one of the first to be sacrificed on Death March Projects. In such projects, schedules  are shoe-horned into  predetermined timelines, with estimates cooked up by those who have little or no idea of the actual effort involved in doing the work.

4.   Subjectivity: This is where estimates are plucked out of thin air and “justified” based on gut-feel and other subjective notions. Such estimates are prone to overconfidence and a range of other cognitive biases. See my post on cognitive biases as project meta-risks for a detailed discussion of how these biases manifest themselves in project estimates.

5.  Coordination neglect: Projects consist of diverse tasks that need to be coordinated and integrated carefully. Unfortunately, the time and effort needed for coordination and integration  is often underestimated (or even totally overlooked)  by project decision makers. This is referred to as coordination neglect. Coordination neglect is a problem in projects of all sizes, but  is generally more significant for projects involving large teams (see this paper for an empirical study of the effect of team size on coordination neglect).  As one might imagine, coordination neglect also becomes a significant problem in projects that consist of a large number of dependent tasks  or have a large number of external dependencies.

6.  Too coarse grained: Large tasks are made up of smaller tasks strung together in specific ways. Consequently, estimates for large tasks should be built up from estimates for the smaller sub-tasks. . Teams often short-circuit the process by attempting to estimate the large task directly. Such estimates usually turn out to be incorrect because sub-tasks  are overlooked. Another problem is coordination neglect  between sub-tasks, as discussed in the earlier point. It is true – – the devil is always in the details.

I should emphasise that the above list based on personal experience, not on any systematic study.

I’ll conclude this piece with another fragment from Glass, who  is not very optimistic about improvements in the area of project estimation. As he states in his book:

The bottom line is that, here in the first decade of the twenty-first century, we don’t know what constitutes a good estimation approach, one that can yield decent estimates with good confidence that they will really predict when a project will be completed and how much it will cost. That is a discouraging bottom line. Amidst all the clamor to avoid crunch mode and end death marches, it suggests that so long as faulty schedule and cost estimates are the chief management control factors on software projects, we will not see much improvement.

True enough,  but being aware of the ways in which estimates can go wrong is the first step towards improving them.