Eight to Late

Sensemaking and Analytics for Organizations

Archive for the ‘portfolio management’ Category

Symptoms, not causes: a systems perspective on project failure

with 21 comments


A quick search reveals that the topic of project failure gets a fair bit of attention on the Internet. Many of the articles identify factors such as lack of executive support or incomplete/misunderstood requirements as the prime causes of failure. In this post I use concepts from systems theory  to argue that commonly identified “causes” of project failure – such as the ones noted above – are symptoms rather than causes.  I then  surface the real causes of project failure by taking a systems perspective –  a viewpoint that considers the project and the hosting organisation as a whole.

As I will argue, project failures can  often be traced back to dysfunctional structures and processes within the organisation.  These factors usually have little to do with the project directly and are therefore not always obvious at first sight.

Setting the scene

Readers who have waded through the vast literature on failed projects will have noted that there are diverse opinions on the prime reasons for failure. It would take me too long to wade through these articles, so I’ll just pick a source that is well known even  if not entirely credible: The Chaos Report by the Standish Group.  As per this summary  the Chaos Report 2009 claimed the following were the top three causes of project failure:

  1. Lack of user input.
  2. Incomplete or changing requirements/specifications.
  3. Lack of executive support.

Although the report lists the top ten factors, in the interests of space I’ll focus on just these three.

I should mention that the report refers to the above as  “Project Challenged Factors.” Grammar issues aside, this is a somewhat strange way of putting it. Anyway, I interpret this phrase to mean reasons (or driving factors) for failure.

Systems theory and projects

First up, what exactly is a system?

Here is a jargon-free definition from the wonderful book by Donella Meadows entitled, Thinking in Systems: A Primer. Incidentally, I highly recommend the book as an easy-to-read and engaging introduction to systems theory:

A system is a set of things interconnected in such a way that they produce their own pattern of behaviour over time. The system may be buffeted, constricted , triggered or driven by outside forces. But its response to these is characteristic of itself and is seldom simple in the real world.  (italics mine)

The word interconnected is  important because it tells us that when we study something from a systems perspective, we must identify all the important connections it has with its environment.  In the case of projects, the important connections are fairly obvious. A project is usually carried out within an organisational setting so it would have many connections to the hosting organisation. Chief amongst these is the fact that  a project is staffed and resourced by the hosting organisation (or by an organisation designated by it). Another important connection is that  a project will affect  different stakeholder groups within the hosting organisation in different ways.  However,  since all stakeholders have ongoing organisational roles  that go beyond the project, the project is not their only interest. This is a key point to which we will return later.

The phrases “own pattern of behaviour over time” and “characteristic of itself”  tell us that systems have a unique pattern of behaviour that is characteristic to them.  The important point to note is that characteristic behaviour implies that different systems  that have the same signature – i.e. identifying features – will tend to behave in the same way. From the perspective of projects  this tells us that projects within similar organisations will evolve in similar ways.

A systems view of the causes of project failure

With only this very brief look at systems theory we are now in a position to get some insights into  the real causes of project failure. As mentioned earlier, we will focus on the top three reasons ala Standish.

Lack of user input

First up, consider lack of user input. Systems  theory tells us that we need to look at the issue from the perspective of the project and the organisation. From this point of view it is clear that users who have been asked to work on the project in addition to their normal duties will view the project as a burden rather than something that may benefit them in the future.

This by itself is not a new insight. In fact, project management gurus have talked themselves hoarse about the need to free up resources etc. However, the point is that organisational structures typically work against this.  Firstly, people who are asked to work on a project know that it is an initiative that will end in a finite (usually, reasonably short) time. Therefore, their long terms interests lies in their ongoing roles rather than  in projects. Secondly, most organisations are  still structured along functional lines and people’s identities are anchored within their functional reporting lines rather than ephemeral project hierarchies.

Changing requirements

The issue of changing requirements and specifications can also be understood from a systems point of view.  A characteristic of many systems is that they are stable – i.e. they resist change.  Organisations are typically stable systems – they tend to retain their identity despite changes in their environment.  One of the characteristics of such organisations is that people in them tend to think and act in set ways – that is, their actions and thinking processes follow well worn patterns to the point where they do not need to think about what they do too deeply.

One of the consequences of this is that when they are asked for requirements users often provide incomplete description of what they do, leaving out significant items that are obvious to them (but not to the analysts who are gathering requirements!).  Although I don’t have the figures to back this – I speculate that a fair proportion of  changes in requirements  are the result of  inadequate detail or thought put into developing  initial requirements. The point is users and sponsors don’t necessarily  see these as changes, but project teams do.

Lack of executive support

Finally, let’s look at the the problem of lack of executive support.  Project sponsors usually hold important executive-level positions within the hosting organisation. By virtue of their positions,  they have a number of important things that compete for their attention. A project – even an important one – is only one of many things going on in an organisation at any one time.  Moreover,  organisational priorities do shift, perhaps more often than executives may want to admit.  So a project that was the key focus yesterday may be superseded by other priorities today.

There are of course many other ways in which project sponsors can be distracted, but I think I’ve made my point which is that lack of executive support is due to features that are inherent in organisations. So no amount of forcing executives to pay attention to their projects is going to work unless the entire system  (project + organisation) changes.  And this is difficult if not impossible to achieve because stable systems such as organisations tend to resist change, and therefore continue to display their characteristic patterns of behaviour.


So we see that the causes of project failure can be traced back to the organisations in which they are embedded.  Specifically, they lie in unwritten norms and formal policies that dictate how the hierarchy operates and how things are done within the organisation. The most important consequence of this is that standard fixes (of encouraging user input and executive support, or instituting change management, say)   will not cure the problem of project failure because they do not address the dysfunctional norms and policies that are the root cause of failure.

The above is not news. In fact, the matrix organisation structure was proposed as a response to the need for “project friendly” organisations. I’m no expert on matrix organisations so I will leave it to others to comment on how successful they are. The only point I would make is that  in my experience multiple reporting lines (even if dotted and solid) do not work too well. There are always conflicting interests that cause divided loyalties and conflicting interests.

So,  the natural question is : what – if anything – can we do about this? The answer is implicit in the foregoing paragraphs. One has to align the project with the organisation, not just at the level of objectives or structure, but also in operational matters such as timelines, budget and resources –  the very things that make up the so-called  iron triangle.  The best time to do this is at the front-end of the project,   i.e. the start. At this time, the person(s) driving the project have to engage all stakeholders who will be affected by the initiative and find out their motivations, interests and – most importantly – their concerns regarding the project.  If this discussion happens in an open and frank manner,  it should surface the issues highlighted In the previous section.  Since the discussion takes place even before the project starts, there is at least some hope of addressing these concerns.

There are many ways to structure and facilitate such discussions. Check out this post for an introduction to one and have a look at my book co-authored with Paul Culmsee for much more. That said, one doesn’t need any particular technique – the willingness to discuss difficult matters openly and an openness to other points of view is all that’s needed. That, however, is not always easy to come by…


We have seen that the top causes of project failure can be traced back to the hierarchies and incentive systems of the hosting organisation. Therefore, superficial attempts to fix the problem at the level of individual projects (or even a PMO) will not work. The only hope of addressing the root causes of project failure is to focus on the systemic dysfunctions that cause them.

Written by K

June 18, 2013 at 9:18 pm

A visit from the methodology police – a PMO satire

with 15 comments

They came for me at 11:00 am.

I  was just settling down to finishing that damned business case when I heard the rat-a-tat-tat on my office door. “Come in,” I said, with a touch of irritation in my voice.

The door opened and there they were. They looked at me as though I was something that had crawled out from under a rock. “Mr. Hersey, I presume,” said the taller, uglier one.

“Yes, that’s me.”

“Joe  Hersey?” He asked, wanting to make sure before unloading on me.

“Yes, the one and only,” I said, forcing a smile. I had a deep sense of foreboding now: they looked like trouble;  I knew they couldn’t be enquiring after my welfare.

“You need to come with us,” said the shorter one. I did imply he was handsomer of the two, but I should clarify that it was a rather close call.

“I have better things to do than follow impolite summons from people I don’t know. I think you should talk to my manager. In fact, I will take you to him,” I replied, rising from my chair. “He won’t be happy that you’ve interrupted my business case. He wants it done by lunchtime,” I added, a tad smugly.

“We’ve already seen him. He knows. I would advise you to come with us. It would make life easier for everyone concerned,”  I forget which one of the two said this.

“What is going on?” I asked, toning down my irritation. To be honest, I had no clue what they were on about.

“We’re the methodology police,” they said in unison. I guess they’d had a fair bit of practice scaring the crap out of hapless project managers. “We’re from the PMO,” they added unnecessarily – I mean, where else could they be from.

“Holy s**t,” I said to myself. I was in big trouble.

“Well, Hersey,” said the short one, “I think you owe the PMO an explanation.” Ah, I loved his use of the third person– not “us” but “the PMO.”

We were seated at a table in a meeting room deep in the bowels of the PMO: windowless, with low wattage lighting sponsored by one of those new-fangled, energy-saving, greenie bulbs . The three chairs were arranged in interrogation mode , with the two goons on one side and me – Joseph M. Hersey, Project Manager Extraordinaire – on the other.

I was in trouble alright, but I have this perverse streak in me, “I don’t know what you are talking about,” I said, feeling a bit like a hero from a Raymond Chandler novel. I knew what I had done, of course. But I also knew that I was one of the good guys. The clowns sitting opposite me were the forces of evil…such thoughts, though perverse, lifted my spirits.

I must have smiled because the tall one said, “You think this is funny, do you? We have a direct line to the board and we could make life really unpleasant for you if you continue this uncooperative attitude.”

That was bad. I did not want to be hauled up in front of the big cheese. If I was branded a troublemaker at that level, there would be no future for me in the company. And to be absolutely honest, I actually enjoyed working here – visits from the methodology police excepted, of course.

“OK, tell me what you want to know,” I said resignedly.

“No, you tell us, Hersey. We want to hear the whole story of your subversion of process in your own words. We’ll stop you if we need any clarification.” Again, I forget which one of the two said this. Understandable, I think – I was pretty stressed by then.

Anyway, there is no sense in boring you with all the PMO  and process stuff. Suffice to say, I told them how I partitioned my big project into five little ones, so that each mini project would fall below the threshold criteria for major projects and thus be exempt from following the excruciating methodology that our PMO had instituted.

Process thus subverted, I  ran each of the mini projects separately, with deliverables from one feeding into the next. I’d got away with it; with no onerous  procedures to follow I was free to devise my own methodology, involving nothing more complicated than a spreadsheet updated daily following informal conversations with team members and stakeholders. All this held together – and, sorry, this is going to sound corny – by trust.

The methodology cops’ ears perked up when they heard that word, “Trust!” they exclaimed, “What do you mean by trust?”

“That’s when you believe people will do as they say they will,” I said. Then added, “A concept that may be foreign to you.” I regretted that snide aside as soon as I said it.

“Look, “ said the uglier guy, “I suggest you save the wisecracks for an audience that may appreciate them. “You are beginning to annoy me and a report to the board is looking like a distinct possibility if you continue in this vein.”

I have to say, if this guy had a lot of patience if he was only just “beginning to get annoyed.” I was aware that I had been baiting him for a while. Yes, I do know when I do that. My wife keeps telling me it will get me into trouble one day. May be today’s the day.

“…I do know what trust is,” the man continued, “but I also know that you cannot run a project on warm and fuzzy notions such as trust, sincerity, commitment etc. The only thing I will trust are written signed off project documents.”

Ah, the folly, the folly. “Tell me this, what would you prefer – project documentation as per the requirements of your methodology or a successful project.”

“The two are not mutually exclusive. In fact, methodology improves the chance of success.”

“No it doesn’t,” I retorted.

“It does,” he lobbed back.

Jeez, this was beginning to sound like recess in the local kindergarten. “Prove it,” I said, staking my claim to the title of King of Kindergarten Debates.

“There are several studies that prove the methodologies efficacy,” said the short one, “but that is not the point.”

“All those studies are sponsored by the Institute,” I said, referring to the August Body that maintains the standard. “so there is a small matter of vested interest….anyway, you say that isn’t the point. So what is your point then?.”

“The methodology is an internal requirement, so you have to follow It regardless. We could have a lot of fun debating it, but that is neither here no there. Compliance is mandatory, you have no choice.”

“I did comply,” I said, “none of my projects were over the threshold, so I did not need to follow the methodology.”

“That was subterfuge – it was one project that you deliberately divided into five so that you could bypass our processes.”

I was getting tired and it was close to my lunchtime. “OK, fair point“ I said, “I should not have done that. I will not do it again. Can I go now?”

“Hmm,” they said in unison.  I don’t think either of them believed me. “That’s not good enough.”

I sighed. “What do you want  then?” I asked, weary of this pointless drama.

“You will read and sign this form,” said the short one, “declaring you have been trained in the PMO processes – which you were last year, as you well know – and that you will follow the processes henceforth. I particularly urge you to read and digest the bit about the consequences of non-compliance.” He flicked the form in my direction.

I was not surprised to see that the form  was a multi-page affair, written in 8pt bureaucratese, utterly incomprehensible to mere mortals such as I. I knew I would continue to bypass or subvert processes that made no sense to me, but I also knew that they needed me to sign that form – their boss would be very unhappy with them if I didn’t.  Besides, I didn’t want to stay in that room a second longer than necessary.

“OK, where do I sign,” I said, picking up a pen that lay on the table.

“Don’t you want to read it.”

“Nah,” I said, “I have a pretty fair idea of what it’s about.”

I signed.

“We’re done, Hersey. You can go back to your business case now. But you can be sure that you are on our radar now. We are watching you.”

“Well Gents, enjoy the show. I promise, to lead a faultless life henceforth. I will be a model project manager,” I said as I rose to leave.

“We’re counting on it Hersey.  One more violation and you are in deep trouble.”

I refrained from responding with a wisecrack as I exited, leaving them to the paperwork that is their raison d’etre.

Projects as networks of commitments – a paper preview

with 8 comments

Mainstream project management standards and texts tend to focus on the tools, techniques and processes needed to manage projects. They largely ignore the fact that projects are conceived, planned, carried out and monitored by people. In a paper entitled, Towards a holding environment: building shared understanding and commitment in projects,  Paul Culmsee and I present a viewpoint that puts people and their project-relevant concerns at the center of projects. This post is a summary of the main ideas of the paper which is published in the May 2012 issue of the International Journal of Managing Projects in Business.

Conventional approaches to project management tend to give short shrift to the social aspects of projects –   issues such as stakeholders with differing viewpoints regarding the rationale and goals of a project. Our contention is that problems arising from stakeholder diversity are best resolved by helping stakeholders achieve a shared (or common) understanding of project goals and, based on that, a shared commitment to working towards them. Indeed, we believe this is  a crucial but often overlooked facet of  managing the early stages of a project.

One of the prerequisites to achieving shared understanding is open dialogue –dialogue that is free from politics, strategic behaviours and power games that are common in organisations. Details of what constitutes such dialogue and the conditions necessary for it are described by the philosopher Juergen Habermas in his theory of communicative rationality. Our paper draws extensively on Habermas’ work and also presents a succinct summary of the main ideas of communicative rationality.

The conditions required for open dialogue in the sense described by Habermas are:

  1. Inclusion: all affected stakeholders should be included in the dialogue.
  2. Autonomy: all participants should be able to present their viewpoints and debate those of others independently.
  3. Empathy: participants must be willing to listen to viewpoints that may be different from theirs and make the effort to understand them.
  4. Power neutrality: differences in status or authority levels should not affect the discussion:
  5. Transparency: participants must be completely honest when presenting their views or discussing those of others.

We call an environment which fosters open dialogue a holding environment.  Although a holding environment as characterised above may seem impossible to create, it turns out that an alliance-based approach to projects can approximate the conditions necessary for one. In brief, alliancing is an approach to projects in which different stakeholders agree, by contract, to work collaboratively to achieve mutually agreed goals while sharing risks and rewards in an equitable manner. There are a fair number of large projects that have been successfully delivered using such an approach (see the case studies on the Center for Collaborative Contracting web site).

Once such an approach is endorsed by all project stakeholders, most of the impediments to open dialogue are removed. In the paper we use a case study to illustrate how stakeholder differences can be resolved in such an environment. In particular we show how project-relevant issues and the diverse viewpoints on them can be captured and reconciled using the IBIS (Issue-based information system) notation (see this post for a quick introduction to IBIS).  It should be noted that our concept of a  holding environment does not require the use of IBIS;  any means to capture issues, ideas and arguments raised in a debate will work just as well.  The aim is to reach a shared understanding,  and once stakeholders do this –  using IBIS or any other means – they are able to make mutual commitments to action.

It should be emphasised that an alliance-based approach to projects  takes a fair bit of effort and commitment from all parties to implement successfully. In general such effort is justifiable only for very large projects,  typically public infrastructure projects (which is why many government agencies are interested in it). It is interesting to speculate how such an approach can be “scaled down” to smaller projects like the ones undertaken by corporate IT departments. Unfortunately such speculations are not permitted in research papers. However,  we discuss some of these at length in our book,  The Heretic’s Guide to Best Practices.

In their  ground-breaking book on design Terry Winograd and Fernando Flores describe organisations as networks of commitments.  We believe this metaphor is appropriate for  projects too. As we state in the paper,  “Organisations and projects are made up of people, and it is the commitments that people make (to carry out certain actions) that make organisations or projects tick. This metaphor – that projects are networks of commitments – lies at the heart of  the perspective we propose in this paper. The focus of project management ought to be on how commitments are made and maintained through the life of a project.

%d bloggers like this: