Symptoms, not causes: a systems perspective on project failure

Introduction

A quick search reveals that the topic of project failure gets a fair bit of attention on the Internet. Many of the articles identify factors such as lack of executive support or incomplete/misunderstood requirements as the prime causes of failure. In this post I use concepts from systems theory  to argue that commonly identified “causes” of project failure – such as the ones noted above – are symptoms rather than causes.  I then  surface the real causes of project failure by taking a systems perspective –  a viewpoint that considers the project and the hosting organisation as a whole.

As I will argue, project failures can  often be traced back to dysfunctional structures and processes within the organisation.  These factors usually have little to do with the project directly and are therefore not always obvious at first sight.

Setting the scene

Readers who have waded through the vast literature on failed projects will have noted that there are diverse opinions on the prime reasons for failure. It would take me too long to wade through these articles, so I’ll just pick a source that is well known even  if not entirely credible: The Chaos Report by the Standish Group.  As per this summary  the Chaos Report 2009 claimed the following were the top three causes of project failure:

1. Lack of user input.
2. Incomplete or changing requirements/specifications.
3. Lack of executive support.

Although the report lists the top ten factors, in the interests of space I’ll focus on just these three.

I should mention that the report refers to the above as  “Project Challenged Factors.” Grammar issues aside, this is a somewhat strange way of putting it. Anyway, I interpret this phrase to mean reasons (or driving factors) for failure.

Systems theory and projects

First up, what exactly is a system?

Here is a jargon-free definition from the wonderful book by Donella Meadows entitled, Thinking in Systems: A Primer. Incidentally, I highly recommend the book as an easy-to-read and engaging introduction to systems theory:

A system is a set of things interconnected in such a way that they produce their own pattern of behaviour over time. The system may be buffeted, constricted , triggered or driven by outside forces. But its response to these is characteristic of itself and is seldom simple in the real world.  (italics mine)

The word interconnected is  important because it tells us that when we study something from a systems perspective, we must identify all the important connections it has with its environment.  In the case of projects, the important connections are fairly obvious. A project is usually carried out within an organisational setting so it would have many connections to the hosting organisation. Chief amongst these is the fact that  a project is staffed and resourced by the hosting organisation (or by an organisation designated by it). Another important connection is that  a project will affect  different stakeholder groups within the hosting organisation in different ways.  However,  since all stakeholders have ongoing organisational roles  that go beyond the project, the project is not their only interest. This is a key point to which we will return later.

The phrases “own pattern of behaviour over time” and “characteristic of itself”  tell us that systems have a unique pattern of behaviour that is characteristic to them.  The important point to note is that characteristic behaviour implies that different systems  that have the same signature – i.e. identifying features – will tend to behave in the same way. From the perspective of projects  this tells us that projects within similar organisations will evolve in similar ways.

A systems view of the causes of project failure

With only this very brief look at systems theory we are now in a position to get some insights into  the real causes of project failure. As mentioned earlier, we will focus on the top three reasons ala Standish.

Lack of user input

First up, consider lack of user input. Systems  theory tells us that we need to look at the issue from the perspective of the project and the organisation. From this point of view it is clear that users who have been asked to work on the project in addition to their normal duties will view the project as a burden rather than something that may benefit them in the future.

This by itself is not a new insight. In fact, project management gurus have talked themselves hoarse about the need to free up resources etc. However, the point is that organisational structures typically work against this.  Firstly, people who are asked to work on a project know that it is an initiative that will end in a finite (usually, reasonably short) time. Therefore, their long terms interests lies in their ongoing roles rather than  in projects. Secondly, most organisations are  still structured along functional lines and people’s identities are anchored within their functional reporting lines rather than ephemeral project hierarchies.

Changing requirements

The issue of changing requirements and specifications can also be understood from a systems point of view.  A characteristic of many systems is that they are stable – i.e. they resist change.  Organisations are typically stable systems – they tend to retain their identity despite changes in their environment.  One of the characteristics of such organisations is that people in them tend to think and act in set ways – that is, their actions and thinking processes follow well worn patterns to the point where they do not need to think about what they do too deeply.

One of the consequences of this is that when they are asked for requirements users often provide incomplete description of what they do, leaving out significant items that are obvious to them (but not to the analysts who are gathering requirements!).  Although I don’t have the figures to back this – I speculate that a fair proportion of  changes in requirements  are the result of  inadequate detail or thought put into developing  initial requirements. The point is users and sponsors don’t necessarily  see these as changes, but project teams do.

Lack of executive support

Finally, let’s look at the the problem of lack of executive support.  Project sponsors usually hold important executive-level positions within the hosting organisation. By virtue of their positions,  they have a number of important things that compete for their attention. A project – even an important one – is only one of many things going on in an organisation at any one time.  Moreover,  organisational priorities do shift, perhaps more often than executives may want to admit.  So a project that was the key focus yesterday may be superseded by other priorities today.

There are of course many other ways in which project sponsors can be distracted, but I think I’ve made my point which is that lack of executive support is due to features that are inherent in organisations. So no amount of forcing executives to pay attention to their projects is going to work unless the entire system  (project + organisation) changes.  And this is difficult if not impossible to achieve because stable systems such as organisations tend to resist change, and therefore continue to display their characteristic patterns of behaviour.

Discussion

So we see that the causes of project failure can be traced back to the organisations in which they are embedded.  Specifically, they lie in unwritten norms and formal policies that dictate how the hierarchy operates and how things are done within the organisation. The most important consequence of this is that standard fixes (of encouraging user input and executive support, or instituting change management, say)   will not cure the problem of project failure because they do not address the dysfunctional norms and policies that are the root cause of failure.

The above is not news. In fact, the matrix organisation structure was proposed as a response to the need for “project friendly” organisations. I’m no expert on matrix organisations so I will leave it to others to comment on how successful they are. The only point I would make is that  in my experience multiple reporting lines (even if dotted and solid) do not work too well. There are always conflicting interests that cause divided loyalties and conflicting interests.

So,  the natural question is : what – if anything – can we do about this? The answer is implicit in the foregoing paragraphs. One has to align the project with the organisation, not just at the level of objectives or structure, but also in operational matters such as timelines, budget and resources –  the very things that make up the so-called  iron triangle.  The best time to do this is at the front-end of the project,   i.e. the start. At this time, the person(s) driving the project have to engage all stakeholders who will be affected by the initiative and find out their motivations, interests and – most importantly – their concerns regarding the project.  If this discussion happens in an open and frank manner,  it should surface the issues highlighted In the previous section.  Since the discussion takes place even before the project starts, there is at least some hope of addressing these concerns.

There are many ways to structure and facilitate such discussions. Check out this post for an introduction to one and have a look at my book co-authored with Paul Culmsee for much more. That said, one doesn’t need any particular technique – the willingness to discuss difficult matters openly and an openness to other points of view is all that’s needed. That, however, is not always easy to come by…

Conclusion

We have seen that the top causes of project failure can be traced back to the hierarchies and incentive systems of the hosting organisation. Therefore, superficial attempts to fix the problem at the level of individual projects (or even a PMO) will not work. The only hope of addressing the root causes of project failure is to focus on the systemic dysfunctions that cause them.