Symptoms, not causes: a systems perspective on project failure

Introduction

A quick search reveals that the topic of project failure gets a fair bit of attention on the Internet. Many of the articles identify factors such as lack of executive support or incomplete/misunderstood requirements as the prime causes of failure. In this post I use concepts from systems theory  to argue that commonly identified “causes” of project failure – such as the ones noted above – are symptoms rather than causes.  I then  surface the real causes of project failure by taking a systems perspective –  a viewpoint that considers the project and the hosting organisation as a whole.

As I will argue, project failures can  often be traced back to dysfunctional structures and processes within the organisation.  These factors usually have little to do with the project directly and are therefore not always obvious at first sight.

Setting the scene

Readers who have waded through the vast literature on failed projects will have noted that there are diverse opinions on the prime reasons for failure. It would take me too long to wade through these articles, so I’ll just pick a source that is well known even  if not entirely credible: The Chaos Report by the Standish Group.  As per this summary  the Chaos Report 2009 claimed the following were the top three causes of project failure:

1. Lack of user input.
2. Incomplete or changing requirements/specifications.
3. Lack of executive support.

Although the report lists the top ten factors, in the interests of space I’ll focus on just these three.

I should mention that the report refers to the above as  “Project Challenged Factors.” Grammar issues aside, this is a somewhat strange way of putting it. Anyway, I interpret this phrase to mean reasons (or driving factors) for failure.

Systems theory and projects

First up, what exactly is a system?

Here is a jargon-free definition from the wonderful book by Donella Meadows entitled, Thinking in Systems: A Primer. Incidentally, I highly recommend the book as an easy-to-read and engaging introduction to systems theory:

A system is a set of things interconnected in such a way that they produce their own pattern of behaviour over time. The system may be buffeted, constricted , triggered or driven by outside forces. But its response to these is characteristic of itself and is seldom simple in the real world.  (italics mine)

The word interconnected is  important because it tells us that when we study something from a systems perspective, we must identify all the important connections it has with its environment.  In the case of projects, the important connections are fairly obvious. A project is usually carried out within an organisational setting so it would have many connections to the hosting organisation. Chief amongst these is the fact that  a project is staffed and resourced by the hosting organisation (or by an organisation designated by it). Another important connection is that  a project will affect  different stakeholder groups within the hosting organisation in different ways.  However,  since all stakeholders have ongoing organisational roles  that go beyond the project, the project is not their only interest. This is a key point to which we will return later.

The phrases “own pattern of behaviour over time” and “characteristic of itself”  tell us that systems have a unique pattern of behaviour that is characteristic to them.  The important point to note is that characteristic behaviour implies that different systems  that have the same signature – i.e. identifying features – will tend to behave in the same way. From the perspective of projects  this tells us that projects within similar organisations will evolve in similar ways.

A systems view of the causes of project failure

With only this very brief look at systems theory we are now in a position to get some insights into  the real causes of project failure. As mentioned earlier, we will focus on the top three reasons ala Standish.

Lack of user input

First up, consider lack of user input. Systems  theory tells us that we need to look at the issue from the perspective of the project and the organisation. From this point of view it is clear that users who have been asked to work on the project in addition to their normal duties will view the project as a burden rather than something that may benefit them in the future.

This by itself is not a new insight. In fact, project management gurus have talked themselves hoarse about the need to free up resources etc. However, the point is that organisational structures typically work against this.  Firstly, people who are asked to work on a project know that it is an initiative that will end in a finite (usually, reasonably short) time. Therefore, their long terms interests lies in their ongoing roles rather than  in projects. Secondly, most organisations are  still structured along functional lines and people’s identities are anchored within their functional reporting lines rather than ephemeral project hierarchies.

Changing requirements

The issue of changing requirements and specifications can also be understood from a systems point of view.  A characteristic of many systems is that they are stable – i.e. they resist change.  Organisations are typically stable systems – they tend to retain their identity despite changes in their environment.  One of the characteristics of such organisations is that people in them tend to think and act in set ways – that is, their actions and thinking processes follow well worn patterns to the point where they do not need to think about what they do too deeply.

One of the consequences of this is that when they are asked for requirements users often provide incomplete description of what they do, leaving out significant items that are obvious to them (but not to the analysts who are gathering requirements!).  Although I don’t have the figures to back this – I speculate that a fair proportion of  changes in requirements  are the result of  inadequate detail or thought put into developing  initial requirements. The point is users and sponsors don’t necessarily  see these as changes, but project teams do.

Lack of executive support

Finally, let’s look at the the problem of lack of executive support.  Project sponsors usually hold important executive-level positions within the hosting organisation. By virtue of their positions,  they have a number of important things that compete for their attention. A project – even an important one – is only one of many things going on in an organisation at any one time.  Moreover,  organisational priorities do shift, perhaps more often than executives may want to admit.  So a project that was the key focus yesterday may be superseded by other priorities today.

There are of course many other ways in which project sponsors can be distracted, but I think I’ve made my point which is that lack of executive support is due to features that are inherent in organisations. So no amount of forcing executives to pay attention to their projects is going to work unless the entire system  (project + organisation) changes.  And this is difficult if not impossible to achieve because stable systems such as organisations tend to resist change, and therefore continue to display their characteristic patterns of behaviour.

Discussion

So we see that the causes of project failure can be traced back to the organisations in which they are embedded.  Specifically, they lie in unwritten norms and formal policies that dictate how the hierarchy operates and how things are done within the organisation. The most important consequence of this is that standard fixes (of encouraging user input and executive support, or instituting change management, say)   will not cure the problem of project failure because they do not address the dysfunctional norms and policies that are the root cause of failure.

The above is not news. In fact, the matrix organisation structure was proposed as a response to the need for “project friendly” organisations. I’m no expert on matrix organisations so I will leave it to others to comment on how successful they are. The only point I would make is that  in my experience multiple reporting lines (even if dotted and solid) do not work too well. There are always conflicting interests that cause divided loyalties and conflicting interests.

So,  the natural question is : what – if anything – can we do about this? The answer is implicit in the foregoing paragraphs. One has to align the project with the organisation, not just at the level of objectives or structure, but also in operational matters such as timelines, budget and resources –  the very things that make up the so-called  iron triangle.  The best time to do this is at the front-end of the project,   i.e. the start. At this time, the person(s) driving the project have to engage all stakeholders who will be affected by the initiative and find out their motivations, interests and – most importantly – their concerns regarding the project.  If this discussion happens in an open and frank manner,  it should surface the issues highlighted In the previous section.  Since the discussion takes place even before the project starts, there is at least some hope of addressing these concerns.

There are many ways to structure and facilitate such discussions. Check out this post for an introduction to one and have a look at my book co-authored with Paul Culmsee for much more. That said, one doesn’t need any particular technique – the willingness to discuss difficult matters openly and an openness to other points of view is all that’s needed. That, however, is not always easy to come by…

Conclusion

We have seen that the top causes of project failure can be traced back to the hierarchies and incentive systems of the hosting organisation. Therefore, superficial attempts to fix the problem at the level of individual projects (or even a PMO) will not work. The only hope of addressing the root causes of project failure is to focus on the systemic dysfunctions that cause them.

The paradox of the learning organisation

Introduction

The term learning organisation  refers to an organisation that continually modifies itself in response to changes in its environment.   Ever since Peter Senge coined the term in his book, The Fifth Discipline, assorted consultants and academics have been telling us that a learning organisation is an ideal worth striving for.  The reality, however,  is that most organisations that undertake the journey actually end up in a place far removed  from this ideal. Among other things, the journey may expose managerial hypocrisies that contradict the very notion of a learning organisation.  In this post, I elaborate on the paradoxes of learning organisations, drawing on an excellent and very readable paper by Paul Tosey entitled, The Hunting of the Learning Organisation: A Paradoxical Journey.

(Note:  I should point out that the term learning organisation should be distinguished from organisational learning: the latter refers to processes of learning whereas the former is about an ideal type of organisation. See this paper for more on the distinction.)

The journey metaphor

Consultants and other experts are quick to point out that the path to a learning organisation is a journey towards an ideal that can never be reached.  Quoting from this paper, Tosey writes, “we would talk about the fact that, in some ways, the learning organization represented all of our collective best wishes for Utopia in the workplace.” As another example, Peter Senge writes of it being, “a journey in search of the experience of being a member of `a great team.”  Elsewhere, Senge  suggests that the learning organisation is a vision that is essentially unattainable.

The metaphor of a journey seems an apt one at first, but there are a couple of problems with it. Firstly, the causal connection between initiatives that purport to get one to the goal and actual improvements in an organisation’s capacity to learn  is tenuous and impossible to establish.  This suggests the journey is one without a map. Secondly, the process of learning about learning within the organisation – how it occurs, and how it is perceived by different stakeholders – can expose organisational hypocrisies and double-speak that may otherwise have remained hidden.  Thus instead of progressing towards the the ideal one may end up moving away from it.  Tosey explores these  paradoxes by comparing the journey of a learning organisation to  the one described in Lewis Carroll’s  poem, The Hunting of The Snark.

Hunting the Snark (and the learning organisation)

Carroll’s poem tells the story of ten characters who set of in search of a fabulous creature called a  Snark.  After many trials and tribulations, they end up finding out that the Snark is something else:  a not-so-pleasant creature called a Boojum. Tosey comments that the quest described in the poem is a superb metaphor for the journey towards a learning organisation. As he states:

Initially, when reflecting on personal experience of organizational events… I was struck by the potential of the dream-like voyage of fancy on which Carroll’s characters embarked as an allegory of the quest for the learning organization. Pure allegory has limitations. Through writing and developing the article I came to view the poem more as a paradigm of the consequences of human desire for, and efforts at, progress through the striving for ideals. In other words the poem expresses something about our `hunting’. In this respect it may represent a mythological theme,a profound metaphor more than a mere cautionary moral tale.

There are many interesting parallels between the hunt for the Snark and the journey towards a learning organisation. Here are a few:

The expedition to find the Snark is led by a character called the Bellman who asserts: “What I tell you three times is true.” This is akin to the assurances (pleas?) from experts who tell us (several times over) that it is possible to transform our organisations into ones that continually learn.

The journey itself is directionless because the Bellman’s map is useless. In Carroll’s words:

Other maps are such shapes, with their islands and capes!
But we’ve got our brave Captain to thank:
(So the crew would protest) “that he’s bought us the best—
A perfect and absolute blank!

Finally, the Snark is never found. In its stead, the crew find a scary creature called  a Boojum that has the power to make one disappear. Quoting from the poem:

In the midst of the word he was trying to say,
In the midst of his laughter and glee,
He had softly and suddenly vanished away—
For the Snark was a Boojum, you see.

The journey towards a learning organisation often reveals the Boojum-like dark side of organisations.  One common example of this is when the process of learning surfaces questions that are uncomfortable for those in power. Tosey relates the  following tale  which may be familiar to some readers,

…a multinational company intending to develop itself as a learning organization ran programmes to encourage managers to challenge received wisdom and to take an inquiring approach. Later, one participant attended an awayday, where the managing director of his division circulated among staff over dinner. The participant raised a question about the approach the MD had taken on a particular project; with hindsight, had that been the best strategy? `That was the way I did it’, said the MD. `But do you think there was a better way?’, asked the participant. `I don’t think you heard me’, replied the MD. `That was the way I did it’. `That I heard’, continued the participant, `but might there have been a better way?’. The MD fixed his gaze on the participants’ lapel badge, then looked him in the eye, saying coldly, `I will remember your name’, before walking away.

One could argue that a certain kind of learning – that of how the organisation learns – occurred here:  the employee learnt that certain questions were out of bounds. I think it is safe to say, though, that this was not the kind of learning that was intended by those who initiated the program.

In the preface to the poem, Carroll notes that the Bellman there is a rule  –  Rule 42 – which states, “No one shall speak to the Man at the Helm,” to which the Bellman (the leader) added, “and the Man at the Helm shall speak to no one.” This rendered communication between the helmsman and the crew impossible. In such periods the ship was not steered. The parallels between this and organisational life are clear: there is rarely open communication between the those steering the organisational ship and rank and file employees. Indeed, Tosey reformulates Rule 42 in organisational terms as, “the organization shall not speak to the supervision, and the  supervision shall not speak to the organization.” This, he tells us, interrupts the feedback loop between individual experience and the organisations which renders learning impossible.

(Note:  I can’t help but wonder if Douglas Adams’ famous  answer to the life universe and everything was inspired by Carroll’s rule 42…)

In the poem,  the ship sometimes sailed backwards when Rule 42 was in operation. Tosey draws a parallel between “sailing backwards” and unexpected or unintended consequence of organisational rules.  He argues that organisational actions can result in learning even if those actions were originally intended to achieve something else. The employee in the story above learnt something about the organisational hierarchy and how it worked.

Finally, it is a feature of Rule-42-like rules that they cannot be named. The employee in the story above could not  have pointed out that the manager was acting in a manner that was inconsistent with the intent of the programme – at least not without putting his own position at risk. Perhaps that in itself is a kind of learning, though of a rather sad kind.

Conclusion

Experts and consultants have told us many times over that the journey towards a learning organisation is one worth making….and as the as the Bellman in Carroll’s poem says: “What I tell you three times is true.” Nevertheless, the reality is that instances in which learning actually occurs tend to be more a consequence of accident than plan, and tend to be transient than lasting. Finally, and perhaps most important, the Snark may turn out to Boojum:  people may end up learning truths that the organisation would rather remained hidden.   And therein lies the paradox of the  learning organisation.