Eight to Late

Symptoms, not causes: a systems perspective on project failure

with 18 comments

Introduction

A quick search reveals that the topic of project failure gets a fair bit of attention on the Internet. Many of the articles identify factors such as lack of executive support or incomplete/misunderstood requirements as the prime causes of failure. In this post I use concepts from systems theory  to argue that commonly identified “causes” of project failure – such as the ones noted above – are symptoms rather than causes.  I then  surface the real causes of project failure by taking a systems perspective –  a viewpoint that considers the project and the hosting organisation as a whole.

As I will argue, project failures can  often be traced back to dysfunctional structures and processes within the organisation.  These factors usually have little to do with the project directly and are therefore not always obvious at first sight.

Setting the scene

Readers who have waded through the vast literature on failed projects will have noted that there are diverse opinions on the prime reasons for failure. It would take me too long to wade through these articles, so I’ll just pick a source that is well known even  if not entirely credible: The Chaos Report by the Standish Group.  As per this summary  the Chaos Report 2009 claimed the following were the top three causes of project failure:

  1. Lack of user input.
  2. Incomplete or changing requirements/specifications.
  3. Lack of executive support.

Although the report lists the top ten factors, in the interests of space I’ll focus on just these three.

I should mention that the report refers to the above as  “Project Challenged Factors.” Grammar issues aside, this is a somewhat strange way of putting it. Anyway, I interpret this phrase to mean reasons (or driving factors) for failure.

Systems theory and projects

First up, what exactly is a system?

Here is a jargon-free definition from the wonderful book by Donella Meadows entitled, Thinking in Systems: A Primer. Incidentally, I highly recommend the book as an easy-to-read and engaging introduction to systems theory:

A system is a set of things interconnected in such a way that they produce their own pattern of behaviour over time. The system may be buffeted, constricted , triggered or driven by outside forces. But its response to these is characteristic of itself and is seldom simple in the real world.  (italics mine)

The word interconnected is  important because it tells us that when we study something from a systems perspective, we must identify all the important connections it has with its environment.  In the case of projects, the important connections are fairly obvious. A project is usually carried out within an organisational setting so it would have many connections to the hosting organisation. Chief amongst these is the fact that  a project is staffed and resourced by the hosting organisation (or by an organisation designated by it). Another important connection is that  a project will affect  different stakeholder groups within the hosting organisation in different ways.  However,  since all stakeholders have ongoing organisational roles  that go beyond the project, the project is not their only interest. This is a key point to which we will return later.

The phrases “own pattern of behaviour over time” and “characteristic of itself”  tell us that systems have a unique pattern of behaviour that is characteristic to them.  The important point to note is that characteristic behaviour implies that different systems  that have the same signature – i.e. identifying features – will tend to behave in the same way. From the perspective of projects  this tells us that projects within similar organisations will evolve in similar ways.

A systems view of the causes of project failure

With only this very brief look at systems theory we are now in a position to get some insights into  the real causes of project failure. As mentioned earlier, we will focus on the top three reasons ala Standish.

Lack of user input

First up, consider lack of user input. Systems  theory tells us that we need to look at the issue from the perspective of the project and the organisation. From this point of view it is clear that users who have been asked to work on the project in addition to their normal duties will view the project as a burden rather than something that may benefit them in the future.

This by itself is not a new insight. In fact, project management gurus have talked themselves hoarse about the need to free up resources etc. However, the point is that organisational structures typically work against this.  Firstly, people who are asked to work on a project know that it is an initiative that will end in a finite (usually, reasonably short) time. Therefore, their long terms interests lies in their ongoing roles rather than  in projects. Secondly, most organisations are  still structured along functional lines and people’s identities are anchored within their functional reporting lines rather than ephemeral project hierarchies.

Changing requirements

The issue of changing requirements and specifications can also be understood from a systems point of view.  A characteristic of many systems is that they are stable – i.e. they resist change.  Organisations are typically stable systems – they tend to retain their identity despite changes in their environment.  One of the characteristics of such organisations is that people in them tend to think and act in set ways – that is, their actions and thinking processes follow well worn patterns to the point where they do not need to think about what they do too deeply.

One of the consequences of this is that when they are asked for requirements users often provide incomplete description of what they do, leaving out significant items that are obvious to them (but not to the analysts who are gathering requirements!).  Although I don’t have the figures to back this – I speculate that a fair proportion of  changes in requirements  are the result of  inadequate detail or thought put into developing  initial requirements. The point is users and sponsors don’t necessarily  see these as changes, but project teams do.

Lack of executive support

Finally, let’s look at the the problem of lack of executive support.  Project sponsors usually hold important executive-level positions within the hosting organisation. By virtue of their positions,  they have a number of important things that compete for their attention. A project – even an important one – is only one of many things going on in an organisation at any one time.  Moreover,  organisational priorities do shift, perhaps more often than executives may want to admit.  So a project that was the key focus yesterday may be superseded by other priorities today.

There are of course many other ways in which project sponsors can be distracted, but I think I’ve made my point which is that lack of executive support is due to features that are inherent in organisations. So no amount of forcing executives to pay attention to their projects is going to work unless the entire system  (project + organisation) changes.  And this is difficult if not impossible to achieve because stable systems such as organisations tend to resist change, and therefore continue to display their characteristic patterns of behaviour.

Discussion

So we see that the causes of project failure can be traced back to the organisations in which they are embedded.  Specifically, they lie in unwritten norms and formal policies that dictate how the hierarchy operates and how things are done within the organisation. The most important consequence of this is that standard fixes (of encouraging user input and executive support, or instituting change management, say)   will not cure the problem of project failure because they do not address the dysfunctional norms and policies that are the root cause of failure.

The above is not news. In fact, the matrix organisation structure was proposed as a response to the need for “project friendly” organisations. I’m no expert on matrix organisations so I will leave it to others to comment on how successful they are. The only point I would make is that  in my experience multiple reporting lines (even if dotted and solid) do not work too well. There are always conflicting interests that cause divided loyalties and conflicting interests.

So,  the natural question is : what – if anything – can we do about this? The answer is implicit in the foregoing paragraphs. One has to align the project with the organisation, not just at the level of objectives or structure, but also in operational matters such as timelines, budget and resources –  the very things that make up the so-called  iron triangle.  The best time to do this is at the front-end of the project,   i.e. the start. At this time, the person(s) driving the project have to engage all stakeholders who will be affected by the initiative and find out their motivations, interests and – most importantly – their concerns regarding the project.  If this discussion happens in an open and frank manner,  it should surface the issues highlighted In the previous section.  Since the discussion takes place even before the project starts, there is at least some hope of addressing these concerns.

There are many ways to structure and facilitate such discussions. Check out this post for an introduction to one and have a look at my book co-authored with Paul Culmsee for much more. That said, one doesn’t need any particular technique – the willingness to discuss difficult matters openly and an openness to other points of view is all that’s needed. That, however, is not always easy to come by…

Conclusion

We have seen that the top causes of project failure can be traced back to the hierarchies and incentive systems of the hosting organisation. Therefore, superficial attempts to fix the problem at the level of individual projects (or even a PMO) will not work. The only hope of addressing the root causes of project failure is to focus on the systemic dysfunctions that cause them.

Written by K

June 18, 2013 at 9:18 pm

18 Responses

Subscribe to comments with RSS.

  1. Yet another well-done post, Kailash. I salute your consistency and skill in adding value to the field of organizational development (or whatever you’d like to call it).

    There’s a project challenge factor that has been bugging me for years. (This may be a subtle topic shift, we’ll see.) I am reminded of it by the “lack of user input” factor. One could call it the “Law of the Excluded Middle Manager”. It basically says: any project aimed at improving organizational efficiency and information flow will be undermined or even sabotaged by middle managers if they feel it reduces their power or relevance in the organization. The irony is that this issue holds even (or especially!) when middle managers are *not* the users of the proposed system. They are excluded because they are already too busy to attend the design meetings, and they’re not the primary users of the proposed system, but it nonetheless will have a significant impact on them because it streamlines them out of some process.

    I could write a book about the number of times and ways that this has happened in efforts to role out networked Compendium in policy and strategy organizations, because of the strong “democratization of knowledge” force that is intrinsic to Compendium’s design center. And I’m loosing patience with the traditional approach for dealing with the Excluded Middle Manager factor: letting the project fail, and waiting for middle management to retire or be promoted before trying it again.

    So I turn to my learned friend down under … I assume that there is some literature on this issue, but I don’t know about it. And what I’m really looking for is a theory that gives me some leverage and skill when a project I’m working on is headed straight for the rocky shallows of excluding middle managers.

    jeffconklin

    June 19, 2013 at 2:30 am

    • Hi Jeff,

      I’m honoured by your kind words – thank you for taking the time to read and comment.

      I loved the phrase “The law of the excluded middle (manager)” – that’s brilliant :-).

      Following your comment, I did a literature search, and there is indeed some research on the topic. Here’s one paper I found:

      http://onlinelibrary.wiley.com/doi/10.1002/smj.4250070403/abstract

      I managed to get a copy of the paper. Here is a line from the conclusion:

      Middle management self-interest motivates the degree of commitment to strategy implementation, so divergence between middle management self-interest and organizational interest as perceived by general [senior] management is likely to result in ineffective strategy implementation, unless it is anticipated and managed carefully by general [senior] management.,

      which I think is essentially in line with your observation that a project will be undermined or even sabotaged by middle managers if they feel it reduces their power or relevance in the organization.

      However, the authors note that other factors can play a role as well. In particular, middle managers might be unenthusiastic about a project because:

      1. They do not think they can execute the strategy successfully. One might call this the “mission impossible syndrome.”

      2. They believe that even if they execute it successfully, it will not achieve the desired outcome for the organisation.

      The authors do offer some high-level suggestions on addressing the problem but all of these will be well known you (e.g. understand the politics, provide forums for discussion, focus on higher order goals, change approach etc.). Easy enough to name, but hard to implement.

      I will let you know if I find anything else of interest, but if the problem stumps you Jeff, I don’t think there’s any mortal who can solve it.

      Regards,

      Kailash.

      K

      June 19, 2013 at 9:34 pm

  2. Kailash, sharing Jeff’s sentiments in congratulating you for yet another well written post.

    The topic of project failures is an intriguing one. While many are quick to point out the reasons (or symptoms – as you point out above), not many are coherent enough in articulating a solution. Given your assessment, that the core discussions are predominantly around the symptoms and not the causes, it is not surprising that no credible solutions have emerged from these discussions.

    While your analysis is logically sound in tracing project failures back to the organizations in which they are embedded, I feel further work is still required to identify a robust and workable solution.

    Shim Marom

    June 19, 2013 at 8:51 am

    • Hi Shim,

      Thanks so much for reading and for your appreciative words.

      I agree entirely: my “solution”, such as it is, is the best I could come up with, but is far from satisfactory. For one, not all organisations will welcome such dialogue; in many cases discussions about such matters would simply be deemed out of bounds.

      Do let me know if you come across any promising approaches. Perhaps we we can also discuss this when we chat next.

      Thanks again mate.

      Regards,

      Kailash.

      K

      June 19, 2013 at 9:36 pm

      • Kailash, one way to resolve this issue is implementing a strict system of reward and punishment, making people accountable and responsible to the point that should it succeed they will be rewarded for its success and should it fail they will be adversely financially impacted.

        Sounds harsh but I can’t see any other way around this.

        What do you think?

        Shim Marom

        June 20, 2013 at 8:07 am

        • Shim, I can see the logic of that, but have concerns that extrinsic rewards (such as financial incentives) can have the effect of “crowding out” or reducing intrinsic motivation – i.e. what people will do without external inducement (or coercion!). There’s some nice research on this by Bruno Frey and his colleagues (see this paper (PDF) for example).

          Regards,

          Kailash.

          K

          June 20, 2013 at 8:53 am

          • Kailash, these deal with cases of reward only (i.e no negative incentive is offered). Are you aware of any research that examines the scenario I’ve proposed; i.e. you get a reward in you succeed and you get punished if you fail?

            Shim Marom

            June 20, 2013 at 9:41 am

  3. Kailash

    Why do you think it is that so much more attention is given by commentators to the reasons for failure than to the reasons for project success? Why this attempt at reverse engineering? It really is heavy going and the reasoning is usually an analytical cul-de-sac.

    As I think is implicit in your analysis, evidence of causes of failure do not (and are I suggest are unlikely to) show-up the root causes. And as you suggest I think, a better place to find professional advancement is to seek the reasons for success. My experience tells me that you are wise to recognise these as inspired from an understanding of the human and organisational behaviour observed in successful project regimes (organisations).

    Aspiring and talented young actors do not spend a lot of energy learning from failed actors: they look for excellent performance from the best and to their own insights and imagination. Some classic errors can of course explain failure (as in initiating a project without a plan), but in general the same logic of ‘creation’ applies to any pursuit of practice improvement. The most useful principles and techniques to be learned are most likely to be contemporaneous and local.

    This is all explored in my book, “The Single Minded Project”. Manuscript now with the publisher!

    Martin

    Martin Price

    June 24, 2013 at 7:39 am

  4. […] Kailash Awati looks at the commonly identified causes of project failures, and finds that they are symptoms of organization dysfunction. […]

  5. […] with the well rehearsed list of the contributing factors for project failure, share with them Kailash’s recent article where he challenges the common wisdom and suggests an alternative (and a  better explanation) to […]

  6. Dear Kailash

    I agree that a wider, an organizational perspective is needed, when talking about project management practice. There are a number of forces driving organisational effort. Some time ago I have read the article about strategic alignment by Bruce Campbell. He used Peter Senge’s 5th discipline approach to explain the effect of oscillation of a perceived IT performance. Shortly if IT department improves and delivers more keeping same resources, a business demands more. But because of improvements there is no spare resources, so IT starts to deliver “less”. And so on.

    The other issue is that a project managers (in my opinion) behave selfishly and ask for more resources for longer time than it is required (this is againts critical chain methodology). If some project managers ask for more then it is needed the whole project portfolio starts to be executed suboptimally. In game theory there is concept of a price of anarchy, that can be applicated here. Selfish behavior of a project manager may bring delays to an other projects. In case of a high priority projects if the project manager asks the resource owner for an additional resources he gets them, but the other projects pay.

    Finally, a project portfolio execution depends on a behavior of many players. The question is how to design such a motivation system that will tie a personal goals to support successfun execution.

    Jack

    September 10, 2013 at 7:16 pm

    • Hi Jack,

      Thanks so much for your interesting comment. Campbell’s paper sounds very interesting. I tried to find a copy but I don’t think I got the right one. It would be great if you could send me a link (if it is in the public domain).

      IMO the best analysis (and critique) of strategic alignment is the one by Claudio Ciborra in this paper which I have reviewed/summarised in this post. Ciborra’s problem with the concept is that it is based on a simplistic cause-effect based logic that does not take into account the complex interactions that occur in an organisation. Indeed, this is a wider problem that afflicts most management models that are used in organisations all over the world. Here are a few pertinent lines from his paper:

      [Management Science] deploys careful empirical research, claiming to identify “naturally occurring phenomena” but in reality measures theoretical (and artificial) constructs so that the messiness of everyday reality gets virtually hidden. Or it builds models that should be basic but do not last a few years and quickly fall into oblivion.

      And a few lines later:

      …practitioners and academics increasingly worship simplified models that have a very short lifecycle….managers who have been exposed to such illusionary models, presented as the outcome of quasi-scientific studies, are left alone and disarmed in front of the intricacies of real business processes and behaviors, which in the meantime have become even more complicated than when these managers left for their courses. People’s existence, carefully left out of the models, waits for them at their workplaces.

      He contends that the notion of strategic alignment is based on such impoverished models. True alignment, he contends, is more an emergent process that occurs through improvisation and tinkering rather than grand “strategic” planning. Improvisation and tinkering are necessary because events somehow always seem to surprise us, to “escape our plans.” Hence, as he says:

      …we are confronted with a choice. Either we can do what management science suggests, that is “to realize these surprises in implementation as exceptions, build an ideal world of “how things should be” and to try to operate so that the messy world that in which managers operate moves towards this model….or we suspend belief about what we think we know…and reflect on what we observe. Sticking to the latter we encounter phenomena that deeply enrich our notion of alignment…

      Your point about selfish behaviour is an important one. From a systemic perspective, however, that too is an organisational issue than a personal or project-related one. Such behaviour could be reduced by making resource allocation in the organisation a less adversarial process. Indeed, as Engwall and Jerbrant have argued in this paper (summarised here), resource allocation issues have their origins in flawed organisational procedures rather than poor PM practices.

      Finally, the point you make about portfolio execution is a very important one. Again this is a systemic issue because dysfunctional group behaviours often occur because the environment does not encourage genuine collaboration. One needs to put in place the conditions for genuine teamwork. Paul Culmsee offers some interesting insights on this (drawn from recent research on teamwork) in his recent series on SharePoint maturity models.

      My apologies for this long, rambling reply – I hope it makes sense. Thanks again for taking the time to read and comment.

      Regards,

      Kailash.

      K

      September 12, 2013 at 3:13 pm

  7. […] or even “incomplete requirements.” What these experts do not understand is that these are  symptoms rather than causes.  The true causes of failure invariably lie in the hosting organisation, not the project. For […]

  8. […] Feel like digging deep? Kailash Awati, the author of Eight to Late, regularly provides scholarly insights into project management issues. Read First: A Systems Perspective on Project Failure […]

  9. […] perhaps his most important because of its remarkable implications. The story has, I believe, been told at least once before but, like all such narratives, its effect is much less striking when set forth en bloc in a single […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 274 other followers

%d bloggers like this: