Eight to Late

Sensemaking and Analytics for Organizations

Redefining project success

with 16 comments

Introduction

When can one say that a project has  failed?  Although the answer to this question depends on the criteria used to measure project success, most project managers would not think the question (or its answer) could raise much controversy.  In this post I discuss why the issue of project success is more ambiguous (and contentious!) than seems at first sight.

The crux of the issue lies in the observation that managers and programmers often use different criteria to judge project outcomes. As Robert Glass states in his wonderful book on facts and fallacies of  software engineering:

There is a disconnect between management and their programmers. In one research study of a project that failed to meet its estimates and was seen by its management as a failure, the technical participants saw it as the most successful project they had ever worked on.

I’ll expand on this statement, drawing from Glass’ book, the research study he mentions, and personal experience.

The disconnect

Much of  Glass’ discussion is based on a paper by Kurt Linberg entitled, Software Developer Perceptions about Software Project Failures: A Case Study.  The  paper details a lot of information about the project, its context, reporting structures and the composition of the team.  Although this is important from the perspective of a research study, it is more relevant to look at how the various project stakeholders perceived the project.

First a few words about the project that Linberg studied. The project was aimed at creating a software-driven medical device. The management-approved schedule for the project ran over 14 months, but the project actually took 27 months to complete.  Further, the software development portion of the product was originally budgeted at a little over a million dollar but ended up costing more than four million! In view of this there’s little surprise that management deemed the project a failure.

What about the programmers? Linberg interviewed/surveyed many of the developers who were a part of the team. One of the questions he asked was: what was the most successful project that you have worked on, and why? Surprisingly, more than 50% of the team answered that the project described above was the most successful one they had ever worked on! The reasons they gave were revealing:

  1. The project was a technical challenge.
  2. The product worked as intended.
  3. The team was small and high performing.

The following quote from one of the team members says it all:

[the project] was the most successful because of what we accomplished. The code was more solid than any I have ever written. The verification testing was extensive. All the other places that I’ve worked relied on the software developer doing the testing. I always thought that I could have done more testing. I also really enjoyed the people I worked with. I also think that it was the best-managed project of any that I’ve experienced. Also, the product was well designed. I did not see the scope creep that is so prevalent on other projects. I never felt pressure from schedule. I would ask the project manager and technical lead if we were in trouble. They would be cool, calm and collected. They emphasized the importance of doing the best job we could do. We were able to focus our energy on the tasks at hand!

So, we see a complete disconnect: according to (majority of) the developers the project was success because it delivered a quality product, but by management metrics of budget and time it was a failure!

Bridging the gap

To bridge the disconnect between management and developers, Linberg asked the team members why the project was late and over-budget. The following reasons were offered:

  1. Unrealistic schedule and expectations.
  2. Poor understanding of scope (underestimating the effort required)
  3. Lack of resources

One of the team members summarised the schedule situation as follows:

Unfortunately, I believe program management established over-ambitious and incorrect expectations during early meetings with the executives. In reality, we had no idea how long it would take to develop this system. It is a real tribute to this team that they pulled together and developed an excellent product.

Another had this to say about the poor understanding of scope / lack of resources:

We ignored history. Although we didn’t have experience estimating this type of development, there are groups of people in our building that have been working these types of applications for years. They typically have 6 to 10 people working on their applications. How could management not see that assigning one or two people and constraining the schedule would be unrealistic?

Yet, the team thought that the project was managed quite well. Quoting from the paper:

Compared with other projects, the software developers believed the project management on the project was between average and best. In two cases, the software developers said it was the best that they had experienced.  One of the software developers that gave high marks for the project management stated that the project manager had good technical skills and could manage time lines.

In fact, one of the developers said:

The project manager, team lead, and program manager each provided leadership. The leadership worked very well together. The program manager knew his strengths and his boundaries. He left the software and firmware management to the software project manager. The software project manager was not threatened and left technical decisions to the team lead. We were always told of changes before they happened. The leadership was the best that I have experienced in my 14 years.

The overall impression that Linberg conveys is that the project was well-managed.   So where is the problem?

The programmers felt that upper management was largely responsible for the disconnect. This makes sense: front line managers rarely have the clout to set overall allocations of time and resources. They have to work within the parameters set by executives.

Based on this, Linberg proposes a new definition of software project success, which is based on industry averages rather than arbitrary metrics.

Completed Cancelled
Failure Does not meet customer expectations Not learning anything that can be applied to future projects
Low Success Below average cost, effort, and schedule performance compared to industry AND meeting quality expectations Some learning can be applied to future projects
Success Average cost, effort, and schedule performance compared to industry AND meeting quality expectations Some learning can be applied to future projects and some artifacts can be used on future projects.
High Success Better than average cost, effort, and schedule performance compared to industry AND meeting quality expectations Substantial learning can be applied to future projects and a large number of artifacts used.
Exceptional Success Meeting all quality, cost, effort and schedule expectations. A Cancelled project cannot be considered an exceptional success.

This definition begins to bridge the chasm between developer and senior management perceptions of project success. It does so by broadening the definition of success. The current project management-driven definition is too narrow because it takes only one perspective into account – the perspective of those who hold the purse-strings.  Such a definition cannot work because it completely ignores the viewpoint of those who know software development best: the developers.

Conclusion

Linberg’s case study highlights the fact that there can be a huge gap between how developers and managers view project outcomes. Interestingly, the factors that cause the gap are usually present from the start of the project: unrealistic schedules, poorly understood scope and lack of adequate resources. One could hypothesise that most of the projects that “fail” do so for one or more of these reasons. Also interesting is the fact that even such “failed” projects can be deemed successful in the eyes of developers because they see things from a different perspective: what was learnt and what can be used again. Seen in this light, a project that goes over-time and over-budget can be successful because the experience gained can be used on future projects.

I’ll end with a story from experience. Some years ago I was a part of a team developing a product for a telecom software company. The product was ambitious in scope, and the timeline and budget tight. Nevertheless,  the team looked forward to the challenge of designing and building a product from scratch. The architect, project manager  and several members of the development team (excluding me) had years of experience in building software for the telecom industry. The product specs were developed by domain experts.  Management’s initial intentions were good: they assured the dev team that they (management) knew that it would take time to develop a solid product from the ground up. To reduce risk it was decided to build the product using incremental approach with monthly deliverables.

Progress was slow but steady.  Unfortunately, as the product started taking shape, business reality intruded: the sales guys who had been working hard  had not been able to convince anyone to sign up.  Funds began to dry up and management responded by cracking the whip.  Predictably, the project started to unravel and was canned a few months later.  Officially the project was deemed a failure, yet many  on the dev team believed that  a) it was one of the best projects they had worked on b) they had learnt a lot and  c) what they had learnt wouldn’t go waste.

Was the project a failure? I suppose the answer is that it failed from a purely  commercial standpoint. From a broader perspective, however, it fits quite neatly into Linberg’s definition of a successful cancelled project.

Perhaps it is time to rethink the definition of project success.

Written by K

April 16, 2010 at 5:50 am

16 Responses

Subscribe to comments with RSS.

  1. A case of the operation was a success but the patient died.

    Like

    Nick

    April 16, 2010 at 1:11 pm

    • Nick,

      Nice analogy, but I would put it a bit differently: the patient wasn’t cured but the doctor learnt how to do it better next time.

      Regards,

      Kailash.

      Like

      K

      April 16, 2010 at 9:26 pm

      • Actually in this case he was sued for malpractice and not delivering what the procedure promised, lost his job and reputation, went bankrupt, was forced out of town and committed suicide..but on his gravestone he still claimed the operation was a success…on another note..in my book, if I pay for a project and it is not delivered as to my needs then it failed, I don’t worry about defining a special project success criteria for the team that produced a motionless car, a dead patient or a piece of broken, over budget, late delivered piece of software that I can’t use.

        Like

        Nick

        April 21, 2010 at 1:43 pm

        • Nick,

          Perhaps that’s stretching the analogy a bit but it is a good point, and one that is absolutely justified from the customer’s perspective. My point is that there’s a disconnect (even contradiction) between different stakeholder views on many (most?) projects. A broader definition of success – such as the one Linberg proposes – is necessary in order to reconcile these.

          Regards,

          Kailash.

          Like

          K

          April 21, 2010 at 6:20 pm

          • Only if you feel project success needs to be unecessarily and infuriatingly bullied by postmodernist nonsense as everything else seems to be today. Many things, (despite the world shattering views of your cheese-&-wine Tuesday evening-partying intelectual set with strange French accents) are still clear-cut, black and white, obvious even. The man with the money calls the tune, pulls the plug and determines the success of project or not. If he’s not happy with the result he’ll take his business elsewhere. I’m sure at the beginning of a busy project the customer would be dying to set up a sort of HR-style luv-in to discuss what project success means from everybodies individual perspective so that they can take on all views in some happy town cookoo world where nothing is deemed a failure because nobody should have any hurt feelings. Certainly you can tick the boxes in Linberg pointless classification but the failure or success of a project itself is usually pretty obvious to all involved (ie the customer says you screwed it up).

            Like

            Nick

            April 22, 2010 at 10:12 am

            • Nick,

              Sure, as you say, “The man with the money calls the tune, pulls the plug and determines the success of project or not. If he’s not happy with the result he’ll take his business elsewhere….”

              Nevertheless, the fact that many failed projects are doomed from the start because of unrealistic schedules, poorly understood scope and lack of adequate resources suggests that something isn’t quite right. Linberg’s definition (whether one agrees with it or not), draws attention to the need for all stakeholders to achieve a shared understanding of the project objectives and a commitment to achieving them. IMO, this isn’t optional, it is absolutely essential. Yes, this means getting everyone’s views on how things should be done, but does not need to be a long-winded “HR-style” discussion. Such a shared understanding would remove the disconnect and render Linberg’s definition irrelevant. As long as the disconnect remains, however, I think Linberg makes a very good point.

              Regards,

              K.

              Like

              K

              April 22, 2010 at 6:39 pm

              • I would have thought any well run project has an understanding “of the project objectives and a commitment to achieving them”. I also suggest projects that are doomed to fail due to lack of resources, poor scope etc etc need a more experienced project manager at the helm rather than a redefinition of project success to hide the failure.

                Like

                Nick

                May 19, 2010 at 2:13 pm

                • Nick,

                  Linberg’s definition does not attempt to hide failure: it states very clearly that a completed project that does not meet customer expectations is a failure.

                  Further, it is undeniable that many projects fail due to unrealistic schedules, poorly understood scope and lack of resources. On such projects there is indeed a lack of a shared understanding of project objectives and a commitment to achieving them. I guess you would dub these as projects that aren’t well run, but I’m not sure that one can attribute the failure of these projects to the inexperience of those who managed them.

                  Regards,

                  K.

                  Like

                  K

                  May 19, 2010 at 6:28 pm

  2. A few years ago on a project I asked the question “How will you define success for this project?” to different people I was working with.

    The Business analyst: The requirements are met
    The tester: No defects
    The operations manager (who would ‘own’ the system after the project; We are doing business smarter
    The project manager; Schedule and budget targets are met
    The programme manager: We are asked back for another job.

    Like

    Craig

    April 16, 2010 at 9:54 pm

  3. Craig,

    That’s a great example! The disconnect isn’t just between managers and programmers…

    Regards,

    Kailash.

    Like

    K

    April 16, 2010 at 10:02 pm

  4. What about end users? Aren’t they stakeholders as well? There’s no mention of user satisfaction – though you might actually say,’the program has no defects’, but it might not be user friendly, for example, as compared with another.

    Possibly, repeat customers are the best measure of a successful project – they’ve used your systems and seen your costing and service and are still there. What say you?

    Like

    Narendra

    April 16, 2010 at 11:31 pm

    • Narendra,

      Absolutely: end user satisfaction is indeed a measure of success, and an important one. In the post, however, I’m focusing on the other end of the spectrum, where a project might be cancelled and yet be considered a success (in a sense). My point is that the definition of project success could (should!) be expanded to encompass such situations.

      Regards,

      Kailash.

      Like

      K

      April 17, 2010 at 8:11 am

  5. Hi all – It’s remarkable to me that people are finding value in research that I completed nearly 10 years ago. Thanks for the great discussion. Sounds like there are opportunities to do more research in this area and see if organizations are doing a better job with their softare project success! — Kurt

    Like

    Kurt Linberg

    May 18, 2010 at 3:23 am

    • Kurt,

      I’m glad you enjoyed the discussion. Thank you for a very thought-provoking paper.

      Regards,

      Kailash.

      Like

      K

      May 18, 2010 at 7:36 am

  6. I would have thought an experioenced PM would see such early signs of failure, – poor scope, lack of resources and unrealistic schedules, after all the have to earn their big bucks for siome reason other than provuiding project lunches.

    Like

    Nick

    May 20, 2010 at 9:54 am

  7. […] written a number of articles on project failure, covering topics ranging from  definitions of success to the role of biases in project failure.  As interesting as these issues are, they are somewhat […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: