Redefining project success

Introduction

When can one say that a project has  failed?  Although the answer to this question depends on the criteria used to measure project success, most project managers would not think the question (or its answer) could raise much controversy.  In this post I discuss why the issue of project success is more ambiguous (and contentious!) than seems at first sight.

The crux of the issue lies in the observation that managers and programmers often use different criteria to judge project outcomes. As Robert Glass states in his wonderful book on facts and fallacies of  software engineering:

There is a disconnect between management and their programmers. In one research study of a project that failed to meet its estimates and was seen by its management as a failure, the technical participants saw it as the most successful project they had ever worked on.

I’ll expand on this statement, drawing from Glass’ book, the research study he mentions, and personal experience.

The disconnect

Much of  Glass’ discussion is based on a paper by Kurt Linberg entitled, Software Developer Perceptions about Software Project Failures: A Case Study.  The  paper details a lot of information about the project, its context, reporting structures and the composition of the team.  Although this is important from the perspective of a research study, it is more relevant to look at how the various project stakeholders perceived the project.

First a few words about the project that Linberg studied. The project was aimed at creating a software-driven medical device. The management-approved schedule for the project ran over 14 months, but the project actually took 27 months to complete.  Further, the software development portion of the product was originally budgeted at a little over a million dollar but ended up costing more than four million! In view of this there’s little surprise that management deemed the project a failure.

What about the programmers? Linberg interviewed/surveyed many of the developers who were a part of the team. One of the questions he asked was: what was the most successful project that you have worked on, and why? Surprisingly, more than 50% of the team answered that the project described above was the most successful one they had ever worked on! The reasons they gave were revealing:

1. The project was a technical challenge.
2. The product worked as intended.
3. The team was small and high performing.

The following quote from one of the team members says it all:

[the project] was the most successful because of what we accomplished. The code was more solid than any I have ever written. The verification testing was extensive. All the other places that I’ve worked relied on the software developer doing the testing. I always thought that I could have done more testing. I also really enjoyed the people I worked with. I also think that it was the best-managed project of any that I’ve experienced. Also, the product was well designed. I did not see the scope creep that is so prevalent on other projects. I never felt pressure from schedule. I would ask the project manager and technical lead if we were in trouble. They would be cool, calm and collected. They emphasized the importance of doing the best job we could do. We were able to focus our energy on the tasks at hand!

So, we see a complete disconnect: according to (majority of) the developers the project was success because it delivered a quality product, but by management metrics of budget and time it was a failure!

Bridging the gap

To bridge the disconnect between management and developers, Linberg asked the team members why the project was late and over-budget. The following reasons were offered:

1. Unrealistic schedule and expectations.
2. Poor understanding of scope (underestimating the effort required)
3. Lack of resources

One of the team members summarised the schedule situation as follows:

Unfortunately, I believe program management established over-ambitious and incorrect expectations during early meetings with the executives. In reality, we had no idea how long it would take to develop this system. It is a real tribute to this team that they pulled together and developed an excellent product.

Another had this to say about the poor understanding of scope / lack of resources:

We ignored history. Although we didn’t have experience estimating this type of development, there are groups of people in our building that have been working these types of applications for years. They typically have 6 to 10 people working on their applications. How could management not see that assigning one or two people and constraining the schedule would be unrealistic?

Yet, the team thought that the project was managed quite well. Quoting from the paper:

Compared with other projects, the software developers believed the project management on the project was between average and best. In two cases, the software developers said it was the best that they had experienced.  One of the software developers that gave high marks for the project management stated that the project manager had good technical skills and could manage time lines.

In fact, one of the developers said:

The project manager, team lead, and program manager each provided leadership. The leadership worked very well together. The program manager knew his strengths and his boundaries. He left the software and firmware management to the software project manager. The software project manager was not threatened and left technical decisions to the team lead. We were always told of changes before they happened. The leadership was the best that I have experienced in my 14 years.

The overall impression that Linberg conveys is that the project was well-managed.   So where is the problem?

The programmers felt that upper management was largely responsible for the disconnect. This makes sense: front line managers rarely have the clout to set overall allocations of time and resources. They have to work within the parameters set by executives.

Based on this, Linberg proposes a new definition of software project success, which is based on industry averages rather than arbitrary metrics.

	Completed	Cancelled
Failure	Does not meet customer expectations	Not learning anything that can be applied to future projects
Low Success	Below average cost, effort, and schedule performance compared to industry AND meeting quality expectations	Some learning can be applied to future projects
Success	Average cost, effort, and schedule performance compared to industry AND meeting quality expectations	Some learning can be applied to future projects and some artifacts can be used on future projects.
High Success	Better than average cost, effort, and schedule performance compared to industry AND meeting quality expectations	Substantial learning can be applied to future projects and a large number of artifacts used.
Exceptional Success	Meeting all quality, cost, effort and schedule expectations.	A Cancelled project cannot be considered an exceptional success.

This definition begins to bridge the chasm between developer and senior management perceptions of project success. It does so by broadening the definition of success. The current project management-driven definition is too narrow because it takes only one perspective into account – the perspective of those who hold the purse-strings.  Such a definition cannot work because it completely ignores the viewpoint of those who know software development best: the developers.

Conclusion

Linberg’s case study highlights the fact that there can be a huge gap between how developers and managers view project outcomes. Interestingly, the factors that cause the gap are usually present from the start of the project: unrealistic schedules, poorly understood scope and lack of adequate resources. One could hypothesise that most of the projects that “fail” do so for one or more of these reasons. Also interesting is the fact that even such “failed” projects can be deemed successful in the eyes of developers because they see things from a different perspective: what was learnt and what can be used again. Seen in this light, a project that goes over-time and over-budget can be successful because the experience gained can be used on future projects.

I’ll end with a story from experience. Some years ago I was a part of a team developing a product for a telecom software company. The product was ambitious in scope, and the timeline and budget tight. Nevertheless,  the team looked forward to the challenge of designing and building a product from scratch. The architect, project manager  and several members of the development team (excluding me) had years of experience in building software for the telecom industry. The product specs were developed by domain experts.  Management’s initial intentions were good: they assured the dev team that they (management) knew that it would take time to develop a solid product from the ground up. To reduce risk it was decided to build the product using incremental approach with monthly deliverables.

Progress was slow but steady.  Unfortunately, as the product started taking shape, business reality intruded: the sales guys who had been working hard  had not been able to convince anyone to sign up.  Funds began to dry up and management responded by cracking the whip.  Predictably, the project started to unravel and was canned a few months later.  Officially the project was deemed a failure, yet many  on the dev team believed that  a) it was one of the best projects they had worked on b) they had learnt a lot and  c) what they had learnt wouldn’t go waste.

Was the project a failure? I suppose the answer is that it failed from a purely  commercial standpoint. From a broader perspective, however, it fits quite neatly into Linberg’s definition of a successful cancelled project.

Perhaps it is time to rethink the definition of project success.

Relational factors in managing outsourced projects

Introduction

More often than not the success of an offshored IT initiative is measured using financial or operational metrics –  client-side managers  tend to focus on cost savings  and vendor-side managers on achieving service level metrics.  This is fine as it goes, but  is a limited view of the business arrangement. It is far more important for both parties to to focus on building trust and commitment to a long-term relationship. In a paper entitled, Evaluating the Success in International Sourcing of Information Technology Projects: The Need for a Relational Client-Vendor Approach, Peter Haried and K. Ramamurthy look into the role of  relational factors in  internationally sourced projects.  This post is a summary and review of the paper.

Project management methodologies and standards focus on the economic, contractual and operational aspects of  projects; they have very little to say about how client-vendor relationships should be managed.  Perhaps as a consequence, much of the research on offshored projects is based on agency theory or transaction cost economics, both of which focus on risk  and opportunistic behaviour from perspective of  clients (rather than vendors).  However, it is clear that vendors would generally have a different view of what constitutes risk or opportunism – a risk for a client is not necessarily one for the vendor. Further, as Haried and Ramamurthy mention in their introduction, success is in the eye of the beholder;  clients and vendors will differ on what constitutes success.  This suggest that a view which emphasizes relationships rather than contracts and SLAs  may provide a basis for a more inclusive approach to offshored initiatives.

Recent research has begun to look at the relational aspect of offshoring (see this paper or this one, for example).  According to Haried and Ramamurthy, however, little work has been done on integrating the relationship perspective with the traditional outcome-focused view.  Their work is an attempt to bridge this gap.

Theoretical Background

In a post entitled, To outsource or not to outsource, I discussed how transaction cost theory can provide organisations an insight into whether or not they should outsource work.  According to transaction cost theory, the cost of completing a transaction (from the client perspective) depends critically on:

1. The complexity of the transaction: for example, implementing an ERP system is more complex than implementing a new contract management application.
2. Whether or not it involves assets that are worth more within a relationship between two parties than outside of it: for example, custom IT services, tailored to the requirements of a specific company have more value to the two parties – provider and client – than to anyone else. This is called asset specificity in economic theory.

These factors can help determine whether or not an organisation should outsource their work, but they neglect the social-relational aspects of the outsourcing relationship. For instance, if the relationship is strong, a firm may feel confident enough to offshore highly complex, organisation-specific work.  Clearly, the relational aspect needs to be factored into in any offshoring decision.

The relational view is founded on the assumption that the relationship between the client and vendor is more important for success than a purely economic view. Haried and Ramamurthy use social exchange theory as the basis for their investigation. Quoting from Wikipedia, “Social exchange theory posits that all human relationships are formed by the use of a subjective cost-benefit analysis and the comparison of alternatives.” As such, it assumes that human relationships are based on an economic analysis, albeit a subjective one.

Research Model

Based on a survey of research literature, Haried and Ramamurthy identify the following as key variables that affect offshoring relationships:

1. Information Sharing: this defines the degree to which the parties agree to share relevant information with each other. This is really just another term for effective communication, one of  the key  success factors of any project.
2. Legal bonds: these are the legally binding agreements that are set out in the offshoring contract. The authors contend that many problems in offshoring deals can be traced back to the contract between the client and vendor. It is important for both parties to realise that the contracts are necessarily incomplete, and must be interpreted in a far-sighted manner.
3. Client adaptations: adaptations made by the client to fit the processes and procedures of the vendor.
4. Vendor adaptations:  adaptations made by the vendor to fit the processes and procedures of the client.
5. Mutual obligations: beliefs that each party hold about their obligations to each other. These generally refer to items that have not been (or could not be) contractualised. Mutual obligations are a form of psychological contract.
6. Intercultural competence: The ability to develop good interpersonal and working relationships with people from other cultures, and to resolve any conflicts or misunderstandings that may arise due to cultural differences.  Traditionally, intercultural differences have been viewed as a major obstacle in international sourcing arrangements. However, research seems to be somewhat equivocal on this point.

In my opinion, the selection of these factors is arbitrary: the authors do not offer any justification for choosing these six factors rather than others. For example, anecdotal evidence suggests that offshoring clients often complain about  the high turnover of vendor staff -the rationale being that it becomes difficult to build good working relationships when key contacts on the vendor-side change often. Given this, continuity (of relahionships) could have been considered a separate variable in much the same way as information sharing has been factored out of mutual obligations (information sharing is a mutual obligation).  Although this is just an example, it begs the question as to which other important relational variables might have been omitted from the study.

Haried and Ramamurthy identify the following factors as being measures of the  relational success of  an offshoring arrangement. In their model  these factors are assumed to be caused (or driven) by the above mentioned variables.

1. Trust:  In most cases the client and vendor are located in different countries, so there’s plenty of scope for (negative) opportunistic behaviour on both sides. Given this, it is important that the  two parties trust each other. As such, the degree of trust developed is a good measure of the relational success of a project.
2. Commitment: This refers to the effort that each side is willing to put in to maintaining the relationship. Very often this entails going beyond the contract. The ideal situation as far as commitment is concerned is when both parties have an exclusive arrangement to work with each other. Clearly, the degree of commitment is another measure of relational success. However, it isn’t clear to me that it is an independent factor because commitment depends on trust.
3. Conflict: All relationships are prone to conflict, outsourcing arrangements are no exception. However, working through and resolving conflicts in a mutually acceptable manner can strengthen the relationship. Conflict can be quantified (sort of) via the overall level of disagreement between the two parties over matters such as goals, procedures, timelines etc. Clearly, the level of conflict is also a good measure of the relational success of the offshoring arrangement.

Again, these three success factors are chosen without justification,  omitting other possibly significant ones. For example, longevity of the relationship might also be a good indicator of the success of the relationship. Further, the causal connection  between variables and success factors  is far from clear; the authors hypothesise a cause-effect relationship  between the six relational dimensions and the three measures, but there is no justification offered.

Research Methodology

The authors gathered qualitative data through interviewing a number of offshoring client/vendor pairs about the relational aspects of the outsourcing arrangement. Vendors were contacted through the client – this is an important point which I’ll return to later. The clients were asked to provide information on two projects – one that had been completed recently and the other ongoing. The authors asked for permission to interview multiple stakeholders from both the client and vendor organisations. This provided diverse perspectives on the all questions that were asked. The questions asked pertained to the model discussed in the previous section, in the context of the project that the interviewees had worked on. The questions were open-ended so the interviewees had to think through their answers, not just tick a box.

The authors surveyed five  US-headquartered firms from the areas of financial service and manufacturing. These were the clients.  The outsourcing partners (vendors) – six in all – are all India-based outsourcing majors. The gathered data covered eight projects across the surveyed organisations.

Analysis

Below are the conclusions the authors drew from an analysis of their data. The findings are listed by each of the model variables.

Information exchange

Client Perspective

Most of the client-side interviewees indicated that communication (information exchange) was a key determinant of the success of the relationship.  In many projects the vendor had a on-site  team with whom client-side stakeholders could communicate directly. This was highlighted by clients as being an effective means to communicate. It placed the onus of communicating with the offshore team on the on-site vendor team.

Vendor Perspective

The vendor perspective matched that of the client. All vendors agreed that having an onsite team is critical to the success of the project.  Although some of the reasons offered by vendor-side stakeholders differed in detail from that of  those on the client-side,  this indicates that communication is indeed a key relational success factor.

Based on the above, the authors formulate the following proposition:

Proposition 1: Intense, open, and timely information exchange is crucial for relational success for both client and vendor stakeholders.

Vendor Adaptations

Client Perspective

Clients viewed adaptations made by the vendor as critical to the success of the outsourcing relationship.  Typically, the adaptations cited by clients included, working hours, processes and procedures, investments made in training staff and placing personnel on site.

Vendor Perspective

Most vendor-side interviewees agreed that the vendor needed to adapt to the client rather than the other way round. Further, many vendors recognized that this was a key to continuing the relationship. As one vendor-side  analyst put it,  “Up to a certain level, no matter what, we will accommodate because they know if we do not accommodate we will be out.”

The authors point out that these findings are consistent with transaction cost theory in that vendor adaptations are, in effect, a type of asset specificity.  They give the vendor an advantage over other vendors who may want to compete for the client’s business because they indicate that the vendor is serious about maintaining and enhancing the relationship with the client.

Based on the above, the authors hypothesise the following:

Proposition 2. Adaptations by vendors are vital for relational sourcing success evaluations by client and vendor stakeholders. These are even more important for client stakeholders than for vendor stakeholders.

I’m not sure I agree with the statement, “[vendor adaptations are] even more important for client stakeholders than for vendor stakeholders”. I’d say they’re at least equally important to vendors because the client could walk away from the relationship if the vendor is not willing to make adaptations.

Client Adaptations

Client Perspective

The responses here were a bit mixed. Clients-side managers believed that some adaptations would have to be made, and were worthy of making too. For example, one manager stated that offshoring would entail more rigorous documentation and adherence to internal processes– which he perceived as a Good Thing. On the other hand, one of the business analyst’s interviewed thought it odd that they (the client) should be making adaptations when they were the customer in the relationship. The perception seemed to be that the vendor should make adaptations, not the client.

Vendor Perspective

Some vendor-side interviewees indicated that clients would need to make adaptations to get the most out of offshoring.  Two areas singled out were: 1) adjusting to distributed teams and 2) adapting to cultural differences.  That said, vendors valued whatever adaptations their clients made, and realised that these adaptations indicated the seriousness of the client about the relationship.

Based on their findings, the authors propose the following regarding client-side adaptations:

Proposition 3. Adaptations by clients are essential for relational sourcing success evaluations by client and vendor stakeholders. These are even more important for vendor stakeholders than for client stakeholders.

Legal Bonds

Client Perspective

Most client stakeholders considered the offshoring contract to be of limited importance. In most cases, clients viewed the contract as a basis for the relationship, but not a comprehensive document containing last detail of the business agreement.  Client-side managers emphasized that the contract was intended to be flexible.

Vendor Perspective

The vendor, quite naturally, placed more emphasis on the contract. That said, most vendor managers noted that the contract described only the high-level goals, not how they would be achieved. Most vendors saw the contract as a means to engender trust and reduce conflict.  Based on interviewee responses, the authors make the following hypothesis:

Proposition 4. Legal bonds are considered key to relational sourcing success for vendor stakeholders to a greater degree than for client stakeholders.

Although the authors claim that these findings are consistent with social exchange theory, I suspect that the respondents may have glossed over difficulties –  see my post on some  pitfalls of contracts for more on why.

Mutual Obligations

Client Perspective

Many clients did not acknowledge that mutual obligations played a role in the relationship. However, some client-side stakeholders acknowledged that many vendor personnel went above and beyond the call of duty (and the contract). On the other hand, some senior managers felt they weren’t getting value for the large sums money they were spending.

Vendor Perspective

Vendor stakeholders were consistent in their belief that the contract could not capture everything, and that mutual obligations were an important determinant of the success of the relationship. Many vendor-side personnel claimed that they were doing several tasks that were not in the contract.

The authors suggest that the differences in perceptions of mutual obligations may be because many client-side personnel were simply not aware of what was written in the contract. In contrast, most vendor stakeholders had a good idea of what was contracted. Based on this, the authors make the following hypothesis:

Proposition  5. Mutual obligations are viewed by vendor stakeholders as essential for relational sourcing success, but not necessarily by client stakeholders.

Intercultural Competence

Client Perspective

Clients acknowledged the importance of intercultural understanding to the success of the relationship. However, most clients displayed a good awareness of cultural differences and what needed to be done to address issues arising from them.

Vendor Perspective

Like clients, all vendors acknowledged the importance of cultural differences. Like clients, they did not believe these presented any major problems.

The authors emphasise that their findings are not consistent with previous research (see this paper for example).  Based on their findings they propose the following:

Proposition 6. Intercultural competence is not a key determinant of relational sourcing success valuations by client and vendor stakeholders.

Summary results and conclusions

Based on their findings, the authors make the following points:

1. Success of the relationship depends on many factors (in particular, those listed above), and any analysis must include both client and vendor perspectives. Many earlier studies included only the client perspective.
2. Information exchange (communication) is acknowledged as a key factor by both clients and vendors.
3. The clients expected vendors to make adaptations to the clients’ processes, but did not necessarily believe that they (the clients) needed to make any adaptations. The authors point out that this finding is inconsistent with transaction cost theory
4. Clients did not always acknowledge the importance of contract, but the vendor always did.  The same was true of the role of mutual obligations. The authors speculate that this may be because client stakeholders weren’t always aware of what was contracted whereas vendor stakeholders invariably were.
5. Neither clients or vendors believed that cultural differences were an unmanageable issue. The authors suggest that this may be because offshoring is now a well-accepted practice and that client firms have a better understanding of the potential problems that intercultural issues can cause.
6. They acknowledge the limitations of their research: sample size, location of clients and vendors.  However, they do not acknowledge that clients and (especially!) vendors may not have been open in their responses. This is a particular concern because vendors were contacted through the client.
7. They believe that their work may be useful to project managers who work on internationally sourced projects because it provides a set of key relationship dimensions that managers should focus on.  Maybe so, but readers should be aware that there could be other key  dimensions as I’ve noted earlier in the review.

To conclude: although the conceptual model presented by the authors has some shortcomings, project managers will find the paper a worthwhile read because it highlights the importance of relational factors in managing outsourced projects.