Eight to Late

Sensemaking and Analytics for Organizations

On the intrinsic complexity of IT projects

with 3 comments

Several years ago Fredrick Brooks wrote his well known article, No Silver Bullet , in which he explained why software development is intrinsically hard. I believe many of the issues that make software development inherently difficult have close parallels in IT project management – parallels which apply even in projects that don’t involve much writing of code. In this post I look at Brooks’ article from the perspective of an IT project manager, with a view to gaining some insight into why managing IT projects is hard.

Brooks defines the essence of a software entity as, “…a construct of interlocking concepts: data sets, relationships among data items, algorithms and invocations of functions…” This construct is abstract; that is, it has many different representations or implementations. A line later he states, “…I believe the hard part of building software to be specification, design and testing of the construct, not the labor of representing it and testing the fidelity of the representation…” The connection with project management (especially, IT project management) is immediately apparent: the hard part in many projects is figuring out what needs to be done, not actually doing it. Put another way, requirements, rather than implementation, are the key to successful projects. Project management methodologies deal with implementation reasonably well, but have little to say on how requirements should be elicited. Why is this so? To answer this it is helpful to take a closer look at the parallels between inherent properties of software entities (as elucidated by Brooks) and IT projects.

According to Brooks, the essence of software systems (as defined above) is irreducible – i.e. it cannot be simplified further. This irreducible essence, he claims, has the following inherent properties: complexity, conformity, changeability and invisibility. These characteristics are present in any non-trivial software entity. Furthermore – and this is the crux of the matter- any advances in software engineering or development methodologies cannot, in principle, solve difficulties that arise from these inherent properties. I discuss these properties and some of their consequences below, pointing out the very close connections with IT project management.

  • Complexity: Brooks describes software entities as complex in that no two parts are alike. This complexity increases exponentially with entity size. Deriving from this complexity, he says, are issues of difficulty of communication among team members leading to product flaws, cost overruns and schedule delays. This should sound extremely familiar to IT project managers.In an earlier post I looked into definitions of project complexity. In a nutshell, there are two dimensions of project complexity: technical and management (or business) complexity. Brooks defines complexity in technical terms, as he is concerned with building software. However, in large part the complexity he talks about arises from business complexity (or complex user requirements). The latter is often what makes IT projects difficult, even when there’s not much actual code cutting involved. Furthermore, this characteristic is intrinsic to all but the most trivial IT projects.
  • Conformity: software entities must conform to constraints of the environment into which they are introduced. To paraphrase Brooks, “…These constraints are often arbitrary, forced without rhyme or reason by the many human institutions and systems to which interfaces must conform…” This too has obvious parallels with IT project management – the deliverables of any IT project have to fit into the environment they are intended for. This fit has to be at the technical level (eg: interfaces) but also at the business level (eg: processes). I’m sure many IT project managers would agree that the technical and business constraints imposed on them are often arbitrary (but compliance is always mandatory!). So we see that this characteristic, too, is intrinsic to most business and technical environments in which projects are conceived and implemented.
  • Changeability: Brooks describes the software entity as being “…constantly subject to pressures for change…” This, he reckons, is partly because software embodies function (i.e. it does something useful) and partly because it is perceived as being easy to change (italics mine). One would think twice (or many more times!) before asking for large-scale changes to a building that has already been erected, but there’s considerably less restraint shown when asking for major changes to software. This, again, has parallels with IT project management – shifting requirements are the norm, despite the high cost in terms of time, if not money. Change control processes are put in place to dampen this tendency, but it is ubiquitous nonetheless.
  • Invisibility: According to Brooks, software entities are, “… invisible and unvisualizable…,” in all but the simplest cases. Unlike the floor plan of a building, which helps project personnel visualise the finished product, no pictorial model is available to software builders. Sure, modelling techniques do exist, but they do not (and cannot!) depict the complexity of a non-trivial software system in any meaningful way. This, says Brooks, “…not only impedes the process of design within one’s mind, it severely hinders communication among minds…” Here too, the parallels with IT project management are clear – communicating the requirements of the project would be so much simpler if there were a visual representation of what’s required. If Brooks is right, though, a search for such a representation is akin to the search for the philosopher’s stone.

Yet Brooks is not a pessimist. Towards the end of his article, he mentions some techniques that can alleviate some of the essential difficulties of building software. These include: rapid prototyping and iterative / incremental approaches. Grow software, he says, instead of building it. Such an approach, which incorporates frequent interactions between users and developers, reduces risk associated with missed or misunderstood requirements and clarifies design in small, digestible steps. This is also good advice for IT project managers, as I’ve pointed out in a previous post.

In the last section of his article, Brooks states, “The central question of how to improve the software centers, as it always has, on people” and then goes on to discuss how talented designers (or architects) can greatly reduce the essential difficulties in building software. I believe the parallel here is between designers and project managers. A talented project manager can make all the difference between the success and failure of an IT project. What are the attributes of a talented project manager? Well, that’s a topic for another post, but I think most of us who’ve worked in the field can recognise a good project manager when we see one.

Brooks believes the essential difficulties associated with building software make silver bullet solutions impossible, in principle. The parallels outlined above lead me to believe that the same applies to project management. Methodologies may help us along the road to project success (and some do so more than others! ), but there are no silver bullets : managing IT projects is intrinsically hard.

Written by K

May 4, 2008 at 6:30 pm

3 Responses

Subscribe to comments with RSS.

  1. […] It is well established that software projects are notoriously hard to estimate (see my article on complexity of IT projects for more on why this is so). The authors studied how project managers handle incorrect estimates. […]

    Like

  2. Very nice analysis. I was doing some research regarding complexity in software development and like you said there are technical and management aspects of complexity in software development (like in many other fields). For the technical part of it indeed requirements (business value for customer) are backbone of the system. In fact the software is developed because of those requirements.

    However there is a gap between two groups of people involved in this process: One group talks about business and another group talks about implementation. And it is very difficult to find a person who could understand business and have a very good idea of IT part of it. IT industry itself is in blame for that. Every time when we hire person as a developer we are asking only technical questions by assuming that person will get all the requirements ready to use and developer is not interested in business questions either. She or he is getting paid for coding/some testing/delivering. So, why to loose value by thinking about business.

    The gap between developers and business people (domain experts) generates information entropy which in turn creates broken connections between requirements and implementation. Also we have to mention difference between Real Requirements (RR) and Collected Requirements (CR). CR/RR ratio is very important and should never be underestimated.

    Now complexity is about how much we understand about business that we are trying to conceptualize and then provide interface to it. As long as we will ignore existing gap and will continue hiring “coders” and “business analysts” this complexity will stay with us.

    Like

    Farhad

    June 18, 2008 at 5:53 am

  3. Farhad,

    Thanks for your comments.

    I agree – it is difficult to find someone who has both technical and business skills. However, I believe developers can pick up business skills, providing they have the motivation and management support to do so. I elaborate below.

    In a corporate environment, development staff should be encouraged to work with the business on a day to day basis, even when there are no projects involved. One way to do this is to “attach” developers to business units, so that they can serve as technical advisors to the business. In the bargain, these folks pick up substantial business knowledge. Then, when the time comes to develop a new application, IT has the requisite technical and domain knowledge in hand. This knowledge can help reduce the gap between real and collected requirements significantly.

    Some caveats. This works well only if a) the developers have motivation to learn about the business and b) the business (including IT management!) understands that IT is more than just technology. Of course, good communication and related skills are also necessary . If a business is serious about going down this route, they have to ensure that potential technical hires have so-called soft skills, or the aptitude (and attitude!) to develop them.

    Regards,

    Kailash.

    Like

    K

    June 18, 2008 at 9:51 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: