A roadmap to agility

Many corporate IT shops use big design up-front methodologies to guide their internal software development projects.  Generally, IT decision makers seem reluctant to trial  iterative/incremental approaches, which have proven their worth in diverse development environments. The best known amongst these techniques are the ones based on agile development principles. “Agile principles are OK for software development houses,” say these  managers,  “but they’ll never work in the corporate world.”  I don’t quite agree with this because I’ve had some minor successes in using agile principles (continual customer collaboration, for instance) within  corporate IT environments. However – and I freely admit it – my efforts have been piecemeal and somewhat ad-hoc. Now, finally, help is at hand for those who have wondered how they might “add agility” to their development processes: A book entitled Becoming Agile…in an imperfect world, by Greg Smith and Ahmed Sidky, shows how non-agile development environments can be transformed through a gradual adoption of agile techniques. This post is an extensive review of the book.

I should add a caveat before proceeding any further: this review is written from the perspective of a development manager / team lead working in corporate IT – for no better reason than it’s what I do at present. That said, I hope there’s enough detail and commentary for it to be of interest to those working in other environments too.

The book begins with a story about a mining rescue, which provides an excellent illustration of agile principles in practice. The analogy is apt because, to be successful, any rescue effort must be collaborative (must involve many people with diverse skills), adaptive (must be responsive to changes in conditions) and, above all, must produce results (those trapped must be rescued unharmed). Traditional project management, with its insistence on complete, up-front requirements analysis and inflexibility to change would be hopelessly inappropriate for any rescue effort. Why? Because one cannot know a priori what might lead to a successful rescue – it is a complex process that unfolds and evolves with time. Similarly, as Fredrick Brooks emphasised more than 20 years ago, software development is intrinsically complex. What makes it so is the in-principle impossibility of obtaining and assimilating user requirements upfront. This is the essential difference between – say – a construction project and a software development effort. Recent research on project complexity suggests that agile techniques offer the best hope of dealing with this complexity. The essential advantage conferred by agile processes is the built-in adaptability to change via iterative development and continual customer involvement. In the end, this is what enables development teams to build applications that customers really want. An obvious corollary – if it needs to be stated at all – is that the adoption of agile techniques provides demonstrable business value. This is important if one wants to get management buy-in for a move to agility.

The book provides a roadmap for software development teams that want to improve their agility. Although the authors claim they do not favour a specific methodology, much of their discussion is based on Scrum. There’s nothing wrong with this per se, but I believe it is more important to focus on principles (or intent) behind the practices rather than the practices themselves. Folks working in corporate IT environments would have a better chance of introducing agility into their processes by adopting principles (or ways of working) gradually, rather than attempting to introduce a specific methodology wholesale – the latter approach being much too radical for the corporate world. The book also lists some common “roadblocks to agility” and a brief discussion of how these can be addressed. The authors emphasise that the aim should be to create a customised agile development process that is tailored to the needs of the organisation. Furthermore, instead of aiming for “agile perfection”, one should aim at reaching the right level of agility for one’s organisation. Excellent advice!

The path to agility, as laid out in the book, is as follows:

1. Assessment: evaluating current processes and developing a path to agility. Following Boehm and Turner, the authors suggest that upfront analysis be done to identify mismatches between organisational culture / practices and the agile techniques the organisation wishes to adopt. A proper assessment will help identify mismatches (or risks) associated with the transition. The book also provides a link to an online readiness assessment (registration required!). The assessments are to be provided in an appendix to the book. However, the review draft I received did not have this appendix, so I can’t comment on the utility of the tool.
2. Getting buy-in: Introducing an agile methodology is impossible without management support. One needs to make a case for this upfront. The authors note that the move to agility should be undertaken only if there are demonstrable benefits for the company. When canvassing support, the costs, benefits (for the company and management) and risks must be clearly articulated in a business case for the migration to agile practices. The book provides some examples of each.
3. Understanding current processes and modifying them appropriately: The authors emphasise that one needs to understand ones existing processes thoroughly before attempting to change them. Only when this is done can one determine which processes would benefit the most from change. The basic idea here is to make one’s processes as agile as possible, within organisational and other constraints. Transplanting another organisation’s processes into one’s environment is unlikely to work. The book outlines how organisations can develop customised processes suited to their specific environments. I found the book’s case-study based approach very helpful, as it provided a grounded example of how a company might approach the transition. In cases where companies have no pre-existing processes (or completely dysfunctional processes), the authors suggest starting with a packaged agile methodology such as Scrum.
4. Piloting the new process: The new processes have to be tested on a real project. The authors recommend doing a pilot project using the new methodology. Much of the book is dedicated to discussing a case study of a pilot project in a fictitious organisation. The discussion is useful because it highlights common issues that any organisation might face in using agile processes for the first time. The pilot project is a useful vehicle to illustrate how feasibility studies, estimation and planning, iterative development, release and delivery work in an agile environment. I really liked this approach as it provided a grounded context to the principles.
5. Retrospective: A retrospective or post-mortem offers the opportunity to improve the development process. Unfortunately, post-mortems are rarely done right. The book offers excellent advice on planning retrospectives. The basic idea: improve the process, don’t dissect the specific project.

Of course, achieving agility is more than modifying or adopting processes – it involves changing organisational culture as well. One of the main cultural obstacles is the command and control management style that is so prevalent in the corporate world. Another cultural issue is the lack of communication across organisational functions. The book provides advice on how to engender an agile culture within an organisation. Essentially, executives must endorse agile principles, line managers need to become coaches rather than supervisors, and teams need to adapt and adopt agile practices. Another characteristic of an agile culture is that teams are empowered to make their own decisions. This can be a challenge for managers and teams attuned to working in corporate IT environments that subscribe to the command and control approach.

The authors recommend engaging consultants to help with the transition to agility, but I think organisations may be better served by honest self evaluation first, followed by the development of an action plan. The action plan (in true agile fashion!) must be developed collaboratively, by involving all stakeholders who will be affected by the transformation. Books (such as the one being reviewed) and training courses can help one along the way, but there’s really no substitute for introspection and change from within. On a related note, the book mentions that agile teams should be composed of generalists – people with a broad range of technical skills. Corporate IT teams, on the other hand,  tend to made up of specialists. The authors point out that this can be a barrier to agility, but not one that is insurmountable.

Finally, the authors use the Technology Adoption Cycle to illustrate the difficulties of moving to an enterprise wide adoption of agile techniques. Given the huge culture change involved, they recommend an evolutionary transition to agile processes. In this connection, the authors identify five levels of agility: Collaborative, Evolutionary, Integrated, Adaptive and Encompassing, and recommend that enterprises progress through each of these steps on their way to agility nirvana. The book presents a chart outlining what each level of agility entails (see this article for more). This approach enables the organisation (and people involved) to “digest and assimilate” the changes in bite-sized pieces. The really good news is that the lower levels of agility are eminently achievable, as they emphasise agile principles such as customer collaboration and evolutionary (iterative) development, whilst placing no great demands on technical skills. This puts agility within reach of most organisations. So if you work in a non-agile environment, you may want to consider getting yourself a copy of the book as a first step towards becoming agile.

References:

Greg Smith and Ahmed Sidkey, Becoming Agile…in an imperfect world, Manning Publishers, Manning Early Access release, Sep 2007; Softbound print release, Feb 2009 (est).