Why Change Fails

When it’s all said and done, those of us helping organizations adopt Agile practices and values are really in the organizational change business. This is not to say that we develop no new ideas within the space of Agile software development and project management. As numerous books by various consultants in the field attest, we have a lot of good ideas, but the key value we offer is the application of those ideas to the challenging context of an actual organization. The best ideas within our industry can be at your fingertips for no more than $50 and a quick visit to Amazon.com. In spite of this, there is a massive gap between what we know works and how large organizations actually conduct themselves. Closing this gap is the key challenge, and value proposition, for a coach or consultant.

Most people see intrinsic value in learning new skills and capitalizing lessons learned from other organizations and practitioners. However, without a meaningful application of those ideas, we realize no benefit. Logically, we understand that adopting new practices is a good thing, and yet our own experiences shows just how difficult it can be. Fear, pressure for the status quo and pure inertia conspire to keep us and our organizations from realizing our maximum potential. So much so that the new organizational initiative is announced to great fanfare in the beginning of a year and consigned to the organizational dust bin by the start of the next one. So why do change initiatives fail? No two journeys are the same, and specific situations as well as organizational complexity make statistical analysis a futile task. Yet, my experience has shown that many sources of failure can fall into one of four categories.

Immune Response

The moment a new change is introduced within an organization, forces deep below the placid surface begin to move against it. This is not necessarily malicious, but simply the way the organization operates. Chances are people have a significant amount of experience doing things “the old way”, and they have set up their policies, practices and behaviors based on it. Some of these things are simple, like the requirements being documented with too much detail because someone drafted an enormous template that everyone uses. Some are more complex like an organization so used to splitting people across projects and geographies that the concept of a “team” is lost and people are encouraged to work from home as often as possible. I don’t know how many times I’ve heard people say that change requires “executive buy in”, but I fear that is a meaningless comment. Of course they must buy in eventually, but how do you get them to buy in? They also can’t be everywhere at once and a change initiative can appear to have support from the top but then suffer death by a million cuts within the bowels of an organization.

And it is within those bowels that we can see the immune response, where an entity is threatened by change and it seeks to remove the foreign agent. Perhaps a better analogy would really be more like an allergic response, where an over-active immune system creates more chaos responding to a something it perceives to be a threat. There are numerous small forces within any entity that will resist foreign elements, some will do so out of ignorance others out of malice. Regardless, the outcome will be the same if you do not confront them. As Machiavelli observed, no initiative is more fraught with risk than to undertaking change.

A change agent needs to understand these pressures, know where they are coming from and pursue strategies to counter them or provide countervailing pressure. My colleagues George and Giora have written numerous posts and presentations about mapping the change battlefield and how they will methodically analyze stakeholders to understand who is supporting and who is opposing a given change. This is just one example, and it is not always about organizational espionage. A good change agent understands that they must find those places which quietly push to revert back to the status quo and win them over so that they can become a positive part of the change.

Gap Between Where You Are and Where You’re Going

I must confess I have fallen into this trap several times as an Agile coach. The optimist in me wants to see every person as a fully empowered craftsman within their profession, yearning to be set free and realize their potential. This is not to say that people can’t reach that goal, but many people are not yet able to take that level of initiative in the short term.  Frequently, when teams are first confronted with the opportunity to self-organize, they will not succeed on their own. Like an animal raised in captivity, they keep waiting for someone to come and provide for them.

The Agile community has embraced the idea of servant leadership and team self organization. While I agree that should be our end goal, my experience is that most teams can’t make that leap immediately. Indeed, I have personally seen the disappointment of trying to coax a team to manage itself before it was ready. Paul Hersey and Ken Blanchard developed what is now known as Situational Leadership Theory, which posits that there is no right leadership style. Rather, it is situational in nature. Based on a team’s level of skill and autonomy, they may need more hands on direction, but as they mature, a coach needs to step back and let them practice on their own. Indeed, this means that change agents must be aware of the state of teams and adjust their behavior accordingly.

Much as it may be inappropriate to offer insufficient direction to a team early on, it would be equally inappropriate to continue to explicitly direct them months into a change initiative. As a coach or change agent, your goal is to help move individuals and teams to the point where they can adopt a new set of behaviors and carry them out on their own. Failing to know when to move into a supporting or delegating role can be equally destructive as not providing direction and coaching early on. Such a failure to adopt may allow a coach to be successful for a period of time, only for the team to eventually chafe and try to break free or possibly reject the new values and behaviors.

Inappropriate Level of Tension

Imagine that you are trying to move a small paper weight from one end of the desk to the other and your only instrument to move the object is a rubber band. This is a very appropriate metaphor for change. The band has no strength behind it, you can’t push with the rubber band any more than you can force people to adopt new behaviors. Rather you must create tension sufficient enough to draw the object towards the direction you want it to go. But having tension alone is not enough. Too little and the object won’t move. Too much and the rubber band will snap or the paper weight may be flung across the room. In fact, even if you find the right level of tension, as the object moves, the tension will change, forcing you to increase your pull or see the object move a small amount and then stop.

So how do you create tension? Peter Senge refers to what he calls creative tension as the pull of a desired or ideal state. An analysis or understanding of the problems is not enough, rather people need a vision of what they can achieve. This vision must be a realistic and desirable one, such that it will motivate people to push themselves towards that goal. There are many arguments about if people should be doing “Pure Scrum”, if they should adopt incremental or hybrid approaches, or even if the application of Agile is a conditional thing to different situations.

Let’s just look at Scrum for a minute, where a major challenge imposed by the framework is the delivery of production ready code no less frequently than every month (hopefully more like two weeks). This is a major challenge for many organizations, and rightly so. It’s meant to put forward a compelling vision, the ability for development teams to offer software in a dynamic nature such that they can play a strategic role in supporting the business. To do this will require them to change they way they operate. Here’s the key point: if the challenge is too great, people won’t do it or they will try, fail and eventually give up. The rubber band will snap. Thus, we may seek a more achievable goal. Perhaps the team will try to develop code that is not integrated, just developed and tested in a local environment. For some organizations this alone may be noble goal and one to which they can aspire to achieve. Of course, for some, it may be basically what they can do without too much effort. In this case, the modification effectively removes any pressure, and thus the team and organization do not change. This is manifest in places where you may hear things like, “we could NEVER do that here because of the [Insert acronym that refers to a standardized process that probably is no longer relevant, yet sacrosanct within the organization] standard”. This is the case where the organization basically talks itself out of the change, they effectively tailor it such that it is nothing more than a veneer for what they are already doing.

The other challenge with the situation of the more modest goal is knowing when to increase the tension. The team that can only hope to achieve a local development version within their 4 week sprint will eventually meet that goal. At that point, they need to try for more and not simply declare victory. Thus, our vision will need to continue to change in order maintain proper tension.

Static Interpretations

Be careful of your successes, for within them are planted the seeds of your destruction. I know this sounds dramatic, but it is absolutely true. We do not live in a static world, and yet much of the way we operate is designed to reinforce a single static impression of our environment. Follow me on a thought experiment for a moment. Imagine a company where 10 years ago a few bright people implemented a process for developing software that allowed them to successfully deliver some major products, thereby allowing the company to grow. These people were regaled as heroes within the organization. Their story is still told to this day about how they made the company what it is. Many of them are now the executives of the company, and their process has been immortalized as a standardized set of “best practices” that is applied to every project. I’m not saying those bright young executives didn’t have a good idea 10 years ago; in fact, they probably did. But of course, that was 10 years ago, and our thinking, not to mention our business domain, has changed a bit in the last decade.

In many cases, the careful analysis of a situation and creation of an appropriate response – the principles that got them to the process –  are forgotten and only the behaviors of the response are remembered. The behavior is not unlike the cargo cults that emerged in the Pacific Islands during World War II. As US and Japanese forces landed on islands and established operating bases, they would engender the good will of the locals by offering them food and provisions. The natives did not understand the nature of what was going on. To them, these were godlike beings delivering goods from the heavens. Years after the troops left, anthropologists returning to the islands found what they dubbed cargo cults, where natives would mimic the behaviors of the soldiers, building crude airstrips, going through the routines the soldiers did, in the hope that more supplies would be delivered to them. They understood the behaviors, but not the underlying principles. Likewise, we can frequently fail understand how the nature of the world around us is changing, and by simply focusing on the behaviors that worked in the past, fall victim to the unintended consequences they introduce.

This concept could apply to either the organization resisting change due to an adherence to behaviors that helped them in the past or to the change agent who may be trying to apply a process or value system that worked in another organization without fully understanding the values behind it. This second situation can be quite dangerous, especially as we look at the field of Agile Software development and see it being applied to more and more practices. While the values of delivering value rapidly, being responsive to changing dynamics and focusing on collaboration are quite applicable, the precise mechanics used on a software team may not translate perfectly to teams that are not building software, but doing other things entirely.

This last dynamic is probably the most insidious, because it also alludes to how a change effort, while quite appropriate at the time, may calcify and lead to a poor performing organization over the term. In the 1990’s, gasoline was cheap and the US auto industry made massive profits by building large SUVs. One decade later, it is very clear that cheap gas was not going to last forever, but the US automakers were not able to respond quickly enough and very nearly closed entirely due to their inability to adapt.

Here is the final, most important lesson about change: it is never done. Change initiatives are discrete points in time where we see where we are, say we want to go somewhere else, do it and declare victory. They are slow gradual processes, where our destination probably continues to change over time as the circumstances around us change. Based on this perspective, we see there is no single truth, quick fix, or even one “best practice”. Indeed, that is probably the biggest reason why changes fail. Sometimes, our initial vision is not correct, but we aren’t willing to “change the change” based on our evolving circumstances.

I’m sure that when I look at this post several years from now, I will shake my head at how naive my perspective was. But still, at this point in time, it’s served me pretty well to see where the Agile initiatives I am a part of may get in trouble and to hopefully intervene. 

Subscribe to Business Agility Trends and Insights
Business leaders need insight into how business agility and Agile are shaping the world. With a monthly subscription to Business Agility Trends and Insights, you'll receive curated trend reports, case studies, white papers and more that demonstrate how business agility is a core competency of the future. Topics include innovation, portfolio management, leadership, budgeting and finance and more!
*By entering your email address you give Accenture | SolutionsIQ permission to send you marketing emails. You may unsubscribe at any time by clicking the unsubscribe link located at the bottom of any email. View our Privacy Policy