Wireless Carrier Increases Productivity by 50 Percent

I wanted to share an experience that I had recently training and coaching several teams in XP practices that ultimately helped them significantly improve their deployment process.

Here’s some background:

The customer was an international telecommunications wireless carrier based in the US. They were deploying a release every quarter when they brought me in. Their development staff consisted of six teams each with one ScrumMaster, two testers, and about six developers.

Pre-process

The entire dev staff was the butt of quality jokes, because their implementations of new features into an old and legacy code base was buggy and inefficient. Development was stopped three to four weeks prior to the quarterly release to stabilize the build and do “final QA”. Deployment consisted of three days of round-the-clock coverage by two members from each team working in eight-hour shifts. The initial eight hours of deployment was an extreme, high-stress blame fest with finger pointing and an inferno requiring significant resources to shore up the deployment and “just get things fixed enough” to deploy the release. For the next four to six weeks, one person from each team maintained a ten-hour triage call every day. During this time, all teams worked mostly on Sev1 and Sev2 defects, 360-400 defects all told.

Post-Process

After a year of intense coaching and process implementation, did-some-tests-found-no-bugsdeployment was as follows: Only two members from the entire group were required for the eight-hour deployment. During the deployment, other folks were watching movies, reading books, sleeping and helping other groups fix their defects. One team member from the group was on-call for the entire weekend, but never got called. The last deployment I took part in had a total of 16 Sev2/3 defects.

In addition, the “final QA” phase was completely done away with, and work was brought in until two or three days before deployment. The four to six weeks of triage and firefighting was non-existent.

What the Customer Got

The customer effectively gained about one to one and a half months of extra development time because their developers were no longer fighting fires; instead they were completing new stories/work. Towards the end of the engagement, the Business Analysts were starting to have problems coming up with high/medium priority work for the teams to work on. The entire company began to take notice of the group and started asking how they managed such a turn around.

What it Took

  • Full Agile teams from the Product Owners through to BAs, testers and of course developers
  • Full XP teams implementing pair programming, TDD, and CI and committed to spending 20% of their development time to refactoring/improving existing code
  • The infrastructure needed to support Agile development (e.g., a fully deployed CI environment, a build engineer, really beefy development boxes, etc.)
  • One embedded process coach/developer per team to coach the team in XP practices
  • Senior Management trusting and allowing embedded coaches to devise and implement process changes for the teams and development environments
  • Active support from senior management
    • Initially senior management attended and coordinated meetings supporting the Agile Coaches. On at least three different occasions, senior managers were brought in to say things like:
      • “We expect every line of code to be written paired (i.e., by two devs pair programming).”
      • “Yes, we fully support writing tests BEFORE writing code.”
      • “We know that learning these processes will slow us down initially, but they will help us go faster in three months.”
  • The willingness to change and implement that change when the need arose

The Bottom Line

Before adopting XP practices, the dev group had one release per quarter, which included more than three weeks of code freeze and another three or more weeks of bug fixing. After XP adoption, there was no code freeze and minimal code fixing post-release. Thus the group spent at least six fewer weeks on fixing a release, which works out to a 50% increase in productivity.