If Agile and DevOps had Facebook pages, their relationship status would say: “It’s complicated”. There is clearly a strong relationship between these two, but I don’t think the industry has agreed on what it is. I know companies that consider DevOps to be part of Agile, I know companies where it’s the other way around. And then there are those companies where the two are treated completely separately.
My group of Agile and DevOps practitioners and coaches decided many years ago that we consider these things as two different sides of the same coin. Luckily, there are indications in the market that the two have merged without anyone really noticing. Go to an Agile conference and people will speak about Continuous Delivery just the same as they speak about stand-ups or retrospectives. Go to the DevOps Enterprise Summit and talks are full of SAFe, Sprints and cross-functional teams. Clearly there is something going on here. So let’s review the “complicated” relationship between Agile and DevOps a little bit more.
Agile came to the front stage a few years before DevOps gained popularity. Organizations adopted Agile because they wanted to be able to go to market more quickly and build better solutions. Ultimately, they wanted to reduce product batch size, as smaller batches can be deployed more quickly and are easier to manage.
Despite recognizing the motivation for these and other changes (e.g., stable teams), many people underestimated the importance of what was originally referred to as Agile technical practices or XP practices – test automation, automated deployments, etc. In other words, while there was some consensus about improving the software and product development process, there was little adoption of the process improvements, technically speaking.
To understand why, we can look at this from two perspectives: one more intuitive one and the other more scientific.
A Little Intuition
Large organizations who started to adopt Agile came from a pretty common starting point, which was the previous “state of the art.” They used to rely on testing centers of excellence who tested software solutions in a dedicated testing phase after development was complete. Most if not all of that testing was performed manually by specialized testers. With the adoption of Agile, the focus shifted to having potentially shippable increments ready at the end of each sprint, which had a ripple effect on the entire development process, including testing. To have a shippable increment at the end of an Agile development cycle, regardless of whether that was two weeks or four, the software needed to be tested at a more accelerated rate. This created a challenge. Regression testing, which used to be run at the end of a 3- or 6-month release cycle, now needed to be run every sprint. That was way beyond the present capability of the organizations adopting Agile, or at the very least it would have been very expensive. Clearly something was missing.
If you look at all the supporting activities in the software delivery lifecycle, there were many other things that were just not ready for an Agile approach to software development: the availability of environments, the speed and reliability of deployments, for example. All of these require a maturity level that just wasn’t required in waterfall development, yet the initial adoption of Agile was focused on the methodology, not the technological capability (not to mention architecture). So organizations learned from that experience and started to invest in Agile technical practices, which were later called DevOps practices, and things got better. One of my clients introduced a delivery “fast lane” for Agile projects, but because the technology capabilities were not in place, hardly anyone could use this fast lane and it was abandoned after a while. It was an adoption of Agile without DevOps that led to this suboptimal outcome.
It was an adoption of Agile without DevOps that led to this suboptimal outcome.
A Little Science
I have had many discussions with organizations about the relationship between Agile and DevOps and the learnings from the early adopters. Luckily organizations started to understand the relationship and the need to increase their technical capabilities if they wanted to be successful with Agile. But I was unsatisfied with this purely empirical approach; I was looking for something that made the relationship even more clear than case studies, previous experience, and an appeal to common sense. And then, when I sat in on a conference talk from Don Reinertsen, things fell into place. I rediscovered something I had learned many, many years ago in college : production management functions. And this gave me a more scientific answer to my challenge of justifying the investment in DevOps practices to make Agile adoption more successful.
Production management in a manufacturing context allows you to determine the optimal batch size if you produce more than one product, by considering the holding costs – how much it costs to store a product that is not yet sold – and the transaction costs – how much it costs to move from one product to another in your construction process. The optimal batch size is then the size of the batch where the cost per unit is the lowest. You can see this visualized in the diagram below. If you want to reduce the batch size, you aim to reduce the transaction costs, which in turn makes a smaller batch size optimal.
You might wonder how this relates to IT development. Let me explain.
We can equate holding cost in IT with the number of features that have been developed but are not deployed yet. The transaction cost is what it will take to go live (e.g. the cost of regression testing, deployment and any associated governance). DevOps practices makes regression testing faster and cheaper and deployments are automated, so transaction costs go down and hence a smaller batch size becomes optimal. But this also shows that if you adopt Agile and make the batch size smaller without investing in reducing the transaction costs, you will have a more expensive delivery mechanism. So here lies the more scientific explanation for why DevOps is key to Agile success: DevOps practices reduce the transaction costs in IT and Agile allows you to deliver smaller batch sizes. Only together are they optimal for your IT delivery in your organization.
DevOps practices reduce the transaction costs in IT and Agile allows you to deliver smaller batch sizes. Only together are they optimal for your IT delivery in your organization.
A Little Advice
Your transaction cost is a great indicator for your transformation, so measure it. As this goes down, you can deliver smaller and smaller batches of software.
There are two ways to measure transaction cost: Ideally you can measure it financially (but this is admittedly difficult). Otherwise, just measure the time from when the last functional line of code is written until the release actually goes live. How many days/hours elapse is a good approximation for transaction costs. This should include regression testing, defect fixes, deployment, release validation, governance steps and anything else required in your organization to go live. I have seen organizations who used release frequency as a measure of progress. Unfortunately, to achieve this they just introduced more parallel releases and each release carried the same transaction cost. That is not real progress; using the above transaction cost measure is a better indicator of progress for your transformation.
So as you can see, while their relationship may be “complicated,” Agile and DevOps cannot really live without each other.
- Six Myths of Product Development (HBR) by Stefan Thomke and Donald Reinertsen
This blog was authored by Mirco Hering. Based out of Melbourne, Australia, Mirco Hering is the author of “DevOps in the Modern Enterprise” and Managing Director of the Modern Engineering Practice in Australia at Accenture.