Principal DevOps manager at Microsoft Donovan Brown has been around the world training teams on how to get the greatest value out of Agile and Scrum. He offers much advice on this episode, advocating for a top-down transformation approach; establishing a shared language about Agile across the organization to improve communication and, thus, success; and never accepting an estimate longer than four hours. But it’s his definition of DevOps that has interests us most. To wit: “DevOps is the union of people, processes and product to enable continuous delivery of value to our end users.” And for business leaders who are itching to implement an entire DevOps pipeline all at once, he says, “Don’t… [Instead] focus on what hurts most first.”
Accenture | SolutionsIQ’s Stas Zvinyatskovsky hosts at the Deliver: Agile Conference in Nashville, Tennessee.
Read the full transcript
STAS ZVINYATSKOVSKY: Welcome to another edition of Agile Amped. I am your host, Stas Zvinyatskovsky, and we’re podcasting today from Deliver:Agile conference in Nashville, Tennessee. Today, my guest is Donovan Brown. Donovan is a principal DevOps manager at Microsoft. He has traveled the globe helping companies in the U.S., Canada, India, Germany and the UK develop solutions using Agile practices, as well as Visual Studio and Team Foundation Server. And Donovan has worked in the industries that’s brought us communications, healthcare, energy, financial services.
DONOVAN BROWN: Yep.
STAS ZVINYATSKOVSKY: Little known fact, but Donovan was a professional air hockey player.
DONOVAN BROWN: Yep.
STAS ZVINYATSKOVSKY: And he was ranked pretty high. Number …
DONOVAN BROWN: Yeah, number 11 in the world at one point.
STAS ZVINYATSKOVSKY: Wow, that’s impressive.
DONOVAN BROWN: Yeah, it’s intense. When you see two professionals play air hockey, it’s not what you see at Chuck E. Cheese. These are … I like to describe it as two grown men taking this game way too seriously.
STAS ZVINYATSKOVSKY: Well, I’d like to play you one day.
DONOVAN BROWN: Oh, it’d be fun. I’d love to.
STAS ZVINYATSKOVSKY: Although, I’d be intimidated. But let’s get into it.
DONOVAN BROWN: Sure.
STAS ZVINYATSKOVSKY: And let’s maybe start … So there’s a lot of things that we can talk about, because he is specialized in Agile and DevOps, many, many different things.
DONOVAN BROWN: Right.
STAS ZVINYATSKOVSKY: But let’s start with Agile.
DONOVAN BROWN: Okay.
STAS ZVINYATSKOVSKY: And Agile transformations. What have you seen?
DONOVAN BROWN: What I’ve seen over … So before I joined Microsoft, I was a process consultant for about seven years. So it was my job to fly all over the world and install and implement team foundation server, and because I was a certified Scrum Master, I would stay on staff and help the companies come from the waterfall world into the Agile world. So I’ve seen a lot of things done well and I’ve seen a lot of things done very poorly. One of the things that I saw people failing with was they would not educate everyone that interacted with your team.
They would only educate the team, and so the team is running around using these new words like sprints and backlogs and burn downs, but the rest of the organization is still using waterfall terminology like change request and burned … back … no, using things like change requests and milestones and Gantt charts, right? So they’re having this complete disconnect in the way that they communicate, and when your executives still want milestones and Gantt charts, they want milestones and Gantt charts. I don’t care if you’re using these new fancy terms, where’s my Gantt chart? Where is my milestone? And you’re trying to explain to them that it doesn’t work that way. And it’s natural for you to resist or fear anything that you don’t understand. And because you didn’t take the time to educate the executives, they’re going to demand that you do things in a way that they’re familiar with.
So, the best advice I can give people who are new for it and what I would do as a Scrum Master, is I would go in and say, I need to talk to everyone who’s going to interact with my team. Like everyone, no matter how high or how low they are on the totem, bring them into this room for me for the next 90 minutes and I’m going to explain to them what Scrum is going to be until I say otherwise, and that way we’re all using the exact same vocabulary. We know what relics from the past we’re no longer going to be producing anymore. How do you read a burn down, what meetings we’re going to have and why that we’re going to have those meetings, how long those meetings are going to be. I would explain this is Scrum, and then we would go off and we’d be able to be far more successful, because people aren’t fearing it any longer, because they now are all part of the discussion versus feeling like they’re on the outside looking in.
So, don’t send one person off to get Scrum Master trained and then bring them back and expect them to be successful. Make sure that everyone gets educated. And then I would highly recommend bringing in a seasoned Scrum Master who’s done this before and can help you avoid those pitfalls that everyone makes, because they’ve already been there and seen that. So, those are just some of the things that I’ve noticed on the transformation is educate everyone and for at least a short period of time hiring a Scrum Master who’s already done it.
STAS ZVINYATSKOVSKY: Great. And in your experience, how receptive have those audiences been?
DONOVAN BROWN: Well, it all depends on the industry, right? So, if you have an industry like automotive who’s been very successful with waterfall, because building an automobile is very waterfallish. Like you can be very successful building a new version of your car with waterfall. It’s very hard to convince that organization that Agile is the future, because they’ve never seen it. They’ve been very successful without it. The larger the organization and the more successful the organization, I’ve also had a lot of trouble there as well, because if it’s not broken, why are we gonna fix it? Our process has made us number one in our industry and this is how we do it. We’re not going to change it. So, I wish there was a, oh yeah, this industry always works this way, and this industry always works that way. It’s on a case by case basis, unfortunately, but I notice young small teams are very susceptible to it, because it’s all new to them, right? They don’t have any bad habits. They don’t even know what they don’t know yet. So, teaching them Scrum right from the beginning is very, very easy and they take to it very, very quickly. It’s the longer, more established organizations that sometimes don’t want to change or resist change.
STAS ZVINYATSKOVSKY: Interesting. And so in organizations like that, what have you found? Have you found that top down works better than grassroots?
DONOVAN BROWN: Oh, top down, in my opinion always works better, because when you have an older organization, some of those engineers have been there for decades and they like the rhythm in the world that they’ve carved out for themselves and without executive buy in, you have your peers trying to promote Agile, they’re just like, no. And they have no motivation to do so because their bosses don’t care either, but when you have executive buy in, all that resistance can be completely squashed by the fact that no, our CEO said so and you’re going to do this because our boss’s boss’s boss’s boss said we’re going to do this, and we’re all going to start doing this.
So, top down is always the easiest. It’s not the only way to be successful, but it is the easiest way to be successful. If you don’t have executive buy in or you’re going to have to go off and kind of do this grassroots, I’d highly recommend that you pick a greenfield, preferably, application, a very small application in a new team. We usually call those pilot projects, and you need that pilot project to be extremely successful. You know what I mean? Do not fail, because if you have this team shining so brightly, others are going to take notice and say, how do we get other teams to perform like this? How was this small little team able to produce this amazing software in such a short period of time where hundreds of these other engineers barely show any input and … Or any progress.
So, that’s an opportunity to say, you know what? If we do this right, everyone’s going to take notice and everybody’s going to want to do this. Because they’re going to be wondering, why is their team outperforming every other team? And they’re going to ask you, how are you doing that. And then it’ll spread almost throughout your organization that way as well. But don’t bite off more than you can chew. Don’t go and take the most important project, the longest project first, because when that fails, it’s going to fail on a big giant stage and then no one’s going to believe in it after that.
STAS ZVINYATSKOVSKY: Great advice.
DONOVAN BROWN: Oh, cool.
STAS ZVINYATSKOVSKY: So you mentioned to me that there’s one advice that you always give to Agile teams. What is that?
DONOVAN BROWN: It’s about estimating, and I think estimates are extremely important. And what too many people do is that they’re bad at it and they don’t change. They just acknowledge that we’re bad at it. Like, yeah, estimates, we suck at estimates, so we’re just going to keep being bad at estimates and there’s ways that you can actually get a lot better at estimates and I think the best advice I can give any team is stop accepting estimates that are a week long. For example, developer comes in, sees a product backlog item, says I think I can get that done in a week and then all of a sudden we start sprinting. The problem with a week long estimate is many factors. First of all, the daily standup is the exact same for the entire week. They haven’t gotten anything done, they’re not going to get anything done and there’s nothing stopping them. They can say that for an entire week. Not until Monday of the next week, when I ask them, what are you working on? And they say, “Task number one,” do I realize we have a problem? And that’s too late to do anything about it.
If you break it down into four hour tasks or less, what I’ve realized is is that they quickly get to that seventh and eighth finger and they realize, yeah, this isn’t a week long task. This is eight days worth of work, and that’s okay, but I need to know that it’s eight days worth of work before we commit to doing it so that I don’t get two five day tasks that are actually really two eight day tasks and we’re going to blow our sprint. So, if I know that it’s a eight day task, I can grab another two day or three day task to tack on to get the spare space out of there. Another thing that you get out of having smaller estimates versus just the fact that they are more accurate is that you can now parallelize the development. If I have a one week long task that requires database changes, unit test to be written, a middle tier to be written modifications to the UI, and I own that entire task, I Donovan Brown as an individual are now responsible for doing all of that in serial. None of it will get done in parallel.
But if I break it down into all the individual tasks, maybe someone can be working on the database schema while I’m working on the user interface, or maybe someone’s working on the unit task while I’m working on some other part and now we’re able to parallelize that work and we’ll be done in a lot shorter time than the aggregate of all that work serially, which is really important. Another benefit you get by breaking it down from within a week to several smaller task is I can see if we agree in our definition of done. Your DoD is extremely important and you want to make sure that when people are estimating that week long … Does that week long task really include everything that I expect you to do?
Does it have unit testing in there? Does it have the database design? Does it have the mockups that we expect? Does it have the manual test cases written? Because our definition of done requires all of those things, but if I just say I’m going to get done task number one in a week, as your Scrum Master in my head, I’m assuming in that week you’re going to do all the things that I thought of, but in your head you’re not. And then all of a sudden I don’t know that until we get to the end and you say, “Okay, I’m done.” And I’m like, “Okay, great. Where are your unit tests? Where are your manual tests?” “Oh, I didn’t write any of those.” “Then, how are you done?” When you break it down into four hour increments and I don’t see a task for a writing unit test, I can challenge you before we even accept this, and you’ll either tell me all the unit tests have already been written or that’s a good point. Let me go add another task to go write those unit tests. And now we again, we have a more accurate estimate of what’s actually going to be happening.
And also, I as a developer, especially when I was younger, I thought a week was a long, long time. I thought I could get anything done in a week. Oh, yes. Put a week on that. I’ll be done well before a week. I mean, I can write an operating system in a week. Come on, give me a break. But no. A week is actually goes by really fast when you have meetings and lunch and distractions and family, you never get an entire week’s worth of work in a week. So, it just helped me go back in and make sure that my estimates more accurate, because I can tell you what I can do in four hours much more accurately than I can tell you what I can do in a week. So, the best advice I can give any team is stop accepting any estimate that’s over four hours long. It needs to be four hours or less, and you’re going to instantly see feedback from that.
STAS ZVINYATSKOVSKY: Great. And at that point you can actually stop estimating. You can just say all of my tasks are four hours or less.
DONOVAN BROWN: Yep, exactly. And you know how many of those you that you can do and you can also, like I said, I, it makes you think about it, which is really important, right? Because when I say a week, I don’t even think about that. I just think … My gut says that’s enough time. But I haven’t thought about the unit test. I haven’t thought about the database design. I haven’t thought about any of it. I just think, yeah, a week, I’m sure I can get it done in a week. That’s not really an estimate, right? That’s just like a safe card that you’re giving yourself. But when you’re really forced to think about it, you are truly estimating it. What am I doing? How long do I think it’s going to take? Let me articulate that. Does everyone on the team agree that I’ve covered all of our bases? And now we can go parallelize that and get it done in a much shorter time.
STAS ZVINYATSKOVSKY: Great.
DONOVAN BROWN: Cool.
STAS ZVINYATSKOVSKY: And I want to switch gears.
DONOVAN BROWN: Okay.
STAS ZVINYATSKOVSKY: And ask you the question. So, you are a principal DevOps manager at Microsoft.
DONOVAN BROWN: I am.
STAS ZVINYATSKOVSKY: How do you define DevOps?
DONOVAN BROWN: That’s a great question. And I was asked that question at Microsoft actually, and it took me 30 days to answer the sentence, because I didn’t want to go off and search the Internet for someone else’s definition and say, “Oh, this is what the industry is saying DevOps is.” I wanted to internalize what DevOps truly meant. If I’m going to be a program manager at Microsoft responsible for this methodology, this movement. So, I labored over it. Took me, like I said, 30 days and I thought about my own career inside of the industry. I’ve been in the industry over 20 years now, started back at Compaq computers as a developer. So I’ve seen a lot of software written well, I’ve seen a lot of software written poorly.
And the first question I asked myself before I defined DevOps was, why is DevOps even a word, right? Let’s step back a second. Why are we … Why am I being asked all over the world to speak about this concept called DevOps? Why is it even a word when it wasn’t a word when I started writing software? And I think the reason that it’s a word now is because it’s what hurts most now, right? And what I mean by that is if you look at Agile, the goal of Agile is to produce an increment of shippable software. Full stop. It doesn’t say anything about shipping it, right? Agile says produce an increment of shippable software, potentially shippable software at that, right? So, once you get really good at that, you’re thinking, why is it still taking us months, if not years, to get this shippable piece of software out into production? We’re producing it every two weeks, every three weeks, but it still takes months before I see it. And I think that’s where DevOps starts to come in and says, what hurts most now is the deployment of that actual increment of shippable software. That’s what we’re here to fix.
So, it’s the continuous delivery of value, that value that you’re producing, that we want to actually get out there. So, once I realized that that’s what hurts most, that’s when it kind of crystallized for me that DevOps is really the union of people, process and products to enable continuous delivery of value to our end users. That’s what we believe that it is. And value is the most important word. I … At one version of that sentence, I had the word software in there and I realized software really is not what we’re trying to do with DevOps.
What we’re trying to deliver is value to our end users, and that might not come in the shape of software. Maybe that comes with scaling up or scouting out our infrastructure, or a completely different network layout. I mean, it could be something that’s not the actual software that you’re writing that’s actually delivering value. I also wanted to focus on value, because I meet a lot of customers who have an IT pro team and a developer team that are working against each other, and the reason that I realized that is because our IT pros are incentivized and rewarded for keeping the lights on the server on. The easiest way to keep the lights on the server on is to not change that server. It’s very simple. But our engineers and our developers are all being rewarded by changing the servers as much as they possibly could. So, if one team is incentivized by not changing it and the other one is incentivized by changing it, they naturally work against each other to get the big bonuses.
But if we refocus their discussion and say it’s not about the software or the lights being on, it’s about delivering value, then all of a sudden both teams naturally work together. So, when I was writing that definition again, I changed the word from software to value. I changed the word from tools to products, because I think products are what you really need. A tool to me is like a hammer and a saw. Now, a hammer and a saw did not build your house. A subcontractor built your house wielding those tools. And I think things like bash and powershell and DSE are great tools. We give them away for free, but they’re not something I would want to build my entire DevOps pipeline in. Something like release management or Azure DevOps, that’s the product that wields those tools for me to actually produce an entire CICD pipeline.
And then why did you not use culture versus people? Because without your people, there is no culture, right? Your people are the people that live and breathe and cultivate that culture. So, to me it was people, process and products coming together that actually gives you DevOps. And I’m really proud of the sentence and it’s been published in books now and I see it on Twitter all over the place where people are using it in their PowerPoints and it’s on a lot of our Microsoft websites as well. So, I think it really resonates really well with a lot of people at Microsoft, and that’s the north star to continuously deliver value.
STAS ZVINYATSKOVSKY: That’s great.
DONOVAN BROWN: Thanks.
STAS ZVINYATSKOVSKY: And I love the definition.
DONOVAN BROWN: Thank you!
STAS ZVINYATSKOVSKY: Yeah. And so what do you see out there? How are people adopting DevOps, per your definition?
DONOVAN BROWN: A lot of people are having us come and tell the story that I told today, right? Here at this conference, it’s about our own internal transformation. And what is really interesting is everyone’s at a different stage of that transformation. Even inside of Microsoft, we’ve all agreed that we’re going to use one engineering system inside of Microsoft, which is Azure DevOps, but every organization, if be it Windows or Office, Xbox, Bing, HoloLens, Azure DevOps, we’re all at different stages of our transformations. And I see the same thing out in our industry. I see some of our customers who have a really nice pipeline, but are struggling with Agile. I see customers who are really good at Agile but don’t have a pipeline in place to actually move that value and no ones at the same place. And what I tell everyone to do is focus on what hurts most first. Don’t stop what you’re doing and go try to implement DevOps all at once.
Focus on your pipeline. And no matter how mature you are, there’s one part of your pipeline that gives you anxiety. You’re like, ooh, if it’s going to go wrong, it’s going to go wrong right here. This is the slowest part. This is the riskiest part. We got to fix this. That’s the only part I want you to focus on. And then what’s going to happen is when you solve that particular problem, you’re then going to realize that there’s another bottleneck in your pipeline, and that’s what I want you to go focus on next. But please do not stop what you’re doing and try to go do an entire DevOps pipeline at once. And I see some people in the industry trying to do that. And I … First thing I tell them is don’t. You already have a process. There today, and no matter how bad it is, there’s a way that your developers get ideas, and there’s a way that those ideas get into production. And what we want to do is analyze that process, find the part that’s the worst, and fix that, because that’s where you’re going to get the most bang for your buck.
Don’t fix the simple itty little stuff, the simple wins, those don’t buy you anything. It’s solving the thing that hurts the most first. So, that’s what I’m seeing people going off and doing, especially after I visited them, and they kind of get on board in what I’m trying to tell them to do, and I think that’s the most important way to go off and try to implement DevOps, is focus on one thing, thing that hurts the most, and then when that’s gone, focus on something else.
STAS ZVINYATSKOVSKY: Great. Thank you so much.
DONOVAN BROWN: My pleasure.
STAS ZVINYATSKOVSKY: We could talk for hours about these topics.
DONOVAN BROWN: Probably.
STAS ZVINYATSKOVSKY: But we should probably stop. Thank you for coming, Donovan.
DONOVAN BROWN: My pleasure.
STAS ZVINYATSKOVSKY: Appreciate your time. Thank you. Thanks again for listening to this edition of Agile Amped. If you learned something new today, please tell a friend, a coworker or a client about this podcast and subscribe to hear more inspiring conversations.