Technical Debt Is Killing Your Business

Technical debt can cause critical issues for organizations. Every year an organization does not address these issues brings them closer to ruin. Some businesses even have to shut their doors due to irreconcilable technical debt that conventional solutions alone cannot address.

Arlo Belshee, full-stack Agile developer and technical coach, shares insightful knowledge on building better software, specifically as it pertains to what he calls safeguarding: “a simple, 22-minute practice that allows you to find, fund, and then execute real changes to your process and product.” One of the things that makes fixing technical debt so difficult is that “there is no one problem of technical debt.” Belshee advocates for empowering teams to own problems, to know what’s going on in their part of the system and to work quickly to address these problems.

Accenture | SolutionsIQ’s Stas Zvinyatskovsky hosts at the Deliver: Agile Conference in Nashville, Tennessee.

Listen on iTunes    Listen on Spotify

Read the full transcript

STAS ZVINYATSKOVSKY: Welcome to another edition of Agile Amped. I am your host, Stas Zvinyatskovsky, and we’re podcasting today from Deliver:Agile conference in Nashville, Tennessee. Today, my guest is Arlo Belshee. Hi, Arlo.


STAS ZVINYATSKOVSKY: Arlo builds high quality teams at large companies. He’s a full stack developer, full stack doer, and he’s held various roles over his life, from developer to CEO and everywhere in between. He uses his vast experience to coach others in their various roles and to unlock potential to work better together. Recently, Arlo has been talking a lot and focusing a lot on technical debt, and we’re going to get into that. Technical debt, of course, is a critical problem for many organizations large and small. Welcome, Arlo.

ARLO BELSHEE: Thank you.

STAS ZVINYATSKOVSKY:: So let’s get straight into it. Why is technical debt important?

ARLO BELSHEE: All right. There’s a couple of things. First of all, it is a critical element for many businesses. Many businesses will die in the next year because of technical debt, and that’ll be true last year and the year after next. Every year, lots of them will keep dying because of technical debt related problems. Bugs cause tremendous loss of revenue, they cause tremendous downstream costs, operational costs to explode, all sorts of things. Technical debt shows up for other companies in a whole bunch of other ways. Companies that don’t have a lot of nominal bugs still will have issues that show up and skew A/B test results and then they start going the wrong direction as a business because they followed data that was, unfortunately, saying more about their debt than about their customer.

So it’s a big problem, and at the same time, unlike planning or product direction, it tends to be a problem that leadership is very happy to delegate to teams and say, “It’s something for the technical team to address.” So that combination of factors is very unusual and allows us to make some fairly significant dramatic shifts in ways that really matter to the company and also then open up companies to whole new ways of working.

STAS ZVINYATSKOVSKY: That’s great. Why do you think the problem of technical debt is still so present?

ARLO BELSHEE: Number one reason is because there is no one problem of technical debt. I did a survey when I was at one company, but I’ve seen results similarly in other places. We looked at 104 teams. We asked them, “What are the top three technical debt problems that you see that you’re working on, you need to go do something? And what are the top three that you see that you can’t do something, you need someone else to do something about it?” We found that the one that had the most votes had six teams, three that thought it was their problem, three that thought it was someone else’s problem, and they didn’t mean those three teams. They meant leadership or someone else. The median technical debt item had one team voting for it; the mode had one team voting for it.

So the job of leaders is to align the group, to get alignment to go after the most important problem. When you’ve got 104 teams and you’ve got 135 problems, which one should you go after? Any direction that you might set is guaranteed to be right for 1% of the group and wrong for 99% of the group. It’s a nearly unique problem in that way. Not actually totally unique, but nearly unique. So it requires a very different type of organization and very different set of approaches. You can’t solve it in the normal ways. Everything that we’ve developed as businesses is about solving problems where we align and we go after the problem. It doesn’t work for technical debt.

STAS ZVINYATSKOVSKY: Interesting. So what can companies do?

ARLO BELSHEE: What can companies do? Okay. There are a number of things, and some of these are places where I’ve done some innovation and some of these are things that already existed. For example, if you create teams that really own problems and they can really effectively own what’s going on, then you can create an organization that can in an aligned way go many directions at once. The simplest way to describe this is Marquet’s Turn The Ship Around, leading by Commander’s Intent, and Red Work Blue Work, all that stuff.

What you need is a practical way for leaders to consistently create that environment in their teams, because ownership is not something you can hope for and it’s something that you have to create. You have to create it as leaders and as teams. So that’s one of the things that we do, call it GRO, Growing Responsible Ownership. We have a set of practical, pragmatic techniques for both teams and leaders to do, that create that ownership. That’s one of the three parts of our framework, and it’s designed to address exactly that one problem of how can you create a group that can go aligned in a good, supporting each other way, in 135 directions at once?

STAS ZVINYATSKOVSKY: Great. You mentioned the framework already, and before we started you mentioned to me that you’re starting a new company.


STAS ZVINYATSKOVSKY: Congratulations on that.

ARLO BELSHEE: Thank you.

STAS ZVINYATSKOVSKY: I wish you all the best with the new company, and this is your specialty, technical debt.

ARLO BELSHEE: Mm-hmm (affirmative).

STAS ZVINYATSKOVSKY: And you created a framework to address technical debt. You just talked about one aspect of it. What are the other aspects?

ARLO BELSHEE: Yeah. We’re deep roots. The other two aspects are safeguarding and code by refactoring. After you’ve built an organization … Well, it’s actually not after. It has to be done in parallel. But in addition to building an organization that can go in multiple directions at once, you have to also build individuals that can make changes in technical debt without externalizing costs. That’s what code by refactoring is about. That’s the second part of this framework, and it’s a different way of doing the coding so that when someone makes a change they don’t cause an impact on another team. That results in fewer cross team dependencies and makes it more possible to actually implement going multiple ways at the same time.

Then the third thing is you need some way to make legitimate trade offs of technical debt and features and priority and which ones to work on, and that is safeguarding. You need to make that in a distributed way. Again, if you’re going 135 directions at once, you can’t have some set of people that meet quarterly for a programming planning make a decision, because it’s too slow and it can’t go 135 ways at once. So safeguarding is a mechanism to allow a set of guidelines and frameworks so that individual teams can make legitimate decisions that incorporate knowledge of the whole throughout the organization and determining what to fund, when, and how.

STAS ZVINYATSKOVSKY: Great. What are the examples of some of those techniques and frameworks?


STAS ZVINYATSKOVSKY: For safeguarding.

ARLO BELSHEE: For safeguarding. Okay. There was a team that had, they noticed that they were spending a tremendous amount of time repairing their automated tests. They had a fair number of large end-to-end automated tests, a small number of unit tests, and one of their biggest debts was that test framework. They were sitting, by the way, right next to a team that had identified that their biggest problem was they didn’t have enough automated tests. So the one team’s metric was to decrease automated tests and the other team’s was to increase automated tests.

So these two teams were working on these things, and they just started looking at the bugs that they have or the times where the test gives a false positive and fails erroneously for each of those, what are the things that they could do to make that a little better? So safeguarding gives a rigorous process of looking at hazards, looking at what set people up to make decisions, what set the situation up to happen that way, and then making incremental partial progress towards that.

So over the period of time, they ended up refactoring the code, changing a couple of development processes, changing some ways that two of the teams were interacting person-to-person, and got it to the point where they still had about 70% of the automated tests that they did before that were large and not too useful, but they didn’t have any false positives coming up anymore. So the problem was gone, and the other 70% they could effectively ignore. They didn’t have to pay to fix them.

STAS ZVINYATSKOVSKY: Interesting. When you come to companies, they’ve gotten to a pretty bad state with their technical debt typically. That’s why they call you.

ARLO BELSHEE: Yeah, the worse the better.

STAS ZVINYATSKOVSKY: Right. So if they keep on doing what they’re doing, it’s just going to get worse.


STAS ZVINYATSKOVSKY: So in order to get out of that mountain of technical debt, they need to radically change their ways. What is your view on how much of it is possible grassroots and how much of it has to be done top-down?

ARLO BELSHEE: I talk about sideways-in transformation. I don’t believe that top-down or grassroots works, and I also don’t talk about transformation as much as transition. It’s about one day at a time, one day better. Habit change is really, really hard. It only happens with support and it needs to be supported by your peers, it needs to be supported by people who are giving you feedback, it needs to be supported by people who are giving you the things that you need to get your job done all around you. And it needs to happen gradually. You can’t just decide, “I’m going to get up in the morning and I’m going to weigh 400 pounds less.” No, you’re going to work at it for two years. And in two years, you’re going to be there.

Technical debt’s the same way. Companies are weighing a lot more than they need to. So what you need is to get a set of people that are mutually reinforcing. I talk about getting the golden chain. You’ve got all the way from a team of doers up the line of anyone who can basically veto a decision that team might make until you get all the way to the top to someone who doesn’t have veto power anymore. At a given organization, that might be all the way to the CEO or it might be halfway up or whatever. Then you get that group to change as a unit. They all change their habits all at once, supporting each other and moving to the new habits, and they change those habits gradually. As a team changes, as a golden chain changes, then everyone in that golden chain can help their peers on other golden chains, other teams vertical report move. So it moves sort of sideways through an organization.

STAS ZVINYATSKOVSKY: So a buy-in still is required, then, from those folks on top.

ARLO BELSHEE: Buy-in is required from everybody. If you don’t have buy-in from the top, it doesn’t work. If you don’t have buy-in from the bottom, it doesn’t work. If you don’t have buy-in from the middle, it doesn’t work. If you don’t have buy-in from HR, it doesn’t work. If you don’t have … Technical debt is a problem that requires everyone to buy in. That’s part of the reason why I go after technical debt and not other problems, is because it’s one of the ones that a champion in the company can look at it and go, “Guys, right now we’re doing medical insurance stuff. We’re charging everybody $700 a month. Of that $700 a month, $400 goes out to doctors, $50 goes to our profits, and $250 a month we pay for software bugs.” What the heck.

STAS ZVINYATSKOVSKY: That’s right. I used to get these questions a lot from individual developers, and the question would go something like, “I’m on a team and we’re drowning in technical debt. My manager is not necessarily buying into the idea of getting out of technical debt. What can I do as an individual contributor?”

ARLO BELSHEE: Safeguarding. I laugh, but safeguarding is designed, invented to solve that problem. The whole point is that the way that individual contributors talk about technical debt with other stakeholders generally doesn’t work. Usually, the discussion goes something like, person outside says, “I want X,” and the person says, “Yes. Okay, sure. It will take me nine months.” And, “That’s too much.” Then, “Yeah, because of the technical debt,” and back and forth. Then, “Well, how much will it take to solve the technical debt?” “I just need you to free me up, just give me four months, without any features, without any anything, just give me four months,” to which the business person says, “Well, yeah, that’ll pretty much kill us.”

The request is one that the business person legitimately should never, ever accept, and so they don’t. What you need to do is find a different way, not just a different way to make a request, but actually a different solution to offer. Safeguarding takes any technical debt big project and turns it into small incremental ones that maximize ROI on a daily basis so that you can identify, “All right, here’s an opportunity where I’m going to increase the cost of this two-day task to two and a half days, and in addition to the value deliver over here, I’ll get this incremental small value deliver over there that goes towards the technical debt costs, goes towards reducing bug costs, etc.” You pay those incrementally and you can choose where and how to balance and how to fund. As a result, you now can get a legitimate funding discussion happening and you can actually make those trade offs between the long-term health of the company and short-term health of the company and get everyone on the same page.

STAS ZVINYATSKOVSKY: That makes a lot of sense. Switching gears a little bit, you mentioned Turn The Ship Around, and it’s one of my favorite books. I recommend it to a lot of people. In the book, he talks a lot about giving people the proper abilities to make decisions on their own and execute on those decisions, so training people, giving them data. How does that apply to the problem of solving technical debt?

ARLO BELSHEE: Yeah, okay. It’s interesting. It’s a little bit different here, and the main reason it’s different is that with technical debt, the vast majority of data actually comes from the individual contributor. With planning, the information is coming from outside the company. With navigation, the information is mostly coming from outside the submarine. So there’s a fair amount of, how do we disseminate the information to the teams? With technical debt, the problem is almost the reverse.

The information’s already in the team. How do we get out of the way of them? And they’ve learned to ignore the data. They’ve learned to ignore the information because of insert your company’s cultural challenge here. Everyone has a different one, but there is something. And there are reasons why that information doesn’t get out, and the decision making loops are too long and they involve people who aren’t involved in the information, like the executives. Like, really? Is an executive going to be able to make a trade off between two designs in code? I hope not. They’ve got better things to do, right?

STAS ZVINYATSKOVSKY: Their time is better spent somewhere else.

ARLO BELSHEE: It really is. So with technical debt, it’s really about unlocking the team’s ability to understand its own data to get better at understanding data, which is usually a weakness of the team. So the GRO technique, Growing Responsible Ownership, is about helping leaders get out of making decisions for a team and into the business of being a mirror for a team and helping them see how effectively they’re using data, how effectively they’re making decisions so that they can improve their effectiveness. They’ve already got all the information. They’ve already everything they need.

STAS ZVINYATSKOVSKY: Perfect. In terms of training, do you find that engineers … We’re talking primarily about software engineers … do you find that software engineers are usually enabled to get out of technical debt, or do they need additional training, help?

ARLO BELSHEE: They need a change in habits. Rarely does that mean they need training. If you look at the behavioral engineering model or any of the influencer models there, they talk about what causes humans to make decisions in whatever way they do. There are a set of influencers, they’re roughly prioritized. So, your norms and expectations, what you believe to be true, your world view, has a stronger impact on your decision making than almost anything else. Peer pressure is sort of an external form of that same thing. Whereas your skills, your knowledge, it’s fourth on the list. The list is about six items long, and fourth on the list is skills and knowledge. It’s a real thing, but it’s not the top three. So usually, you need to solve the top three and then skills and knowledge follows.

Now, often there are also some missing skills and knowledge and you need to fill those in. But even then, when we talk about training, everyone’s thinking, “Let’s do a workshop, let’s take people away from the code and do an intensive and whatever.” Plain and simple truth is those don’t work very well. The way most people learn most things is you’re doing the work and you pick it up by doing the work, doing the work with other people, you get an insight. You invent 3/4 of what you learn and you test it by experimenting and the other quarter you get from a peer and a buddy, and chatting over the lunch table.

So when we’re helping build skills at deep roots, we follow that model. Everything is on the job. There isn’t a go to a workshop ever. It’s designed that it’s the storytelling around the lunch cooler, it’s the play with the work and reinvent things. We give you challenges that will help you invent the right ideas because you’ll learn it a lot faster that way than if we tell you what the right idea is. And you learn it by doing it in your real work.

STAS ZVINYATSKOVSKY: Well, Arlo, we could talk for hours. The problem of technical debt is a huge problem. Thank you for trying to solve it for all of us, for the industry. I appreciate your time with us today.

ARLO BELSHEE: Thank you.

STAS ZVINYATSKOVSKY: Thanks again for listening to this edition of Agile Amped. If you learned something new, please tell a friend, coworker, or client about this podcast and subscribe to hear more inspiring conversations.