Picture the scene. You’re running a Formula 1 motor racing team, and you’ve unearthed the most talented young driver you’ve ever seen. Using the latest state-of-the-art cars, training methods and digital simulators, you hone his skills to make him the fastest human on four wheels—able to respond in milliseconds to every possible on-track event, type of turn or change in road conditions. He’s a master of racing, and nobody in the world is better.
But when you arrive with him at the track for his first live race, you find something’s gone wrong. The only car available for him to race in is a museum piece from the 1980s. Not surprisingly, he trails in last.
Busting the myth of agile
When it comes to driving innovation in business, agility is today’s buzzword. Businesses followed the hype curve in countless articles portraying agile teams and methods as the vital unicorn that makes innovation happen. They invested heavily in training agile teams, adopting agile practices and even changing organizational structures. And often, all of this had little quantifiable benefit and led only to disillusionment. But, in fact, what organizations were doing was training the “driver” while ignoring most critical impediment to innovation: legacy technology. The simple reality is that these agile teams can only move as fast as the old car they are driving will allow.
I once had a client tell me: “When we want to slow things down, we call legal—and when we want things to stop, we call IT”. It’s not because IT is malicious (for what it’s worth I’m guessing legal isn’t either). It’s because IT has the unenviable task of integrating whizzy new digital products into the less sexy legacy core that continues to permeate and dominate today’s organizations. These legacy estates were fundamentally not designed to handle rapid change. The result? No matter how agile a team you build, change will be slow. To return to our analogy, we’re off to the motor race with a great driver…but driving an 80s jalopy.
Legacy: not just a constraint—but key to innovation
Given the drag that existing systems and technologies impose on innovation, it’s little surprise that businesses across the world tend to view their legacy with scorn, as they see it eating away valuable investment dollars and frustrating technologists and users alike. Indeed, the legacy baggage impeding incumbents is often used to justify why startups move faster. This baggage also brings a heavy financial cost, which is often termed the organization’s “technical debt” (see below).
But the irony here is that while legacy systems constrain innovation, the reality is that innovation cannot happen without legacy systems being involved. This symbiotic relationship exists for the simple reason that legacy systems hold the new economy’s most valuable commodity: data. Many companies’ most business-critical information—on customers, products, citizens, partners, distribution channels, and more—is locked up in their legacy systems. And organizations that unlock this value, and act on it with speed and agility, will win the war for relevance.
What is technical debt?
Technical debt—a term originally coined by Ward Cunningham, a programmer renowned for developing the first wiki—is widely understood to refer to the sunk cost of hardware, software, lines of code, storage and other tangible aspects of the existing IT estate. Companies that delay dealing with these legacy systems and technologies will effectively constrain the capabilities of their newer applications and the agility of their business. As a result, they can build up a heavy burden of technical debt.
The cost of technical debt has been estimated at 15 percent of the initial application cost annually. What’s more, in the same way as financial debt incurs interest, if technical debt is left unaddressed it rises over time. The results include poor software performance, an inability to change or update software and an increased risk of failure—all adding further to the costs.
Accenture has expanded the concept of technical debt to the wider “technology debt”, which encompasses all the inefficiencies within the existing technology function. Alongside technical debt, a key component of technology debt is opportunity cost—defined as the benefits that could be achieved with new products and features, and opportunities hindered by system inflexibilities or inefficiencies. This equates to lost agility. Accenture research has found that 85 percent of executives believe that legacy hinders their ability to move to a more digital model.
A race where many are trailing
It’s a race that many companies are losing. Accessing data held in legacy systems is key to both successful innovation and agility. But according to research published by the self-service application and data integration company SnapLogic, organizations in the US and UK are losing an aggregate US$140 billion a year due to “disconnected” data. Based on interviews with 500 IT decision-makers and cloud/hosted application users, SnapLogic’s study found that 41 percent of respondents had critical company data trapped in legacy systems that couldn’t be accessed or linked to cloud services. Some 76 percent had at least some data trapped in this way.
Such issues become less surprising when you consider that what constitutes “legacy” is in the eye of the beholder. It might be that creaking Fortran system from the 1970s or a shiny new SAP application. For the purposes of data, the fact is that as soon as you put a system into production, it’s legacy. Like a new car becomes “used” the moment you drive it off the lot.
What’s more, data in legacy systems is not pretty—or at least not necessarily aligned to data in your other systems. It’s often poorly structured, uncorrelated, fragmented and not easily accessible. But it is there. Organizations that figure out how to build the foundation to get at data in legacy systems rapidly and in a systematic way will become agile. And agile organizations will win. So the message is clear: legacy is yours for the taking—and embracing your legacy is the route to agility.
Build the foundation…
Your legacy estate is not going away any time soon. Neither does it need to: you can still use it to help drive innovation. To do this, organizations need to reimagine their enterprise architecture. Not the process by which you design systems—but the patterns, practices, tools and actual code used to facilitate the integration of systems within your enterprise, so they can share data and logic.
The “North Star” of this vision is not new middleware. It’s actually no middleware. No central data stores or lakes, no queues, no services buses. There is much to unpack in what a modern, digital enterprise architecture looks like, but the key to building it lies in embracing four principles:
- Federate persistence of data. No central data model has ever been designed effectively by a large organization. Instead, the optimal approach is to manage data in specialized micro-services that serve a specific business need and provide tangible value. Over time, you can transform your monolithic legacy systems into sets of micro-services, by carving off pieces tied to specific business objectives one by one and keeping the data managed within the smaller services.
- Data-centric architecture and exchange. Define temporary “documents” to represent data to be shared between systems. These documents are not maintained anywhere in a persistent way, but are simply a means to carry payload between services and are then destroyed. However, while the documents are transient, the format is enduring and—critically—represents business meaning. Each document should represent a business entity, be part of a business domain, and have well-defined relationships to other documents. It is important to not be constrained by how the underlying data is stored or managed in any given system or service. This logical structure is virtual, not physically represented in any single database, and will be designed and evolved over time, not in a single project, but business proposition by business proposition. It is about embedding business meaning into where data is shared—not where it’s stored. This is a strategy to have many databases but a single view of data.
- Real-time eventing. The “documents” I discussed above are implemented as events and underpin an event-driven architecture (EDA). Leveraging an EDA will enable real-time exchange of data, equipping you to surprise and delight your customers or trigger actions based on specific events in real time. Micro-services can act on fragmented events coming from different legacy systems and integrate these into more meaningful business events, in order to decouple dependencies on legacy systems and represent data in the form it is to be consumed in. Crucially for innovation, analytics can be added as just another consumer to real-time data—giving organizations insight into events as they happen, not after a batch window closes. And finally, the publish/subscribe nature of an EDA will reduce the calls made directly to legacy systems, making new digital capabilities faster to build and legacy easier to transform. This is multi-speed IT.
- Integrate and automate. It is one thing to build cross-disciplinary teams, but it’s quite another to make them effective. This requires them to have DevOps tooling to embody integrated processes. Automating builds, deployments, migrations and promotions is crucial to effectiveness—and integrated pipelines and backlog management will drive Ops into Dev and vice versa.
While tools and frameworks will be required to achieve this vision, it’s less about products and more about a way of life. Designing events to be meaningful—and therefore durable—will require business-IT teams to explore use cases together, sharing an understanding of business domains, context and entities. Federation of data will be achieved through creating and clearly communicating a set of principles and ensuring that teams follow them accurately.
…but don’t forget about the driver
Finally, we cannot let focusing on these principles make us forget about the racing-car driver. Or, in our case, the team and the organization. Just as creating an agile operating model without addressing technology constraints won’t enable an organization to innovate at speed, so the reverse also applies. To bring speed and innovation, we need to embrace agile methodologies, product-focused teams, and the right funding models, organizational structures and measurement. Organizations that conquer both objectives will create tremendous value, delight their constituents, and lead our future. Go out and conquer. Go out and win that Formula 1 race—by combining the best possible car with the best possible driver.