Why Agile Spreads Like Wildfire: Software Specifications
The principle reason that Agile development methods have caught on like wildfire is that they address the root cause of most software development project failures: Unreliability of software specifications. To put it differently, if software specifications were as reliable as building blueprints, then the principle justification for investing in Agile competencies would disappear. Agile practices might still have value, but the argument that they should be the dominant approach used by the industry would be much more difficult to make.
With this in mind, it’s worth considering the question: Are software specifications inherently unreliable or is their uncertainty the side effect of something else? For example, maybe it’s caused by widespread failures in execution that are endemic to an immature industry. If lack of industry know-how is the problem, then one should expect that, as the software development industry matures, problems with software specification will gradually diminish and ultimately achieve the reliability of specifications in more mature industries (e.g. building blueprints). On the other hand, if there is something about the nature of software specifications that makes them inherently uncertain, then methods, such as Agile methods, that help us manage through uncertainty will be needed for a long time.
This is not a new topic. Perhaps most famously, Fred Brooks wrote on this topic in his essay No Silver Bullet, first published in 1986. He raised the question, why, although computer technology rapidly improves, the craft of computer programming has not kept up? Even after taking into account huge advancements that have been made through the introduction of things like higher-level programming languages, relative to hardware advancements, software development improvement proceeds at a snail’s pace. After discussing several reasons why software development is inherently difficult, he concludes that indeed, when it comes to software development, there is no silver bullet.
One of the difficulties he explores is the inability to visualize software systems. In other words, you cannot draw a picture of a software system that geometrically corresponds to all of its relevant functions.
For example, consider a blueprint of a building. Each element of the blueprint is in a one-to-one correspondence with some aspect of the finished building and at scale. With a bit of training, most people can project a pretty accurate vision of what the finished product will look like and how they might relate to it. Furthermore, information contained in the blueprint can be used to project an accurate 3-dimensional representation of the future building making visualization quite easy. Blueprints work because they represent buildings that are spatially fixed in three dimensions. Now let’s consider a software system.
Executing software does not correspond to anything that is spatially fixed. We can diagram elements of the software system but often the relationship between diagram and element is quite abstract and is not meant to map to anything physical. And though some elements have visual qualities, there is no unifying spatially-fixed structure that makes everything coherent or intuitively obvious. The following list of aspects of a computer system illustrates these points:
- Computer topology (physical servers, workstations, devices, networks)
- Application architecture (database, user interface windows and representations, servers, web services, browsers, etc.)
- Data model (tables and relationships found in a relational database)
- Business rules
- Consider UML diagrams used to model the object-oriented design of the behavior and structure of software (that’s right there are 14 different diagrams).
Even after you add all these views together, your picture is still nowhere near as complete as the blueprint of a building. Software is much more complicated than a building, and unlike a physical thing it’s difficult (literally and figuratively) to get your arms around it. In addition, software is more complex than a building. Something that is complicated has many moving parts and perhaps a deeply nested bill of material. Something that is complex adapts and changes as it interacts with its environment and has behaviors that are practically (if not theoretically) impossible to exhaustively predict. In addition these interactions may cause unintended side effects, known as bugs. From a specification perspective, there is often no practical way to enumerate all the possible ways that a software system might behave as it interacts with users and other systems, before it is built. For these reasons, we can conclude that at least for highly interactive software systems, written software specifications that precede coding are necessarily incomplete and therefore inherently unreliable.
Agile software development anticipates the need to close the gap in software specification while the software is being constructed, since the spec can't be completed before you start.