At times both the Agile and traditional Waterfall camps get hung up in an unproductive game where they start digging through the details of a problem domain, find a specific circumstance and say, “ah ha! A purely [Agile / Waterfall] approach can’t deal with this situation, therefore that approach is wrong”
Indeed, this binary obsession – that you must pick one specific approach and apply it to all dimensions of a project – is entirely unproductive. In fact, it is usually not the reality in an Agile or traditional project, but rather a limit of our own thought process. The challenge being that if we find merit in one approach, the mind can take an absolutist value. Agile is useful for dealing with product visions, therefore we should apply Agile principles to every aspect of our project.
The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.
F. Scott Fitzgerald, “The Crack-Up” (1936)
Fitzgerald accurately describes the challenge facing us when organizing and executing projects; we must hold two opposing ideas within our head and continue to function. These two opposing ideas well well described by Kevin Kelly when he coined the terms clockware and swarmware as the two different approaches to managing work. They have since been embraced by many complexity theorists and I present them here as my own attempt to hold two opposing ideas and continue to function and show how one builds upon the other, and to argue that a project requires one or the other will inevitably lead to failure.
This is the perspective best described as deterministic or Newtonian in nature. Alluding the the mechanics of a clock, situations within this domain may be very complicated, but they are ultimately predictable and manageable with sufficient analysis and calculation. This perspective is widespread throughout western thought from the view of factory work expounded by Frederick Taylor’s Scientific Management to the modern educational models which lean heavily upon standardized testing and closed problem solving. When applied to modern management and project work, we see deep analysis, detailed plans and an overall effort to control the unfolding process. This is where most Scrum practitioners and Agilists become critical of this mindset, showing how a heavy analysis on planning leads to brittle projects that spend more time trying to adhere to a plan than ensuring they are getting the best outcomes possible.
If clockware assumes a static, closed system, then swarmware is focused on solving problems in a diametrically opposite environment. Swarmware frames problems as infinitely complex and changing in response to numerous factors including the actions one undertakes to manage the system. Based on this perspective, swarmware mandates a highly adaptive approach focused on experimentation, feedback and adaptation. Scrum would be an excellent example of a management approach that embraces this mindset. Frameworks like this are focused on emergent patterns and self organization over centralized planning.
One Building Upon the Other
The challenge in elaborating these two options as distinct choices is that it implies you must use one or the other, that there is some sort of active decision about which approach you think is better. Interestingly, some of the most powerful demonstrations of swarmware are conducted by organisms completely incapable of the simplest decision making, even if it appears they contain a significant level of intelligence. I recall at my old apartment a six month battle I waged against a colony of ants. Had I not known otherwise, I would have sworn there was a plotting queen ant somewhere telling her minions to avoid the places I put poison and to continue searching other cabinets as I tried numerous attempts to move food around and set traps to counter their offensive campaign into my kitchen. [Note of personal disclosure, I am an avid fan of National Geographic and museums of science. If you didn’t know this about me already, you will by the end of this post]
If we consider the ants who plagued my apartment kitchen and terrorized my cat, they were clearly demonstrating complex, adaptive behavior, or as we’re calling it: swarmware. They first found a small spill of olive oil on the floor, and that got them excited. We cleaned up the spill, but it was too late. They found access to a few other spare scraps low to the ground and before long they were climbing up on counter tops and getting in cabinets. For creatures with a brain about 1/40,000th the size of a human, they were displaying some pretty creative solutions to find my food and overcome my attempts eliminate them. However, if you look closer at the behavior they were using, it was incredibly simple. We can clearly define the rules that would drive a foraging ant. In fact, we could almost write it as a series of strict rules. Let’s take a look at foraging behavior where ants use pheromones – scents they emit to lay navigation trails, communicate with other ants, and do a number of other activities – to enlist more ants in the efficient collection of newly discovered food sources.
- Follow the strongest pheromone trail you can find until it comes to food.
- If you can’t find a pheromone trail, begin to explore randomly until you find food
- When you find food, follow the pheromone trail back to the nest
My apologies to any biologists who may stumble upon this post – I’m sure I am simplifying a bit – but that is basically how it works. The ants follow a series of prescriptive rules without any judgement. The ants don’t look around, see that it is raining and adjust their strategy. Simply by blindly these 4 rules, they can get quite effective behavior. Here is a demonstration of how diligent application of a deterministic model – let’s call it clockware going forward – can lead to some highly complex and adaptive behaviors.
Our thought experiment follows the adventures of three ants. Assume it’s the beginning of the day, so no ants have laid down pheromone trails. Thus, they all pass through step #1 and move to step #2, begin foraging looking for food. Each ant heads out in a random direction. Two ants, after searching for some time, manage to find left over pizza. This triggers step #3, they collect food and follow the pheromone path they laid back to the hive. Our poor third ant doesn’t find food, so step #3 doesn’t get triggered for her – worker ants are all females, infact the only males in an ant colony are used for mating and they only live a few weeks, infer from that whatever you would like.
After one hour of foraging, two of the ants have made multiple round trips to the food, strengthening the scent of their trails. Now, four more ants come along to help forage. They start with step #1, look for a strong scent and follow the trail. The ant who wandered off an hour ago and hasn’t returned left the weakest trail – it was only laid by one single way trip – so the ants ignore it. The other two trails are 4x and 6x stronger as they have had multiple round trips laid by a single ant. Let us assume 3 ants follow the middle trail and 1 follows the trail to the right (nobody said ants were terribly smart).
After a second hour, it has become very clear that one pheromone path is significantly stronger than the others. The trail on the left has never been walked by another ant – her fellow ants are probably fearing that she may never be coming back at this point in time. The other two trails have been reinforced by ants traveling back and forth on them, but the middle one is 33% shorter than the far right trail, thus the same number of ants on that trail still make more round trips, thereby making the scent strong, thereby attracting more ants. This feedback loop continues and as more ants show up, by following the 4 rules they would self direct to shortest route leading to the food. I will offer my disclaimer again that I am oversimplifying; ants have demonstrated incredibly complex behavior including visual navigation, training younger ants, and understanding events tied to the time of day. However, the point remains that such a simple set of rules, strictly enforced, can lead to surprisingly complex and dynamic behavior. In the proper context, clockware can be used to empower adaptive swarmware behavior (Weston, 2001). (In case you are more interested in Ant behavior, I would refer you to http://blog.wildaboutants.com/)
Thus we see an elegant demonstration where the ants operate using simple, clockware-like, rules and the absence of a central planning authority allows these simple rules, when combined with feedback, to lead to some pretty adaptive behaviors. I should reinforce that the important characteristic of this sophisticated model is that the clockware was very simple in nature, the ants don’t apply rules about where to go searching for food, nor do they follow some centralized authority or plan. The clockware they use is a simple set of rules for responding to stimuli. The complexity comes in the number of ants applying these rules, the small randomization inherent in their activity, and the feedback where ants attract more when they find food that can be effectively retrieved. Of course, in modern projects, we are dealing with significantly more sophisticated actors and problems, so I would not venture to argue that team members on a software development team could operate with such a simple set of rules. However, the concept still applies: simple rules for small behaviors undertook by a number of people can enable highly complex and adaptive systems. Let me offer some examples.
Standardization and Innovation in Manufacturing
This concept has been realized and exploited by many companies under different names. Toyota offers a powerful example. Most people are very familiar with the innovative concepts Toyota has developed allowing them to implement what they call “operational excellence as a strategic weapon” (Liker, 2003). The company averages over 50 suggestions per employee per year, and implements well over 90% of the suggestions. This level of innovation and adaptive change can only be effective when coupled with a high level of disciplined clockware. Toyota is methodical about defining units of standard work and ensuring that people then work according to those standards. This may sounds like some traditional, heavily bureacratic organizations, but there are a couple key distinctions. Standards are created for the purpose of improving them, they are created by the people who do the work, and those people are empowered to change the process over time as they find ways to improve it. Thus they implement a system not unlike the scientific method with a number of variables under control in order to experiment different options and see their impact. Only with this level of discipline, measurement, and control, can the organization be free to try so many innovations and see if they make a positive performance. Much like the ants, disciplined simple behavior allows for feedback to foster more complex behavior within the system.
Applications within Agile Projects
Within an adaptive framework like Scrum, certain rules are meant to be clockware. With this perspective, the sanctity of what people call canonical Scrum begins to make sense. This is the same paradox illustrated with Toyota, in order to be Agile we execute certain simple behaviors in a deterministic way. The mechanics of something like the daily stand up are strict and limiting, but adherence to them helps people gain better understanding of what’s going on and then leads to more self organization, because everyone has heard what everyone else is doing. Teams hold their stand ups the same time every day so that people are more likely to remember and be able to attend, even if this can be inconvenient from time to time. Participants self-censor their commentary to answer the three questions (what did you do yesterday, what are you doing today, what are your impediments) in order to focus discussion and keep the meeting short, otherwise people won’t be able to get all the information they need in a timely basis.
The counter example to this is the Agile team that, in the spirit of Agile, tries to adapt on everything all the time. People feel conversation is important, so they abuse the rules of a stand up and it ends up taking an hour a day (this could cause this single meeting to consume over 10% of the project time). Teams want to be able to finish a feature in an iteration, so they extend the date a couple days in order to finish, leading to inconsistent measurements of throughput and a false sense of progress. Scrum Masters allow product owners to accept work that is not really done, leading to an accumulation of technical or other forms of debt that will extract a serious toll from the project later. These are all examples where the swarmware concept was applied incorrectly, we tried to be dynamic and flexible on the wrong dimensions, which undermined actual feedback or execution. Generally speaking, you can tell your are experiencing this challenge when you hear people around you state some egregiously awful behavior they are about do to and then say, “it’s okay, we’re AGILE!”
When dealing with how a team will operate, Agile Coaches and ScrumMasters need to ask themselves if something is Swarmware or Clockware. For those things that we agree are in the domain of complex adaptive systems, we will want to leverage things like emergent design, experimentation, and adaptation. This is the swarmware of your project and it probably involves things like the overall technical design, the product backlog and the overall release plan. We expect these to change and will be very careful to not add too much forward looking detail so as to make them brittle or extend our assumptions too far past reality.
On the other hand, there are things that we don’t want to keep spinning on. This would include things like the application of something like test driven development, coding standards, or even the basic rules of a framework like Scrum or Kanban. Iteration dates and agreed upon standards like a definition of done should be set and not changed on the fly.
Evolve Swarmware, Increment Clockware
In any adaptive project, we will most likely change both clockware and swarmware, but we go about changing it in different ways. When dealing with swarmware, we embrace uncertainty and may allow new versions to emerge. Developers will begin with a simple system and take advantage of emergent design to slowly grow a more sophisticated system. In the case of clockware, we don’t evolve so much as increment. When a team sets a definition of done to state that done work must be tested in their QA environment, they shouldn’t adjust that definition on the fly from iteration to iteration if the QA environment is unreliable. By definition that is not a standard. However, the team may decide that they need a new environment for the PO to accept work, and change the definition going forward. This is exactly like the Toyota works setting standards and then methodically improving upon them. In these cases those, certainty is important and the standard should be followed until it is changed.
In the end, the situation is akin the one explored with ants, most of what we do should consist of complex swarmware built on top of a series of simple pieces of clockware. Let me offer an example of a team I worked with recently. The teams developers had to build a new application upon a specific platform with certain agreed upon coding standards. Code was built based on automated unit tests and subject to code scans for things like cyclomatic complexity and other best practices. They delivered work every two weeks with agreed upon review dates for stakeholders and work was only called done when it adhered to an agreed upon list of conditions. This is all clockware. These rules were clear, detailed, and unchanging within the iteration. Between iterations they may adjust their definitions, implement new policies and make other changes, but once established, the rules must be followed. On top of this the developer maintained a dynamic design document they updated as they completed each feature. The product owner had a list of features that were not yet started and she was free to change the order and add new things at any point in time. The team would change task assignments as work progressed along their board. Thus, these more high level activities evolved over time and allowed a greater level of fluidity than the more concrete standards, they were treated as swarmware.
So as you look at your project and get into discussions about how flexible you should be about something, or what level of discipline is needed, ask yourself if you think this is clockware or swarmware. Is it a simple fundamental activity which can probably be clearly defined? Is it fundamental to the value proposition of your product? Using a model like this, hopefully you can maintain within your mind that we must hold two concepts within our mind. Our projects ultimately decompose to a lot of clockware, relatively predictable work that can be standardized, measured and managed. However, these small pieces quickly add up to highly complex things that operate in a qualitatively different way. This is what we would call swarmware, and it requires an adaptive approach leveraging feedback and adaptation. If you neglect the clockware, you won’t have the foundation to get consistent feedback or execute consistently. If you neglect the swarmware, you won’t be able to deal with the essential complexity of your problem domain. Thus in the end, both are important, the challenge is in identifying where one ends and the other begins.