We all know that bureaucracy is bad, right? The tales of insanely complex, rube-golderg-like processes residing within large organizations are too numerous to cite. Most people universally agree that this type of overhead is a negative thing. More process is a hindrance on human creativity, something that should be avoided at all costs. Indeed, many Agile teams see their first big productivity boost from casting aside the detritus of unnecessary rules, roles, and other organizational straight jackets that were keeping a group of individuals from working as a productive team.
If we accept this truth, then does that mean the goal should be to eliminate all bureaucracy? The process rigor seen in most organizations is excessive, but many view it as a necessary evil. Necessary in order to maintain the scale, performance, or consistency required for there particular organization. With this in mind, managers may agree that organizational structure and rules can become stifling, and yet continue to implement it as a necessity for their organization. Quite possibly, they are right. What if the question isn’t about whether or not we have process, but rather what that bureaucracy should look like? Not all processes and organizational structures are created equally.
Organizational experts looking at the nature of organizational bureaucracy have sought to identify different dimensions for characterizing the nature of a given organization’s bureaucracy. The first dimension is one familiar to us, namely the level of formalized policies and structure within the company. This is a familiar dimension to anyone who’s worked in a couple companies of different sizes and seen how some organizations can have very rigid policies, while others are much more ad hoc and informal. Unfortunately, many people perceive this as the single dimension for managing organizational structure. They incorrectly believe we must choose between limited formal processes or implement a stifling bureaucracy. This is a particular challenge for Agile organizations as they try and figure out how to integrate numerous Agile teams, how to leverage the expertise developed across them, and what to do with the various organizational rules that have accumulated over the years.
That’s where the second critical dimension comes in: the type of formalization. Paul Adler and Bryan Borys have hypothesized that organizational bureaucracy can be characterized along a spectrum between coercive and enabling. Enabling bureaucracy encompasses rules built in an flexible and dynamic manner to help amplify the effectiveness of people. Coercive rules are created to mandate compliance, punish violation, and enforce adherence to certain standards. As these two simple descriptions imply, the type of rules and polices adopted and profoundly impact that impact they have on the organization and those working within the constraints of the system. Let’s take a closer look at the four proposed quadrants (Adler & Borys, 1996)
This is the familiar state of many Agile projects, where there is a low level of formalized processes and procedures, and a focus on personal interactions, collaboration, and commitment. When done well, this quadrant is exemplified by individual Scrum teams. They have minimal hierarchy, but they are potentially limited in their ability to scale. As there is little formal structure, it becomes difficult to foster organizational learning or help ensure adherence to agreed upon standards. This is the sweet spot where many organizations start, but feel they must eventually leave as they grow and mature.
A simple organization structure does not guarantee enablement of individuals within a system. Some organizations may have few explicit polices and rules, but be run by the dictates of a small group of individuals mandating what people should do. There is no constraining policy handbook, but people are far from free to do what they think is best. This quadrant serves as a powerful reminder that lack of structure is neither inherently good or bad.
Moving into the dimension of highly formalized processes, let’s first look at mechanistic bureaucracy. This is where an organization has a high level of process maturity, and that maturity has been implemented in a way to control, limit, and restrain the behavior of individuals towards agreed upon norms. This domain is exemplified by the stereotypical experience people may expect from the Department of Motor Vehicles or the infamous “TPS Reports” from the movie “Office Space”. It is typified by excessive written policies and massive hierarchies of managers, directors, and supervisors.
Last we come to an enabling bureaucracy where rules are established as a means to empower individuals, foster organizational learning and help improve the performance of everyone. Whereas coercive processes are designed to limit behaviors to a small subset of allowed activities, enabling processes are designed to encourage autonomous behavior, reflection, and improvement by everyone within the organization.
The Difference Between Coercive and Enabling Design
Words like enabling and empowering have become such corporate fads that we need to clearly articulate what we mean when we differentiate between a coercive and an enabling process. A coercive process is designed based on the limitations of a person, it is meant to control key decisions and activities in order to ensure consistent results according to standards. When I began my career as a project manager, I was working in an organization that had a large planning spreadsheet. All of the cells were locked except for a handful of inputs where I entered the number of requirements elaborated, the number of pages in the requirements document, – I’m not joking, the number of pages was actually an input – and a number of other similar measures. Based on these inputs, I received a projected budget, risk factor, and timeline for my project. No thinking was required and no discretion was allowed. The formulas rated my project and provided the direct inputs into a go / no go decision. This process required minimal understanding of the domain, no investigative work to learn more about the real customer needs, it simply needed some basic clerical work in order to decide how large and risky a given project would be. It could be run by just about anyone in the organization, and that was the point.
Contrast this planning spreadsheet with the behaviors of a team using lean principles and something like a Kanban system to execute their work. Each step has an agreed upon standard of done, but doesn’t explicitly elaborate practices. People are free to innovate in their delivery methods provided they adhere to the agreed upon definition of done. Work is not assigned to individuals, but rather pulled when people are free. The primary constraint is an agreed upon work in process limit that means people may only work on a finite number of things. While the team follows a defined process, they own this process and can modify it over time based on what they learn. In this example, team members are indeed operating with a significant amount of structure. There are key standards and constraints, however they are defined and managed by the team. Thus, the process moves from a stifling imposition to a collaborative means for a team to experiment and improve. Whereas the coercive process limits individuals to a very finite set of actions and outcomes, an enabling process seeks to help people find new, and better behaviors to be incorporated into the way the organization works.
Characteristics of an Enabling Process
Enabling processes can be defined by four specific attributes in how they operate and are conceived.
- Repair: all systems break down or experience exceptions. The question really is if the system is designed to offer individuals within it the ability to respond to those changes. In the case of software development, this characteristic is inversely related to the granularity of task steps and hand offs. Developers solely focused on building to a spec will be unable to recover from a missing piece of information in a document. They will need to escalate the gap, bring it to another group, have them update the document and then move through another review and sign off. On the other hand, an enabling process might allow for that same developer to begin building while working directly with the analyst who would have created the requirements document so that the two can jointly update the requirements based on any gaps or inconsistencies revealed by the development process. A lack of repair capability leads to another problem. Take the example of my old project planning spreadsheet. One of the data inputs was the number of requirements. Let’s say the business got really ambitious in the way they laid out their requirements. They are asking for a simple data feed, but listed every single field as a specific requirement. Now I obligingly enter that data into the spreadsheet and find this project everyone expected to be a small endeavor will take three years to complete. I am unable to change the formulas, so I am faced with escalating and waiting for a specialist who helped create the spreadsheet to come help me, or I undermine the system by taking liberties with the data inputs until I get the answer I was looking for in the first place. One option introduces massive delays, and the other quietly undermines the purpose of the policy to add consistency and historical trends to our estimation and planning process. Enabling processes allow for these inevitability and have capacity for those people using them to make judgments and adjust to the reality unfolding around them.
- Internal Transparency: in situations where we have defined processes to prevent people from thinking, they don’t need to understand how the system is working. Individual actors don’t need to understand, because their behaviors are completely mandated by a defined process. When implementing an enabling process, individuals now have a much broader range of activities, decisions and options available to them. In order to make correct decisions, they must be aware of what’s going on. I have spoken to several colleagues who have worked at an organization so strongly functionally silo-ed that a group of specialists would come in to estimate a project as part of a toll-gate phase. They would look at preliminary requirements and offer an estimate just in time to move on to the next project. They never got to see how those estimates were used, what other projects they might be interacting with, or what the final result was. Thus, individuals were unable to learn and improve. Those producing the estimates did so in a vacuum. An opposite example would be the daily stand up or Scrum in an Agile team where the point is internal transparency. In fact, this is an excellent test of a team. Team members that are not really enabled will give their updates focused on activity (“I worked on this, this and this”) and most likely offer it to someone in a position of authority, as if they are justifying the time they worked. Enabled team members will talk to one another and their dialog will be much more focused upon when a deliverable may be finished and available to others or problems they are facing that they could use help to resolve. The point of this type of meeting is to offer everyone transparency into what is going on within the team so that people can make informed decisions.
- Global Transparency: fish in a school swim with no singular source of coordination, nor do they have a consistent set of rules for moving as a group such as “everyone passes on the left”. Fish have lateral strips of highly sensitive acoustic nerve cells allowing them to sense the fish swimming around them. This is how hundreds and thousands of individual fish can swim in perfect concert together. In organizations, we would call this global transparency – the ability of people outside of a team, project or system to understand what’s going on inside it in order to properly align and coordinate their own activities. This may be something like a “Scrum of Scrums” used by Scrum teams to coordinate activity across teams, or even an effective use of information radiators and big visible charts for people passing by a team room to see what they are working on and what’s going on.
- Flexibility: as I type this blog, I mis-spelled a word several sentences ago. The computer I use features basic spell checking functionality. It knows that I messed up, but rather than simply make the correction automatically, or prevent me from moving on to another word, it shows the word with a red underline. This is a powerful visual indicator allowing me to see that something appears amiss. In this case, I see the clearly incorrectly spelled word, click on it to see a couple suggestions and pick the right option. This is an excellent case where a system helped improve my capabilities without becoming an imposition or causing me to completely give up on trying to spell. In fact, the typo may have been correct. Perhaps it was an acronym for a new project I was working on, and while the word looks like a typo, it actually is something very meaningful to me. In that case, I could have selected and option to add this word to my own dictionary and never have the the application tell me it was a typo again. Or perhaps it’s only clear that the word is not spelled properly, but there are several options that would make fine semantical sense in that case. We can see a similar dynamic in the set up of Agile teams, where they are free to choose from a broad array of tools and techniques. In their better incarnations, this tailoring approach allows a team to adjust a fairly standard approach or method based on circumstances. Flexibility within a process allows for individuals and teams to be steered in the direction of good practices, but offers them the ability to adjust based on their specific circumstances.
Applying Enabling Bureaucracy to Organizational Processes
Large organizations can not function without bureaucracy, and Agile software development and project management do not change that fact. There is a grain of truth to the reluctant manager who views processes and policies as a necessary evil. However, carefully crafting policies to help empower people allows organizations to get the necessary scale and organizational learning without undermining the agility and opportunities for improvement we see in Agile organizations. Let’s take the example of a standardized work and look at how it would be implemented within an enabling and coercive manner. Standard work was first implemented by Frederick Taylor’s scientific management in an incredibly coercive manner. Efficiency experts would come in, measure the time it took people to do different pieces of work, mandate the best practice and then measure against the expectation for that specific practice. Employees had no discretion about how to do the work, they had minimal input to the standard and they were in no way able to offer feedback. They were simply measured by whether or not they were doing work according to the standard for their specific step. Chances are, they also worked in massive batch size, so they only saw the production of their piece of work in large orders.
Now let’s fast forward to modern manufacturing practices exemplified by Toyota Production Systems and lean thinking. In these environments, teams and individuals continue to use standard work. However, the employees doing the work are the ones who own and maintain the standard, and they change it a lot. At Toyota, for example, they average 1 recommendation per employee per week (Liker, 2004). They go to great lengths to visualize the process using things like kanban systems, and employees have access to an andon-line allowing them to stop the production line and make a correction if they see a defect enter the system. The system is setup for improvement and optimization, and is not unlike the scientific method, where we must control for almost all variables in order to test individual ones. In both examples, we have strict rules and discipline. In the first, employees are treated as potential liabilities who must be managed to ensure they don’t work improperly. In the second, employees are treated as capable experts and the process is there to help them maintain institutional knowledge and build upon it. The difference in outcomes is profound.
Recommendations for Building Enabling Processes
Adler and Borys, whose research seems to be entirely independent from the Agile community, end up making recommendations that seem to fit very closely with how Agile practitioners would recommend running a project. When implementing a process, system or policies they recommend four guiding principles:
- Keep a continual focus on the users of the process
- Maintain an integrated view of the usability of the process and how it will interact with other systems
- Engage users for early and continual testing
- Implement a policy using iterative design while striving for continual improvement.
I find it incredibly validating the convergent nature of this research with the principles espoused in Agile software development. As Agile has grown throughout organizations, it only succeeds when groups are able to internalize the principles and figure out how to authentically implement them within their own domain. I hope concepts like these will help managers and bureaucrats within large organizations see how they should be playing a role in an Agile transformation. It is not that the goal is to eliminate all process and overhead, but rather restructure it to enable employees instead of coerce them.