Agile Risk Management: The Problematic Business Case
In this post we will discuss how business case decisions are made and how mistaken business case decisions introduce errors that traditional risk management techniques are not well-equipped to fix.
To briefly summarize the discussion in my last post on Agile Risk management, we discussed how the relationship between risk and uncertainty is analogous to the relationship between signal and noise. We profiled three “project signal” to “background uncertainty” scenarios:
|When we know so little and/or uncertainty is so high
that we cannot begin to formulate a project risk profile,
there is no discernible project opportunity to consider.
|When the project signal is strong and well-defined, while
background uncertainty is relatively weak, we know enough
to confidently calculate the cost benefit trade-off of proceeding
with the project.
Know enough to be
|This is that interesting set of circumstances, when a project
signal appears to be emerging from background uncertainty.
This may be an opening window of opportunity or it might
be a projection of wishful thinking. When confronted with
this intermediate state, we have the most difficulty deciding
whether or not to proceed with a project and therefore the risk
of making a wrong decision is highest. Let’s delve a bit into
what we mean by wrong decision.
The errors of over-confidence and under-confidence
In the following picture, the green upper hemisphere represents the region of a problem space where there is enough information available (in principle) to effectively calculate risk. The orange lower hemisphere represents the region where uncertainty dominates because too little is known or knowable:
The plaid pattern illustrates where we believe we can effectively calculate risk and the spots pattern is where we believe uncertainty is too high to calculate risk. In the circle below, the plaid pattern perfectly corresponds to the upper green hemisphere and the spots pattern perfectly corresponds to the orange hemisphere. This represents the ideal set of circumstances, where our perception of risk and uncertainty perfectly corresponds to reality:
In the next circle, the plaid pattern extends beyond the green hemisphere on the orange hemisphere (the plaid pie wedge). This represents when we believe we can effectively manage risk when we really cannot. This is the risk of over-confidence (AKA, false positive). The risk of over-confidence is particularly treacherous because we don’t believe it exists:
The spotted pie wedge in the fourth circle represents when we believe we can’t manage risk when in theory we could. This is the risk of under-confidence (AKA, false negative). The risk of under-confidence is measured in the cost of lost opportunities:
Although the risks of over- and under-confidence are always present, they have the greatest magnitude and can do us the most harm when we are in that half-in/half-out state of “knowing enough to be dangerous”. It would be nice if we could safely avoid these inherently hazardous decisions, but unfortunately that is not possible. Since “no-brainer” decisions are no-brainers, they are quickly commodified by the competitive marketplace. The result is that the low-hanging fruit is quickly picked and market conditions change so that the “no-brainer” opportunity is no longer available. Similarly, on the other extreme, when we are “clueless” no opportunity is perceivable; therefore there is no potential value to pursue. Consequently, we are frequently forced to make decisions when we know only enough to be dangerous. This is also the space where innovation happens. Whether driven by the mother of invention or boldness of action, innovation takes courage. As Bob Parsons (either you love him or hate him) says, “Not much happens of any significance if we stay in our comfort zone”. Business case analysis exists to help us navigate this treacherous yet potentially fertile ground.
The business case decision
The purpose of the business case analysis is to make a management decision on whether or not to proceed with an initiative. We can break the business case analysis into three steps:
- Identify the business case
- Evaluate the business case
- Codify the business case
STEP 1: Identify the business case
We start by asking the question, do I have sufficient knowledge to formulate a coherent business case?
As we have discussed, sufficient knowledge allows us to formulate a project risk profile that includes both information about “project signals” and “background uncertainty”.
The project signal depends on information about:
1. The desired end. (A clearly defined project outcome) and
2. The necessary means. (A clear understanding of what you will need to do to obtain the outcome).
Background uncertainty depends on information about:
3. The project environment. (Confidence that all the necessary supporting conditions will hold over the duration of the project).
The more we know about the project end, the more precise we can be about the necessary means and the stronger the resulting project signal. However, as the signal-to-noise metaphor suggests, what constitutes a strong project signal in one environment may appear weak under conditions of high background uncertainty. For example, a project to build an apartment building, (i.e., the project signal), which may be close to a no-brainer under stable economic conditions may diminish to a “Know enough to be dangerous” state the day after a housing bubble collapse (i.e., high background uncertainty), even though the building plans and costs have not changed.
If we answer “no,” we decide the business case is incoherent. However, if we answer “yes,” we decide that there is sufficient knowledge to formulate a coherent project risk profile and we move on to STEP 2.
STEP 2: Evaluate the business case
We do this by calculating the project risks and rewards. If the probability and magnitude of the expected return is high, we decide that proceeding with the project is a good bet. Since the relevant factors, the key assumptions, and their relationships were validated during STEP 1, deciding the second question is essentially a matter of assigning values to variables and calculating the likely return on investment.
After this quantitative analysis, we make the go/no go decision. If the decision is “go”, we proceed to the next step.
STEP 3: Codify the business case
We do this by forming a project charter. The project charter memorializes the terms of the preceding two steps into a governance framework. Risk management controls are introduced to makes sure that the project does not deviate from project charter assumptions, including terms for success (e.g. schedule, cost, or outcome).
When the project is completed, the project will be judged successful or unsuccessful according to the terms laid out in the project charter.
Deciding STEP1 is based on a judgment call decision and deciding the STEP 2 is based on an analytical decision.
By definition, valid analytical decisions are entirely objective. That means that different people conducting the same analysis will always reach the same conclusion. The results of analytical decisions are based on facts that (at least for purposes of the analysis) are assumed to be proven. Analytical decisions always take some version of the form: given these factual assumptions the following conclusion is inescapable. Judgment calls also depend on fact-based analysis but, unlike analytical decisions, also involve other kinds of inquiries including those that depend on common sense, tacit knowledge, or even intuition. Since judgment calls are partially subjective, different people will often enough draw different conclusions, even when faced with the same set of initial conditions. The difference in decision outcome is caused in large part by the individual differences of the decisions makers. Consequently the authority to make judgment calls in the business world is usually closely held by executives.
When it comes to practical decision-making, analytic decisions almost always depend on prior judgment calls. In our business case decision, the principal purpose of the judgment call in the first step is to decide if there are sufficient grounds to conduct subsequent logical analysis in the second step, the preferred if not minimum basis for making “rational” business decisions. After the business case is proven analytically, it is treated as a self-evident theory of business success that is then instantiated into the project charter.
Let’s consider some business case decision examples in light of our earlier discussion. When we start with a “clueless” project scenario, we would expect it to be relatively easy to conclude in STEP 1 that the business case is insufficient and decision process should be aborted, since a clueless state is by definition incoherent. Similarly when we start our business case decision with a “no-brainer” scenario we expect to quickly pass through STEP 1 and move onto computational evaluation in STEP 2.
The third case is a little different. We start with “know enough to be dangerous” conditions and then make a judgment call that we should treat this condition as either a “no-brainer” condition or a “clueless” condition. In other words, we disambiguate our initial condition by forcing a choice between one of two extremes. The judgment call transforms the state rather than perpetuating it as it did in the previous two cases.
Way back when, when I started this posting series on Agile risk management, we made a distinction between risk and uncertainty. I stated that traditional risk management helps with risk but very little if at all with uncertainty. Let’s look at this question again. As we have discussed, traditional risk management techniques depend on the ability to evaluate the likelihood and potential costs of discrete risks. This is the very risk analysis that takes place in STEP 2 of the business case decision. Since the risk model that is evaluated in STEP 2 is the outcome of STEP 1 it cannot be used to decide STEP 1. This approach to making a business case decision is perfectly fine, as long as no mistaken judgments are made in Step 1. However, when faced with inherently uncertain circumstances (“know enough to be dangerous”), it’s inevitable we will sometimes make mistaken judgments that lead to the errors of over- and under-confidence.
What is worse is that once these mistakes are made, they are very difficult if not impossible to rectify. If the outcome of STEP 1 is to proceed with STEP 2, then the mistaken outcome of the judgment call is codified in the project charter, which by definition is presumed to be true. Once the charter is implemented no one involved in its execution has the authority to change any of its terms. Alternatively, if the outcome of STEP 1 is to mistakenly reject the business case there is no mechanism to reverse this decision.
Well, we might look at our situation and say, OK so there are problems with uncertainty but there is nothing we can do about it, right? There is no better way, right?