L4.05: Sensemaking

PrintPrint

Challenge Of Geospatial Analysis

Geospatial analysis can be very difficult to do well. The difficulty is cognitive and most frequently not related to an individual's ability to use the tools that Geographic Information Technologies (GIT) provide. GEOINT analysis has two different approaches—the single hypothesis approach and the multiple (or competing) hypotheses approach. The first is most popular in academia; the second is popular in intelligence analysis. Here's a brief comparison:

Table 4.1: Single vs. Multiple Hypothesis
Approaches Description
Single Hypothesis The natural human desire to reach an explanation can, and often does, lead the analyst to an interpretation based on a single hypothesis. Human nature is to trust the hypothesis, and the analyst is now blind to other possibilities. The early hypothesis becomes a tentative theory and then a ruling theory, and the analysis becomes focused on proving the ruling theory. The result is a blindness to evidence that disproves the ruling theory or supports an alternate explanation. Here, the results are left to the chance that the original tentative hypothesis was correct. A variation of this is to test a single working hypothesis for fact-finding that degenerates into a ruling theory.
Multiple Hypotheses The method of multiple hypotheses involves the development of several hypotheses that might explain the focus of the analysis. These hypotheses should be contradictory and compete so most will prove to be false. The development of multiple hypotheses prior to the analysis avoids the trap of the ruling hypothesis and thus makes it more likely that our analysis will lead to meaningful results. The major benefit is that the approach promotes greater thoroughness than an analysis directed toward one hypothesis. This leads to lines of inquiry that we might otherwise overlook, and thus to evidence and insights that might never have been encountered.

Biases

Good geospatial analysis requires you to monitor your mental progress, make changes, and adapt the ways you are thinking. This is self-reflection, self-responsibility, and self-management. Richards Heuer addresses this in his work, the Psychology of Intelligence Analysis. Heuer makes three important points relative to intelligence analysis:

  • Human minds are ill equipped ("poorly wired") to cope effectively with both inherent and induced uncertainty.
  • Increased knowledge of one's own inherent biases tends to be of little assistance to the analyst.
  • Tools and techniques that apply higher levels of critical thinking can substantially improve analysis of complex problems.

The core of his argument is that even though every analyst sees the same piece of information, it is interpreted differently due to a variety of factors. In essence, one's perceptions are molded by factors that are out of human control. These cognitive patterns, or mindsets, are potentially good and bad. On the positive side, they tend to simplify information for the sake of comprehension, but they also bias interpretation. The key risks of mindsets are that:

  • Analysts perceive what they expect to perceive;
  • Once formed, they are resistant to change;
  • New information is assimilated, sometimes erroneously, into existing mental models; and
  • Conflicting information is often dismissed or ignored.

He provides the following series of images to illustrate how poorly we are cognitively equipped to accurately interpret the world.

Question #1: What do you see in figure 4.7 below? 

Three triangles filled with the following phrases; Paris in the the spring, Once in a a lifetime, Bird in the the hand.
Figure 4.7: What do you see in this figure?

Question #2: Look at the drawing of the man in the upper right of Figure 4.8. Are the drawings all of men?

Drawing of a man?
Figure 4.8: Impressions resist change.

Question #3: What do you see in Figure 4.9—an old woman or a young woman? 

Drawing of an old Lady?
Figure 4.9: It is difficult to look at the same information from different perspectives.

Now look to see if you can reorganize the drawing to form a different image of a young woman, if your original perception was of an old woman, or of the old woman if you first perceived the young one.

According to Heuer, and as the above figures illustrate, mental models, mindsets, or cognitive patterns are essentially the analogous images by which people perceive information. Even though every analyst sees the same piece of information, it is interpreted differently due to a variety of factors. In essence, one's perceptions are morphed by a variety of factors that are completely out of the control of the analyst. However, cognitive patterns are critical to allowing individuals to process what otherwise would be an incomprehensible volume of information. Yet, they can cause analysts to overlook, reject, or forget important incoming or missing information that is not in accordance with their assumptions and expectations. Ironically, the experienced analysts may be more susceptible to these mindset problems as a result of their expertise and past success in using time-tested mental models. Since people observe the same information with inherent and different biases, Richards Heuer believes an effective analysis method needs a few safeguards. The analysis method should:

  • Encourage products that clearly show their assumptions and chains of inferences; and
  • Emphasize procedures that expose alternative points of view.

Heuer advocates using Structured Analytic Techniques (SATs) as a means to overcome mindsets.

Structured Analytic Techniques (SAT)

Most people solve geospatial problems intuitively by trial and error. Structured analytic techniques (SAT) are a "box of tools" to help the analyst mitigate one's cognitive limitations and pitfalls. Structured thinking in general, and structured geospatial thinking specifically, is at variance with the way in which the human mind is in the habit of working. Structured analysis is an approach to intelligence analysis with the driving forces behind the use of these techniques being:

  • an increased understanding of cognitive limitations and pitfalls that make intelligence analysis difficult;
  • prominent intelligence failures that have prompted reexamination of how intelligence analysis is generated;
  • policies expecting interagency collaboration; and
  • a desire by policy makers who receive analysis that it be more transparent as to how conclusions were reached.

Taken alone, SATs do not constitute an analytic method for solving geospatial analytic problems. The most distinctive characteristic is that structured techniques help to decompose one's geospatial thinking in a manner that enables it to be reviewed, documented, and critiqued. "A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis" (CIA, 2009) highlights a few of the key structured analytic techniques used in the private sector, academia, and the intelligence profession.

The US Intelligence Community began focusing on structured techniques because analytic failures led to the recognition that it had to do a better job overcoming cognitive limitations, analytic pitfalls, and addressing the problems associated with mindsets. Structured analytic techniques help the mind think more rigorously about an analytic problem. In the geospatial realm, they ensure that our key geospatial assumptions, biases, and cognitive patterns are not just assumed but are considered. The use of these techniques later helps to review the geospatial analysis and identify the cause of any error.

Moreover, structured techniques provide a variety of tools to help reach a conclusion. Even if intuitive and scientific approaches provide the same degree of accuracy; structured techniques have value in that they can be easily used to balance the art and science of GEOINT. Heuer categorized structured techniques by how they help analysts overcome human cognitive limitations. Heuer's grouping is as follows:

  • Decomposition and Visualization: The number of things most people can keep in working memory at one time is seven, plus or minus two. Complexity increases geometrically as the number of variables increases. In other words, it is very difficult to do error-free analysis only in our heads. The two basic tools for coping with complexity in the analysis are to: (1) break things down into their component parts so that we can deal with each part separately, and (2) put all the parts down on paper or a computer screen in some organized manner such as a list, matrix, map, or tree so that we and others can see how they interrelate as we work with them. Many common techniques serve this purpose.
  • Indicators, Signposts, Scenarios: The human mind tends to see what it expects to see and to overlook the unexpected. Change often happens so gradually that we do not see it, or we rationalize it as not being of fundamental importance until it is too obvious to ignore. Identification of indicators, signposts, and scenarios create an awareness that prepares the mind to recognize change.
  • Challenging Mindsets: A simple definition of a mindset is, “a set of expectations through which a human being sees the world.” Our mindset, or mental model of how things normally work in another country, enables us to make assumptions that fill in the gaps when needed evidence is missing or ambiguous. When this set of expectations turns out to be wrong, it often leads to intelligence failure. Techniques for challenging mindsets include re-framing the question in a way that helps break mental blocks, structured confrontation such as devil’s advocacy and structured self-critique such as what we call a key assumption check. In one sense, all structured techniques that are implemented in a small team or group process also serve to question your mindset. Team discussions help us identify and evaluate new evidence or arguments and expose us to diverse perspectives on the existing evidence or arguments.
  • Hypothesis Generation and Testing: “Satisficing” is the tendency to accept the first answer that comes to mind that is “good enough.” This is commonly followed by confirmation bias, which refers to looking at the evidence only from the perspective of whether or not it supports a preconceived answer. These are among the most common causes of intelligence failure. Good analysis requires identifying, considering, and weighing the evidence both for and against all the reasonably possible hypotheses, explanations, or outcomes. Analysis of Competing Hypotheses is one technique for doing this.
  • Group Process Techniques: Just as analytic techniques provide structure to our individual thought processes, they also provide structure to the interaction of analysts within a team or group. Most structured techniques are best used as a collaborative group process because a group is more effective than an individual in generating new ideas, and at least as effective in synthesizing divergent ideas. The structured process helps identify differences in perspective between team or group members, and this is good. The more divergent views available, the stronger the eventual synthesis of these views. The specific techniques listed under this category, such as brainstorming and Delphi, are designed as group processes and can only be implemented in a group.

Others have categorized techniques by their purpose: Diagnostic techniques are primarily aimed at making analytic arguments, assumptions, or intelligence gaps more transparent; Contrarian techniques explicitly challenge current thinking; and, Imaginative thinking techniques aim to develop alternative outcomes. In fact, many of the techniques will do some combination of these functions. These different categories of techniques notwithstanding, the analysts should select the technique that best accomplishes the specific task they set out for themselves. The techniques are not a guarantee of analytic precision or accuracy of judgments; they do improve the usefulness, sophistication, and credibility of intelligence assessments.

Sensemaking

The term “sensemaking” is used as a term to describe an analytic process or method. Sherman Kent, who has been described as "the father of intelligence analysis," is often acknowledged as first proposing an analytic method specifically for intelligence. The essence of Kent’s method was understanding the problem, data collection, hypotheses generation, data evaluation, more data collection, followed by hypotheses generation. Richards Heuer subsequently proposed an ordered eight-step model of “an ideal” analytic method, emphasizing early deliberate generation of hypotheses prior to information acquisition:

  • identifying possible hypotheses,
  • listing evidence for and against each hypothesis,
  • analyzing the evidence, 
  • refining hypotheses,
  • trying to disprove hypotheses,
  • analyzing the sensitivity of critical evidence,
  • reporting conclusions with the relative likelihood of all hypotheses, and
  • identifying milestones that indicate events are taking an unexpected course.

Heuer’s technique has become known as Analysis of Competing Hypothesis (ACH). The technique entails identifying possible hypotheses by brainstorming, listing evidence for and against each, analyzing the evidence and then refining hypotheses, trying to disprove hypotheses, analyzing the sensitivity of critical evidence, reporting conclusions with the relative likelihood of all hypotheses, and identifying milestones that indicate events are taking an unexpected course. The use of brainstorming is critical. The quality of the hypotheses is dependent on the existing knowledge and experience of the analysts since hypotheses generation occurs before additional information acquisition augments the existing knowledge of the problem.

While ACH is widely cited in the intelligence literature as a means for improving analysis, the primary advantage of ACH is a consistent approach for rejection or validation of many potential conclusions. According to David Moore, ACH makes it explicit that evidence may be consistent with more than one hypothesis. "Since the most likely hypothesis is deemed to be the one with the least evidence against it, honest consideration may reveal that an alternative explanation is as likely, or even more likely, than that which is favored. The synthesis of the evidence and the subsequent interpretations in light of multiple hypotheses is also more thorough than when no such formalized method is employed." The ACH matrix might look like:

Table 4.2: ACH Matrix
Evidence Hypothesis 1 Hypothesis .....
Evidence A    
Evidence B    
Evidence C    
........    

An excellent and simple explanation of the ACH approach is found in Structured Analysis of Competing Hypotheses: Improving a Tested Intelligence Methodology by Kristan J. Wheaton and Diane E. McManis.

Analytic Stages

ACH and other problem solving approaches can be said to be applied in three stages. The stages and outputs are:

Table 4.3: Analytic Stages
Stage Description Result
Stage 1: Problem Initiation This stage develops the question. The question focuses on the nature of the geospatial and temporal patterns the analyst is seeking to identify and understand. Many new geospatial analysts struggle to translate the question into the context of spatial concepts. To overcome this, we stress the importance of understanding the general question. The question can be viewed as an active two-way discussion between the client requiring the information and the analyst supplying it. A question that seeks to:
  • Discover significant and previously unavailable geospatial information.
  • Describe a place.
  • Explain why an activity or transaction occurred at a place.
  • Judge the significance of a place.
  • Predict the location of geospatial activities.
Stage 2: Information Foraging The foraging actions are exploring for new information, narrowing the set of items that have been collected, and exploiting items in the narrowed set trade off against one another. Some analysts' work never departs from the foraging loop and simply consists of extracting information and repackaging it without much actual analysis. This stage recognizes that analysts tended to forage for data by beginning with a broad set of data and then proceeded to narrow that set down into successively smaller, higher-precision sets of data (Pirolli,1999). The data and information necessary for sensemaking.
Stage 3: Sensemaking This stage results in the development of a detailed analytic assessment. Sensemaking is the ability to make sense of an ambiguous situation; it is creating situational awareness and understanding in situations of high complexity or uncertainty in order to make decisions. It is "a motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively" (Klein, G., Moon, B. and Hoffman, R.F. 2006). Making sense of sensemaking. IEEE Intelligent Systems, 21(4), 70-73.). An analytic result.

It is difficult for an analyst to appreciate how various SATs and geospatial tools fit together. The following table summarizes this relationship:

Table 4.4: How Various GEOINT Techniques and Tools Fit Together
Stage Possible SAT Example Geospatial Technology Operation
Stage 1: Problem Initiation
  • Decomposition and Visualization
  • Challenging Mindsets
  • Group Process Techniques
  • Geospatial data entry
  • Geospatial data conversion
  • Data validation
  • Geospatial data management
  • Attribute data management
  • Data visualization
Stage 2: Information Foraging
  • Brainstorming
  • Data visualization
  • Geospatial data processing/analysis
Stage 3: Sensemaking
  • Indicators, Signposts, Scenarios
  • Hypothesis Generation and Testing (Analysis of Competing Hypotheses)
  • Challenging Mindsets
  • Output of maps and reports