L4.06: Analytic Standards and Judgements

PrintPrint

Professional Standards—A Core of the Tradecraft

The Analytic Standards of the Intelligence Community are the core of GEOINT's tradecraft. The following is based upon US Intelligence Community (IC) Directive Number 203, Analytic Standards, dated June 21, 2007. The standards articulate a commitment to meet the highest standards of integrity and rigorous analytic thinking, and serve as goals for analysts who strive for excellence in their analytic work. The analytic standards are characterized by:

  1. Objectivity: This standard requires that analysts perform their analytic and informational functions from an unbiased perspective. Analysis should be free of emotional content, give due regard to alternative perspectives and contrary reporting, and acknowledge developments that necessitate adjustments to analytic judgments.
  2. Independence of Political Considerations: Analysts should provide objective assessments informed by available information that are not distorted or altered with the intent of supporting or advocating a particular policy, political viewpoint, or audience.
  3. Timeliness: Analytic products that arrive too late weaken the utility and impact. Analysts will strive to deliver their products in time for them to be actionable by customers. An analyst has a responsibility to be aware of the schedules and requirements of consumers.
  4. Use of All Available Sources of Information: Analysis should be informed by all relevant information that is reasonably available. Where critical gaps exist, analysts should work to develop appropriate sources and access strategies.
  5. Individual Standards of the Analytic Tradecraft. An analytic result (see Table 4.5, below).
Table 4.5: Individual Analytic Standards
Individual Standard Description
Properly describes quality and reliability of underlying sources. Analytic products should accurately characterize the information in the underlying sources and explain which information proved key to analytic judgments and why. Consistent with classification of the product, factors significantly affecting the weighting that the analysis gives to available, relevant information, such as denial and deception, source access, source motivations and bias, age and continued currency of information, or other factors affecting the quality and potential reliability of the information, should be included in the product. When appropriate, analytic products may identify a prospective information strategy to improve the reporting base when significant gaps exist.
Properly caveats and expresses uncertainties or confidence in analytic judgments. Analytic products should indicate both the level of confidence in analytic judgments and explain the basis for ascribing it. Sources of uncertainty—including information gaps and significant contrary reporting—should be noted and linked logically and consistently to confidence levels in judgments. As appropriate, products also should identify indicators that would enhance or reduce confidence or prompt revision of existing judgments.
Properly distinguishes between underlying intelligence and analysts' assumptions and judgments. All assumptions are clearly stated and defined. Assumptions deal with identifying underlying causes and/or behavior of systems, people, organizations, states, or conditions. Assumptions comprise the foundational premises on which the information and logical argumentation build to reach analytic conclusions. Assumptions may also span information gaps that would otherwise inhibit the analysis from reaching defensible judgments. Judgments are defined as logical inferences from the available information or the results of explicit tests of hypotheses. They comprise the conclusions of the analysis. Analytic products should explicitly identify the critical assumptions on which the analysis is based and explain the implications for judgments if those assumptions are incorrect. As appropriate, analytic products should identify indicators that would signal whether assumptions or judgments are more or less likely to be correct.
Incorporates alternative analysis where appropriate. Where appropriate, analytic products should identify and explain the strengths and weaknesses of alternative hypotheses, viewpoints, or outcomes in light of both available information and information gaps. Analytic products should explain how alternatives are linked to key assumptions and/or assess the probability of each alternative. To the extent possible, analysis should incorporate insights from the application of structured analytic technique(s) appropriate to the topic being analyzed and include discussion of key indicators that, if detected, would help clarify which alternative hypothesis, viewpoint, or outcome is more likely or is becoming more likely.
Demonstrates relevance to the domain. Analytic products should provide information and insight on issues relevant to the products' intended consumers and/or provide useful context, warning, or opportunity analysis. The information and insight may be particularly difficult to obtain without extensive expertise. To meet this standard fully, analytic products should examine and explicitly address direct or near-term implications of the information and judgments for the intended audience and/or for appropriate security interests, and, when possible, also examine longer-term implications or identify potential indirect or second-order effects.
Uses logical argumentation. Analytic presentation should facilitate clear understanding of the information and reasoning underlying analytic judgments. Key points should be effectively supported by information or, for more speculative warning or "think pieces," by coherent reasoning. Language and syntax should convey meaning unambiguously. Products should be internally consistent and acknowledge significant supporting and contrary information affecting key judgments. Graphics and images should be readily understandable and should illustrate, support, or summarize key information or analytic judgments.
Exhibits consistency of analysis over time, or highlights changes and explains rationale. Analytic products should deliver a key message that is either consistent with previous production on the topic from the same analytic element or, if the key analytic message has changed, highlight the change and explain its rationale and implications.
Makes accurate judgments and assessments. Analytic elements should apply expertise and logic to make the most accurate judgments and assessments possible given the information available to the analytic element and known information gaps. Where products are estimative, the analysis should anticipate and correctly characterize the impact and significance of key factors affecting outcomes or situations. Accuracy is sometimes difficult to establish and can only be evaluated retrospectively if necessary information is collected and available.

Expressing Analytic Judgments

If you recall, intelligence is not truth. Intelligence is an approximation of truth with some level of confidence. It is necessary for decision makers to know how confident their analysts are in the results of an analysis; however, analysts are often reluctant to state the uncertainties surrounding their work because they may not:

  • Appreciate the value of that information to decision makers.
  • Trust the decision maker to understand any caveats as to their confidence in the work.
  • Expect an honest statement of uncertainty to be rewarded.
  • Know how to express the uncertainty.

An often cited example of the damage that vague statements can have is the US President's Daily Brief (PDB) from August 6, 2001. For example:

  • “Bin Laden since 1997 has wanted to conduct terrorist attacks in the US” (CIA, 2001, para. 1);
  • “Bin Laden implied...that his followers would ‘bring the fighting to America’” (CIA, 2001, para. 1);
  • Bin Laden’s “attacks against...US embassies...in 1998 demonstrate that he prepares operations years in advance and is not deterred by setbacks” (CIA, 2001, para. 6);
  • “FBI information...indicates patterns of suspicious activity in this country consistent with preparations for hijackings or other types of attacks” (CIA, 2001, para. 10);
  • “a call to [the US] Embassy in the UAE in May [said] that a group of Bin Laden supporters was in the US planning attacks with explosives” (CIA, 2001, para. 11).

The PDB briefing did not present the President with a clear estimate of Bin Laden’s likely activities in the coming months. Consider the different message to the decision maker if the above PDB were stated as, “Bin Laden implied...that his followers will probably ‘bring the fighting to America.’” U.S. Joint Publication 2-0, Doctrine for Intelligence Support to Joint Operations suggests terms and expressions to communicate analytic judgments. The terms selected are based on three factors:

  • Number of key assumptions required,
  • the credibility and diversity of sourcing, and
  • the strength of argumentation.

Each factor should be assessed independently and then in concert with the other factors to determine the confidence level. Multiple levels are stated as Low, Moderate, and High. Phrases such as "we judge" or "we assess" are used to call attention to a product's key assessment. Supporting assessments may use likelihood terms or expressions to distinguish them from assumptions or reporting. Table 4.5 shows the guidelines for likeliness terms and the confidence levels with which they correspond.

Terms/Expressions:

  • Will, will not
  • Almost certainly, remote
  • Highly likely, highly unlikely
  • Expect, assert, affirm
Table 4.6: Guidelines for likeliness terms and the corresponding confidence levels (U.S. Joint Publication 2-0, Doctrine for Intelligence Support to Joint Operations)
Low Moderate High
  • Uncorroborated information from good or marginal sources
  • Many assumptions
  • Mostly weak logical inferences, minimal methods application
  • Glaring intelligence gaps exist
  • Partially corroborated information from good sources
  • Several assumptions
  • Mix of strong and weak inferences and methods
  • Minimal intelligence gaps
  • Well corroborated information from proven sources
  • Minimal assumptions
  • Strong logical inferences and methods
  • No or minor intelligence gaps exist
Terms/Expressions:

  • Possible
  • Could, may, might
  • Cannot judge, unclear
Terms/Expressions:

  • Likely, unlikely
  • Probable, improbable
  • Anticipate, appear
Terms/Expressions:

  • Will, will not
  • Almost certainly, remote
  • Highly likely, highly unlikely
  • Expect, assert, affirm