The analysis is rigorous. The source is respected. The recommendation is compelling. The decision can still be wrong. A successful pilot may depend on specialist support unavailable at scale. A study from another jurisdiction may assume institutional capabilities your agency lacks. A forecast may remain technically sound while changes in demand undermine its usefulness.
Credibility establishes whether evidence deserves attention. Applicability determines how much decision weight it should carry.
For senior executives, the risk emerges when confidence in the source substitutes for scrutiny of the proposed application.
Examine the Distance Between the Evidence and the Decision
Evidence is produced under particular conditions: a population, operating environment, delivery model, resource level and time period. As those conditions change, the conclusions may become less transferable. Consider a hypothetical agency preparing to expand a pilot that reduced processing times. The pilot used experienced volunteers, a limited caseload and dedicated technical support. The proposed expansion would serve a broader population through regional offices with uneven capacity.
The original findings remain valid. The expected enterprise benefit requires a separate judgment.
World Bank analysis of external validity examines this challenge: findings from selected implementation sites may not reliably predict results elsewhere. For executives, the implication is to examine the conditions enabling success before committing to wider adoption.
Make Applicability Visible in the Brief
Executive summaries compress complexity. They can also remove the qualifications that determine whether a recommendation is sound. A conditional finding becomes an expected benefit. A result achieved under favorable conditions becomes a forecast for routine operations.
Consequential recommendations should make four elements explicit:
- The claim: What does the evidence actually establish?
- The conditions: Which features of the original setting influenced the result?
- The differences: What has changed in the proposed application?
- The exposure: What happens if the expected result does not transfer?
The UK’s AQuA Book distinguishes verification — whether analysis satisfies its specification, from validation — whether it is appropriate for its intended purpose. That distinction should remain visible at the point of executive decision.
Match the Commitment to the Confidence
Uncertainty does not automatically justify delay. It should influence the scale, sequencing and oversight of the commitment. Where evidence is promising but transferability remains uncertain, leaders can authorize staged expansion, test performance under representative conditions, or preserve the ability to adjust before making larger commitments.
Decisions affecting essential services warrant particular attention to uneven impacts. An overall benefit may conceal poorer results for a region or population whose circumstances differ from those studied. For international initiatives, examine differences in legal frameworks, institutional capacity, incentives and public expectations. Similar policy objectives do not establish equivalent delivery conditions.
The OECD’s work on evidence-informed policymaking emphasizes the importance of context and the organizational capacity to assess and use research. Evidence becomes an institutional asset when leaders can judge where it applies and act on that judgment.
Establish When the Evidence Must Be Revisited
Evidence needs conditions for continued reliance. A change in policy, service population, delivery capacity or observed performance may justify reassessment. Assign an owner to recognize these changes and bring their implications back to decision-makers. The intensity of review should reflect the consequences of error. This preserves analytical capacity for the decisions where misplaced confidence would create the greatest harm.
The Next Executive Conversation
Select one major recommendation supported by reused analysis and ask:
What must be comparable for this evidence to support our decision and have we established that it is?
Where the answer is uncertain, adjust the commitment or strengthen the evidence. Executive confidence is strongest when leaders understand what the evidence supports, where its limits lie and how those limits shape the decision.
Dr. Rhonda Farrell is a transformation advisor with decades of experience driving impactful change and strategic growth for DoD, IC, Joint, and commercial agencies and organizations. She has a robust background in digital transformation, organizational development, and process improvement, offering a unique perspective that combines technical expertise with a deep understanding of business dynamics. As a strategy and innovation leader, she aligns with CIO, CTO, CDO, CISO, and Chief of Staff initiatives to identify strategic gaps, realign missions, and re-engineer organizations. Based in Baltimore and a proud US Marine Corps veteran, she brings a disciplined, resilient, and mission-focused approach to her work, enabling organizations to pivot and innovate successfully.
Photo Credit: Tima Miroshnichenko
Leave a Reply
You must be logged in to post a comment.