Close corrective actions after exercises—not at the after-action meeting

Source facts: evaluation should feed a managed improvement process

FEMA’s Homeland Security Exercise and Evaluation Program policy and guidance resources describe a common approach to exercise program management, design, conduct, evaluation, and improvement planning. The model connects observed performance and analysis to corrective actions rather than treating the exercise as complete when participation ends.

NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities, addresses designing, developing, conducting, and evaluating tests, training events, and exercises for information-technology plans and capabilities. It distinguishes activity types and emphasizes using results to improve plans, procedures, and readiness.

These sources do not set one universal deadline, evidence type, risk-acceptance authority, or closure threshold. An organization must define them according to mission, impact, obligation, and resources. An observation can also be misunderstood; evidence and analysis should separate a one-time participant error from a structural process or technology weakness.

DSE recommendation: require verified capability change before closure

Open improvement work while evidence is fresh, then manage it through the same disciplined ownership, change, and risk processes used for production systems.

1. Preserve the exercise record. Capture objectives, scenario boundaries, assumptions, participants and roles, timestamps, injects, decisions, communications, system evidence, workarounds, safety issues, evaluator notes, and whether actions were simulated. Protect sensitive architecture and personnel details appropriately.
2. Separate observation from finding. State what occurred, expected behavior, consequence, contributing conditions, and supporting evidence. Determine whether the issue involved documentation, training, authority, staffing, supplier dependency, technology, data, communications, or the exercise design itself.
3. Write an outcome-based corrective action. Define the capability to be restored or improved, affected scope, accountable owner, sponsor, milestones, resources, dependencies, due date, success measure, evidence required, and retest method. Avoid tasks such as review the plan that do not describe a verifiable result.
4. Integrate change and risk. Link the action to service, asset, project, ticket, policy, risk, budget, vendor, and change records. Assess whether interim controls are needed. If leadership accepts delay or residual risk, record the authority, basis, duration, monitoring, and review trigger.
5. Track blockers and escalation. Review aging, scope changes, missed milestones, dependency conflicts, and repeated findings. Escalate based on impact and overdue status rather than allowing the exercise team to carry problems it cannot fund or authorize.
6. Verify implementation independently. Inspect the changed configuration, procedure, contract, training record, equipment, contact data, or monitoring evidence. Confirm the change reached the full intended scope and did not introduce a new control or continuity gap.
7. Retest and close. Use a focused test or the next suitable exercise to demonstrate the original objective under representative conditions. Record results and residual limitations. Close only when the named authority accepts verification and retest evidence—not when a document was uploaded or a meeting occurred.

Use trend review to keep the program honest. Compare findings across exercises and real incidents by capability, root condition, owner, supplier, location, and age. Repeated workarounds or findings that migrate between teams often indicate an unresolved design or governance problem rather than a training gap.

Close exercise-design findings too. If objectives were unmeasurable, evaluators lacked system evidence, participants received unrealistic information, or the scenario skipped a critical dependency, improve the next exercise. Do not treat a favorable outcome produced by artificial assumptions as proof that the operational capability will work.

Factual boundary: FEMA and NIST offer program guidance; the organization defines ownership, due dates, evidence, retest depth, risk authority, and closure. A finding can reveal risk without proving that a particular remediation is the only or safest solution.

Measure actions open and overdue, median closure time, repeat findings, actions closed without retest, accepted-risk age, dependency delays, and improvement in objective performance. The after-action meeting should start the improvement cycle; verification should end it.

Official references

• FEMA, Homeland Security Exercise and Evaluation Program policy and guidance.
• NIST, SP 800-84: Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities.