NIST SP 800-84 日本語訳

Special Publication 800-84 Sponsored by the Department of Homeland Security

Tim Grance Tamara Nolan Kristin Burke Rich Dudley Gregory White Travis Good

Guide to Test, Training, and Exercise

NIST Special Publication 800-84

原本PDFの抜粋画像

Programs for IT Plans and Capabilities

Recommendations of the National Institute of Standards and Technology

Tim Grance, Tamara Nolan, Kristin Burke, Rich Dudley, Gregory White, Travis Good

Computer Security Division Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899-8930

September 2006

U.S. Department of Commerce

Carlos M. Gutierrez, Secretary

Technology Administration

Robert C. Cresanti, Under Secretary of Commerce for Technology

National Institute of Standards and Technology

William A. Jeffrey, Director

The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations.

National Institute of Standards and Technology Special Publication 800-84 Natl. Inst. Stand. Technol. Spec. Publ. 800-84, 97 pages (September 2006)

Certain commercial entities, equipment, or materials may be identified in this document to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

原本PDFの抜粋画像

Acknowledgements

The authors, Tim Grance of the National Institute of Standards and Technology (NIST); Tamara Nolan, Kristin Burke, and Rich Dudley of Booz Allen Hamilton; and Dr. Gregory White and Travis Good of the University of Texas-San Antonio (UTSA); wish to thank their colleagues who reviewed drafts of this document and contributed to its technical content. The authors would like to acknowledge Joan Hash, Karen Kent, Peter Mell, Matt Scholl, Marianne Swanson, and Mark Wilson of NIST; Dick Broome, Kara Crawley, Courtney Hawkins, Munir Majdalawieh, and Zara Pyatt of Booz Allen Hamilton; and Dwayne Williams of UTSA for their keen and insightful assistance throughout the development of the document. The authors would also like to express their thanks to Glenn Fiedelholtz, Annabelle Lee, and Jeffrey Wright from the National Cyber Security Division of the Department of Homeland Security, as well as representatives from the Department of State and the MITRE Corporation, for their valuable comments and suggestions.

The National Institute of Standards and Technology would also like to express its appreciation and thanks to the Department of Homeland Security for its sponsorship and support of NIST Special Publication 800-84.

Executive Summary

Organizations have information technology (IT) plans in place, such as contingency and computer security incident response plans, so that they can respond to and manage adverse situations involving IT. These plans should be maintained in a state of readiness, which should include having personnel trained to fulfill their roles and responsibilities within a plan, having plans exercised to validate their content, and having systems and system components tested to ensure their operability in an operational environment specified in a plan. These three types of events can be carried out efficiently and effectively through the development and implementation of a test, training, and exercise (TT&E) program. Organizations should consider having such a program in place because tests, training, and exercises are so closely related. For example, exercises and tests offer different ways of identifying deficiencies in IT plans, procedures, and training.

This document provides guidance on designing, developing, conducting, and evaluating TT&E events so that organizations can improve their ability to prepare for, respond to, manage, and recover from adverse events that may affect their missions. The scope of this document is limited to TT&E events for single organizations, as opposed to large-scale events involving multiple organizations, involving internal IT operational procedures for emergencies. This document does not address TT&E for a specific type of IT plan; rather, the TT&E methodology described in this document can be applied to TT&E events built around any IT plan or around an IT emergency-handling capability that is not necessarily documented in a plan, such as computer security incident response.

As part of creating a comprehensive TT&E program, a TT&E plan should be developed that outlines the steps to be taken. The TT&E plan should define the organization’s roadmap for ensuring a viable capability, and outline the organization’s approach to maintaining plans, as well as enhancing and managing the capability. Enhancing emergency plans, policies, and procedures will promote more efficient utilization of capabilities in responding to cyber attacks. In addition, the TT&E plan should identify resource and budget requirements that enable organizations to achieve an effective, proven capability, and provide a schedule for conducting various types of TT&E events. Creating the TT&E program should also involve several other steps, including developing a TT&E policy, identifying roles and responsibilities, and documenting a TT&E event methodology.

The TT&E program should include several types of events to ensure the availability of a wide range of methods for validating various planning elements in the context of cyber incidents. The types of events covered in this guide are as follows:

Organizations should conduct TT&E events periodically; following organizational changes, updates to an IT plan, or the issuance of new TT&E guidance; or as otherwise needed. This assists organizations in ensuring that their IT plans are reasonable, effective, and complete, and that all personnel know what their roles are in the conduct of each IT plan. TT&E event schedules are often dictated in part by organizational requirements. For example, NIST Special Publication 800-53 requires Federal agencies to conduct exercises or tests for their systems’ contingency plans and incident response capabilities at least annually.

The National Institute of Standards and Technology (NIST) developed this document in furtherance of its statutory responsibilities under the Federal Information Security Management Act (FISMA) of 2002, Public Law 107-347.

NIST is responsible for developing standards and guidelines, including minimum requirements, for providing adequate information security for all agency operations and assets; but such standards and guidelines shall not apply to national security systems. This guideline is consistent with the requirements of the Office of Management and Budget (OMB) Circular A-130, Section 8b(3), “Securing Agency Information Systems,” as analyzed in A-130, Appendix IV: Analysis of Key Sections. Supplemental information is provided in A-130, Appendix III.

This guideline has been prepared for use by Federal agencies. It may be used by nongovernmental organizations on a voluntary basis and is not subject to copyright, though attribution is desired.

Nothing in this document should be taken to contradict standards and guidelines made mandatory and binding on Federal agencies by the Secretary of Commerce under statutory authority, nor should these guidelines be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, Director of the OMB, or any other Federal official.

Although it is important to have plans in place to help an organization respond to and manage various situations involving information technology (IT), it is equally important to maintain these plans in a state of readiness. This includes having IT personnel trained to fulfill their roles and responsibilities; having plans exercised to validate their policies and procedures; and having systems tested to ensure their operability. These three types of events can be carried out efficiently and effectively through the development and implementation of a test, training, and exercise (TT&E) program.

This publication seeks to assist organizations in designing, developing, conducting, and evaluating TT&E events in an effort to aid personnel in preparing for adverse situations involving IT. The events are designed to train personnel, exercise IT plans, and test IT systems, so that an organization can maximize its ability to prepare for, respond to, manage, and recover from disasters that may affect its mission. The guide describes the design, development, conduct, and evaluation of events for single organizations, as opposed to large-scale events that may involve multiple organizations. The TT&E methodology described in this document can be applied to TT&E events built around any type of IT-related plan, including, but not limited to, contingency plans (e.g., disaster recovery plans) and computer security incident response plans. The vocabulary related to TT&E varies across organizations; this document provides definitions of the terms most commonly used for TT&E-related activities and teams.

This document has been created for individuals responsible for their organization’s TT&E program. Specifically, the document is designed to assist the IT personnel responsible for designing, developing, conducting, and/or evaluating TT&E events in fulfilling these responsibilities effectively.

The remainder of this document is organized into five major sections. Section 2 contains information on establishing a TT&E program. Specifically, it describes the need for a TT&E program and the steps involved in creating a TT&E program, including developing a TT&E policy; identifying roles, responsibilities, and activities; establishing an event schedule; and documenting the TT&E event methodology.

Section 3 briefly discusses the role of training in a TT&E program and how training is related to exercises and tests. Section 4 contains information on determining the need for tabletop exercises, and designing, developing, conducting, and evaluating an exercise event. This section describes the design phase in detail, including determining the topics and scope; identifying the objectives; identifying participants and training staff; and coordinating logistics. Sections 5 and 6 contain similar information for functional exercises and tests, respectively.

This document also contains several appendices. Appendices A, B, and C contain samples of the documentation associated with tabletop exercises, functional exercises, and tests, respectively. Appendix D contains a glossary, and Appendix E contains an acronym list. Appendix F identifies print and online resources that may be helpful in scoping, planning, documenting, conducting, and evaluating TT&E events. Appendix G contains an index for the publication.

An organization’s IT plans need to be maintained to sustain the organization’s ability to prepare for, respond to, manage, and recover from disasters affecting its mission.² Common types of IT plans used for this purpose are as follows:

The following are the major types of events used to maintain these plans:

Although an organization could perform tests, training, and exercises as discrete activities without any coordination, organizations should consider having a program in place that addresses all three because they are so closely related. For example, exercises and tests offer different ways of identifying problems with IT plans, procedures, and training. An effective TT&E program should comprise a combination of training, exercise, and testing events.⁸ The program should include a TT&E plan, policy, event methodology, and procedures. Using these elements should cause TT&E events to be performed more consistently and effectively, particularly reducing duplication of effort. A program should also address resource and budget requirements, and provide a schedule for conducting types of TT&E events. This section discusses the steps involved in creating a TT&E program.⁹

Regardless of the type of IT plans an organization has developed, it should have mechanisms in place to validate the plans’ effectiveness and manage their maintenance. Organizations that want to establish a TT&E program should first develop a TT&E plan that outlines the steps to be taken to ensure that personnel are trained in their IT plan roles and responsibilities, IT plans are exercised to validate their viability; and IT components or systems are tested to validate their operability in the context of an IT plan. The TT&E plan should outline all elements of the program and ensure that information surrounding the program is documented. In addition to creating the TT&E plan, other major steps in creating a TT&E program are as follows:

These steps are described in more detail in Sections 2.1 through 2.4.

A TT&E program should include a policy that outlines the organization’s internal and external requirements associated with training personnel, exercising plans, and testing components and systems. The policy forms the framework for the purpose and objectives of the program and cites applicable Federal and internal guidance. The policy further provides the framework or “rules” that govern how the organization develops and administers the TT&E events. The policy establishes a clear and consistent framework for creating all of the documentation associated with TT&E events.

Key steps for developing a TT&E policy are as follows:

The following are suggested elements to include in a TT&E policy:

Once the TT&E policy is developed, the policy statement should be updated as new guidance is applied to or impacts the program.

The office with primary oversight of and responsibility for a TT&E program varies based on the structure or requirements of the organization. In many organizations, it is led within the Office of the Chief Information Officer (OCIO). The TT&E program should be managed by a person or team with direct responsibility for the organization’s IT planning capability. The program should have an IT plan coordinator who is responsible for all aspects of IT planning, including the TT&E element of maintaining the IT plans. The IT plan coordinator has overall responsibility for the IT plans, including development, implementation, and maintenance. One of the IT plan coordinator’s responsibilities is to identify a TT&E program coordinator, who is responsible for developing a TT&E plan and coordinating events. To plan and conduct TT&E events, the TT&E program coordinator works with event design teams. Organizations might elect to purchase specialized software or obtain external support to assist in forming or staffing these teams. Sections 4 through 6 contain information on the individual design teams and the roles within each team.

The TT&E plan should document the projected schedule of activities to be performed within the TT&E program. Although events should be conducted as needed, organizations should evaluate the required frequency of its events and document the frequency of each event in a TT&E schedule. For example, NIST Special Publication (SP) 800-53 requires Federal agencies to conduct exercises or tests for their systems’ contingency plans and incident response capabilities at least annually. Sections 4 through 6 provide additional detail on how to evaluate an organization’s specific TT&E needs.

As part of creating a TT&E program, an organization should select and document a high-level methodology for planning and performing TT&E events. Figure 2-1 shows one commonly used methodology, which has four phases:

Although the details of each phase typically vary based on the type of event conducted, the same phases should be used for each event. Details pertaining to each type of event can be found within Sections 4 through 6.

Figure 2-1. TT&E Event Methodology

Organizations should consider having a TT&E program that validates the effectiveness of IT plans such as contingency plans and computer security incident response plans, and manages their maintenance. The TT&E program should include a TT&E plan, policy, and event methodology. Using these elements should cause TT&E events to be performed more consistently and effectively. The TT&E plan should outline all elements of the program and ensure that information surrounding the program is documented. In addition to creating the TT&E plan, other major steps in creating a TT&E program are as follows:

Training is a continuum of learning activities that enables staff to maintain and enhance their skills and technical proficiencies and to remain current with technological advances. For the purpose of this publication, training refers only to informing participants of their roles and responsibilities within a particular IT plan and teaching them skills related to those roles and responsibilities, thereby preparing them for participation in exercises, tests, and actual emergency situations related to that plan.¹⁰ Training events can be instructor-led (e.g., classroom setting, interactive online) or self-study (e.g., paper, online).

The scheduling of training events that support IT plans should be coordinated closely with the schedules of other events in a TT&E program. For example, training sessions typically precede exercises and tests. This ensures that personnel are familiar with their roles and responsibilities within a given IT plan before exercising the plan itself. Another outcome of performing training is identifying areas where additional training might be necessary.

Other NIST publications have already described training programs and events in detail. Refer to NIST SP 800-50, Building an Information Technology Security Awareness and Training Program, and NIST SP 800-16, Information Technology Security Training Requirements: A Role- and Performance-Based Model, for more information on training.¹¹

This page has been left blank intentionally.

Tabletop exercises are discussion-based events where personnel with roles and responsibilities in a particular IT plan meet in a classroom setting or in breakout groups to discuss their roles during an emergency and their responses to a particular emergency situation. Tabletop exercises are conducted in an informal environment, with a facilitator guiding participants through a discussion designed to meet pre-defined objectives. One or more scenarios may be discussed during a single tabletop exercise. The duration of a tabletop exercise (typically two to eight hours) varies depending on the audience, the topic being exercised, and the exercise objectives. Tabletop exercises are cost-effective tools to validate the content of IT plans, such as contingency plans and incident response plans, to ensure the plan content is viable and implementable in an emergency situation.

This section provides guidance on evaluating the need for a tabletop exercise, and designing, developing, conducting, and evaluating a tabletop exercise. The section then summarizes the key elements to consider before, during, and after the conduct of a tabletop exercise. Appendix A provides a sample tabletop exercise facilitator guide, sample participant guide, and sample after action report.

As part of the TT&E program, the program coordinator should routinely determine the need for a tabletop exercise for a particular IT plan by considering the organization’s overall objectives for conducting a tabletop exercise and answering questions such as the following:

Organizations should conduct tabletop exercises periodically; following organizational changes, updates to an IT plan, or the issuance of new TT&E guidance; or as otherwise needed. For each tabletop exercise, the program coordinator should choose a form of tabletop exercise that is well-suited to meeting the identified needs and objectives. The tabletop exercise schedule should be coordinated closely with the schedules of the other events of the TT&E program. The TT&E program coordinator usually ensures that tabletop exercises are scheduled within a reasonable timeframe after a training event so that the personnel participating in the tabletop exercise are recently trained in their roles and responsibilities. It is important that when an exercise is being scheduled, managers are notified and their approval obtained. Ensuring that management has agreed to an exercise is an essential step in the development of the exercise.

Once the need to conduct a tabletop exercise has been established, the TT&E program coordinator should work with the tabletop exercise design team to design the event. The design phase is often the most time-consuming phase of planning a tabletop exercise. Planning is typically started at least three months before the conduct date for large, complex exercises and at least one month in advance for less complex exercises. Sections 4.2.1 through 4.2.6 describe the major steps in the event design process.

The design team should determine the exercise topic based on the focus of the plan being exercised. General topics can include contingency planning and incident response; specific topics range from sustaining essential functions to managing and reporting IT security incidents. For example, disaster recovery plan exercise discussion topics would likely include the roles and responsibilities of personnel with regard to the processes and procedures associated with restoring an organization’s information systems. Incident response plan exercise discussion topics would likely include processes and procedures for managing and reporting IT security incidents.

The scope of the tabletop exercise should be determined based on the target audience. All personnel with responsibilities under the IT plan should participate in exercises; however, senior-level teams and operational-level teams should participate in separate tabletop exercises initially because of their different levels of responsibility. Once these two groups have been exercised individually, both groups should participate in a combined exercise to validate coordination between the groups.

The exercise should apply to the roles and responsibilities of personnel within the IT plan being exercised and focus on validating that the documented roles, responsibilities, and interdependencies are accurate and current. The types of questions asked of the participants during the course of the exercise should be tailored to the level of personnel exercised. Senior-level tabletop exercises typically range from two to four hours, while operational-level tabletop exercises range from two to eight hours. To ensure that the knowledge of the roles and responsibilities identified in the plan being exercised is current, it is often effective to conduct a training session in conjunction with any tabletop exercise lasting more than four hours.

The objectives of any tabletop exercise should be validating the content of the IT plan and related policies and procedures, validating participants’ roles and responsibilities as documented in the plan, and validating the interdependencies documented in the plan. An additional objective for some exercises is meeting regulatory and other such requirements associated with exercising plans, such as the requirement in NIST SP 800-53 for Federal agencies to conduct exercises or tests for their systems’ contingency plans at least annually.

Based on the topic, scope, and objectives of the exercise, the design team determines who should participate in the event. 13 The participants should be comprised of the personnel with roles and responsibilities identified in the plan to help ensure the exercise meets its stated objectives. For example, senior-level personnel should be invited to participate if the primary exercise objective is to validate the decision-making and oversight processes within the plan. If the primary objective is to validate operational procedures, operational-level personnel should be invited to the exercise. If both groups have participated in previous tabletop exercises separately, it is appropriate to conduct a combined session, where senior-level and operational-level personnel discuss individual and team roles and responsibilities and coordination requirements. Once the appropriate participants have been identified, they should receive a written invitation or announcement of the exercise as soon as possible. This is typically accomplished in the form of an e-mail or memorandum by a member of the tabletop exercise design team, but, if more appropriate, may instead be distributed by a member of management.

The design team usually designates an exercise facilitator, who leads the discussion among the exercise participants, and a data collector, who records information about the actions that occur during the exercise. The facilitator and the data collector should be thoroughly familiar with the content of the IT plan being exercised and with the exercise objectives. The facilitator and data collector should meet before the event to discuss the details surrounding the exercise, including its scope and objectives. At this time, the facilitator and the data collector review the results from previous tabletop exercises, if applicable, to heighten their awareness of potential issues before the event.

One person on the design team should typically be responsible for coordinating the exercise event’s logistics. The logistics coordinator usually begins to do this at least one month before the conduct of the tabletop exercise. The checklist in Table 4-1 can be used as a starting point by the logistics coordinator to ensure the necessary tasks are completed.

Table 4-1. Sample Logistics Checklist for Tabletop Exercise Events

Once the event is designed, the design team should assign roles and responsibilities to its members to develop the tabletop exercise material. Tabletop exercises typically include the following documentation:

The types of questions documented in the facilitator guide should be tailored to the participants. For example, if senior-level personnel are the participants, the questions should be of a more general, high-level nature and focus on decision-making and oversight, which are consistent with their roles and responsibilities within the plan. If operational personnel are the participants, the questions should typically be focused on specific procedures and processes that are followed to carry out roles and responsibilities.

Sample tabletop exercise documentation is located in Appendix A.

A common misconception is that scenarios must be very detailed to be effective. Actually, it is often more effective to develop a short, concise scenario. During tabletop exercises with long, detailed scenarios, participants often spend more time dissecting the scenario and discussing its content than they spend on meeting the objectives of the exercise. If a detailed scenario is desired, a trusted agent with detailed knowledge of the plan and all the procedures documented within the plan should aid in the development of the scenario to ensure accuracy. In addition, the facilitator should have the ability to redirect the participants’ focus from the scenario to the objectives, should they begin focusing too much on the content of the scenario.

Tabletop exercises are usually conducted in a classroom-type setting. This permits a facilitator to address each individual or the participants as a group while facilitating the exercise. This also fosters communication among the participants, as does placing a name tent on the table for each participant before the start of the exercise. This is particularly important if participants and teams work within different operational areas of the organization. Participants are usually not seated with their teammates to encourage independent thought processes and provide exposure to other operational areas. A copy of the participant guide should be placed with each name tent.¹⁵

At the start of the exercise, the facilitator welcomes the participants to the event and request that the participants introduce themselves by name and give a general description of their roles within the organization. The facilitator then projects the briefing and discusses the scope of the exercise and logistics information. The facilitator then walks participants through the scenario and kicks off the discussion with one of the discussion questions documented in the facilitator guide, designed to prompt decision-making or coordination among participants.¹⁶ Following the kickoff, the discussion occurs naturally among participants based on the scenario and the objectives. The facilitator may inject periodic questions from the facilitator guide. If the discussion does not occur naturally, the facilitator should prompt discussion by asking additional questions from the facilitator guide until all objectives are met. During the course of the exercise, the data collector should record observations to be included in the after action report.

Immediately following the facilitated discussion, the facilitator and data collector should conduct an exercise debrief, often referred to as a hotwash. During the debrief, the facilitator asks participants in which areas they felt they excelled, in which areas they could use additional training, and which areas of the plan should be updated.

The comments that surface during the debrief, along with lessons learned documented by the data collector during the exercise, should be captured in the after action report. The introduction to the after action report should describe background information about the exercise such as purpose, objectives, participants, and the scenario. The after action report should also contain documented observations made by the facilitator and data collector during the exercise and recommendations for enhancing the IT plan that was exercised.

Following the development of the after action report, the plan coordinator might assign action items to select personnel to update the IT plan being exercised. The plan coordinator should then update the plan, if appropriate, by implementing recommendations made in the after action report. It may also be necessary to brief certain managers on the results of the exercise, update other security-related documents, and perform other actions based on the exercise.

Tabletop exercises are discussion-based events where personnel with roles and responsibilities in a particular IT plan meet in a classroom setting or in breakout groups to discuss their roles during an emergency and their responses to a particular emergency situation. Tabletop exercises are conducted in an informal environment, with a facilitator guiding participants through a discussion designed to meet pre-defined objectives. One commonly used methodology for planning and performing tabletop exercise events has the following phases:

Functional exercises allow personnel with operational responsibilities to validate their IT plans and their operational readiness for emergencies by performing their duties in a simulated operational environment. Activities for a functional exercise are scenario-driven, such as a particular building’s IT systems becoming unavailable in the simulated environment and the participants then learning that the building is on fire. Additional situations are often simulated during the course of the exercise. Functional exercises are designed to exercise specific team members, procedures, and assets involved in one or more functional aspects of an IT plan (e.g., communications, emergency notifications, IT equipment set-up). Functional exercises vary in complexity and scope, from validating specific aspects of a plan to full-scale exercises that address all plan elements. The duration of functional exercises typically lasts from between several hours to several days, depending on the event’s objectives and the complexity of the plan being exercised.

This section provides guidance on evaluating the need for a functional exercise, and designing, developing, conducting, and evaluating a functional exercise. The section then summarizes the key elements to consider before, during, and after the conduct of a functional exercise. Appendix B provides functional exercises samples, including a scenario, a tracking form, and an after action report.

As part of the TT&E program, the program coordinator should routinely determine the need for a functional exercise for a particular IT plan by considering the organization’s overall objectives for conducting a functional exercise and answering questions such as the following:

Organizations should conduct functional exercises periodically; following organizational changes, updates to an IT plan, or the issuance of new TT&E guidance; or as otherwise needed. It is usually best to ensure adequate staff training and tabletop exercises have taken place before engaging in a functional exercise. The functional exercise schedule should be coordinated closely with the schedules of the other events of the TT&E program. The TT&E program coordinator usually ensures that functional exercises are scheduled within a reasonable timeframe after a tabletop exercise event. It is important that when an exercise is being scheduled, that managers are notified and their approval obtained. Ensuring that management has agreed to an exercise is an essential step in the development of the exercise.

Once the need to conduct a functional exercise has been established, the TT&E program coordinator should work with the functional exercise design team to design the functional exercise event. The team is comprised of personnel who are familiar with the plan’s content and can facilitate the exercise design process. The design phase of a functional exercise is usually started at least a few months before the desired conduct date, depending on the complexity of the exercise. Sections 5.2.1 through 5.2.6 describe the major steps in the event design process.

The design team should determine the overarching objectives for exercising the IT plan (e.g., as part of a strategic long-term plan, in response to ad hoc requirements). These broad objectives represent the topic areas that will be addressed in the exercise. The topic areas chosen will depend on whether the exercise will address the full plan or specific aspects of the plan. Topic areas addressing the full plan could include (but are not limited to) validating the plan’s procedures, evaluating an organization’s ability to implement the plan, and assessing interdependencies of organizations and personnel responsible for carrying out the plan. Examples of topic areas that are more narrowly focused on specific aspects of the plan are assessing the plan’s alert and notification process, validating personnel responsibilities associated with the operational phase of the plan, and evaluating the processes involved in resuming normal operations.

The scope of the functional exercise should be determined based on which portions of the IT plan (or all of it) should be exercised.¹⁷ If only portions of the plan are to be exercised, the design team should consider examining a specific phase of plan implementation, such as activation, operation, or reconstitution, or specific functions.

When determining the scope of a functional exercise, the design team should clearly identify the specific element or elements of the IT plan that will be assessed and consider the types of participants necessary to carry out the exercise. Ultimately, a robust TT&E program ensures that all elements of a plan are exercised; however, the emphasis of initial functional exercises is often placed on operational-level team roles and responsibilities. As an organization’s TT&E program matures, senior-level participants can also engage in functional exercises to fully validate decision-making aspects of the plan.

The objectives of any functional exercise should be validating the content of the IT plan, validating participants’ roles and responsibilities as documented in the plan, validating the interdependencies documented in the plan, and providing an opportunity for participants to get hands-on practice in executing their functions. An additional objective for some exercises is meeting regulatory and other such requirements associated with exercising plans, such as the requirement in NIST SP 800-53 for Federal agencies to conduct exercises or tests for their systems’ contingency plans at least annually. Specific objectives should be documented and clearly articulated to exercise participants.

Based on the topic, scope, and objectives of the exercise, the design team determines who should participate in the event.¹⁸ The participants should be comprised of the personnel with roles and responsibilities under the plan that will be needed to help ensure the exercise meets its stated objectives. For example, senior-level personnel should be invited to participate if the primary exercise objective is to validate the decision-making and oversight processes within the plan. If the primary objective is to validate operational procedures, operational-level personnel should be invited to the exercise. Finally, if the primary objective is to validate the full-scale readiness of a plan, both senior-level personnel and operational-level personnel should participate. Once the appropriate participants have been identified, they should receive a written invitation or announcement of the exercise as soon as possible. This is typically accomplished in the form of an e-mail or memorandum by a member of the functional exercise design team, but, if more appropriate, may instead be distributed by a member of management.

The design team usually designates an exercise director, who is responsible for all aspects of the exercise, including staffing, development, conduct, and logistics. The exercise director designates one or more controllers, who monitor, manage, and control exercise activity; data collectors, who record information about the actions that occur during the exercise; and simulators, who simulate or otherwise represent non-participating individuals and organizations whose input is necessary to the flow of the exercise. The controllers, data collectors, and simulators should be thoroughly familiar with the content of the IT plan being exercised and with the exercise objectives.

The exercise director, controllers, data collectors, and simulators should meet before the event to discuss the details surrounding the exercise, including its scope and objectives. At this time, the exercise director, controllers, data collectors, and simulators review the results from previous tabletop and functional exercises, if applicable, to heighten their awareness of potential issues before the event.

One or more members of the design team should typically be responsible for coordinating the exercise event’s logistics. The logistics coordinator(s) typically begin to do this approximately three months before the conduct of the functional exercise. The checklist in Table 5-1 can be used as a starting point by the logistics coordinator(s) to ensure the necessary tasks are completed.

Table 5-1. Sample Logistics Checklist for Functional Exercise Events

Once the event is designed, the exercise director should assign roles and responsibilities to the team to develop the functional exercise material. Functional exercises typically include the following documentation:

Sample functional exercise documentation is located in Appendix B.

In addition to the members of the design team discussed previously, functional exercise material development might require assistance from other individuals. For example, a trusted agent who has detailed knowledge of the IT plan and the associated procedures could aid in the development of the scenario, MSEL, and message injects to ensure accuracy. As described in Section 4.3, it is often most effective to have a short, concise scenario so that participants do not focus on critiquing the scenario itself.

Functional exercises are typically conducted in real or near-real time and prompt participants to carry out their roles and responsibilities as realistically as possible. A functional exercise is often initiated by a telephone call or other appropriate means, alerting selected personnel of the implementation or activation of a specific IT plan. This alert prompts further notification of all personnel who would be notified via the means identified in the plan. Once the notification process is completed, participants are expected to carry out operational or decision-making activities documented in the plan. Depending on the scope of the exercise, activities could range from implementing notification procedures to deploying to an alternate site to mobilizing resources, including staff and equipment. The exercise scope dictates whether deployments or mobilizations are simulated or if they actually occur. Regardless of location, the participants should carry out the activities as they would according to the plan being exercised. Participants should be informed of any exercise artificialities during the participant briefing.

Controllers, data collectors, and simulators should be pre-positioned at the location where the exercise takes place. Controllers form a control cell—a central location for exercise coordination, typically in a separate area from the exercise participants—from which the controllers introduce the scenario and message injects to participants. Controllers administer the exercise by referring to the message inject tracking form and MSEL to ensure the exercise remains on schedule and within scope.

Data collectors directly observe player actions during the exercise. They refer to the evaluation criteria and any other evaluation forms that the data collection team may create to aid their efforts. Simulators assume the roles of various internal and external entities that are not participating in the event, such as other government organizations, private citizens, or law enforcement. Information provided by simulators should be delivered in accordance with how it would be provided by the organization(s) being simulated. They coordinate closely with the controllers and exercise director to ensure their responses are consistent with the MSEL. Simulators may be collocated with controllers or assemble a response cell in a separate room. During the course of exercise, the exercise director, controllers, data collectors, and simulators should remain in constant contact with each other to ensure that the exercise remains coordinated and on schedule.

The exercise director announces when the exercise concludes. Typically, this occurs when the time allocated for the exercise has ended, or earlier if all objectives have been met or the MSEL and injects have been fully played out. In cases where a real-world emergency occurs, it is the exercise director’s responsibility to call an immediate end to the event. Following the conclusion of exercise play, the exercise director, controllers, and data collectors should conduct an exercise debrief with participants, often referred to as a hotwash. The exercise director leads the hotwash and requests feedback from participants, controllers, simulators, and data collectors. Immediately following the exercise, the controllers, data collectors, simulators, and participants should be asked to provide the exercise director with their notes or any forms completed during the course of the exercise and the hotwash session.

During the evaluation phase, the exercise director relies on the design team or other specified staff to develop the after action report that documents findings and recommendations from the functional exercise. Exercise notes, forms, and other material created during the course of exercise play and during the hotwash are the basis of the after action report. The introduction to the after action report should document background information about the exercise such as the scope, objectives, and scenario. The after action report should also document observations made by the exercise staff and participants during the exercise and recommendations for enhancing the IT plan that was exercised. The after action report should also include a list of exercise participants and may provide information from any participant surveys that were distributed during the hotwash to solicit feedback.

Following the development of the after action report, the plan coordinator might assign action items to select personnel in an effort to update the IT plan being exercised. The plan coordinator should then update the plan, if appropriate, by implementing recommendations made in the after action report. It may also be necessary to brief certain managers on the results of the exercise, update other security-related documents, and perform other actions based on the exercise.

Functional exercises allow personnel with operational responsibilities to validate their IT plans and their operational readiness for emergencies in a simulated operational environment. Activities for a functional exercise are scenario-driven, such as a particular building’s IT systems becoming unavailable in the simulated environment and the participants then learning that the building is on fire. Additional situations are often simulated during the course of the exercise. Functional exercises are designed to exercise specific team members, procedures, and assets involved in one or more functional aspects of an IT plan. Functional exercises vary in complexity and scope, from validating specific aspects of a plan to full-scale exercises that address all plan elements.

One commonly used methodology for planning and performing functional exercises has the following phases:

This page has been left blank intentionally.

Tests are evaluation tools that use quantifiable metrics or expected outcomes to validate the operability of one or more IT systems or system components (e.g., operating system, application, pager, Blackberry) that are identified as critical in an IT plan.²⁰ Tests can take several forms, including the following:

A test is conducted in as close to an operational environment as possible, which means that the test should be conducted in a manner that resembles the everyday work environment in which the system or component is found. If feasible, an actual test of the components or systems used to conduct daily operations for the organization should be used. Tests can potentially be disruptive to an organization’s operations, so tests are sometimes performed on systems that mimic the actual operational systems, especially if there is not strong confidence that the tests will be completely successful.

This section provides guidance on evaluating the need for testing; creating a test plan; and designing, developing, conducting, and evaluating a test. The section then summarizes the key elements to consider during a test and after conducting a test. Appendix C provides test documentation examples, including a test plan, a test briefing, test validation and evaluation worksheets, and an after action report.

As part of the TT&E program, the program coordinator should routinely determine the need for a test by considering the organization’s overall objectives for conducting a test and answering questions such as the following:

Tests are usually conducted after personnel have been trained on the use of the systems or components being tested and before the systems or components become operational, to ensure they do not adversely affect the security posture or other operational aspects of the organization. If personnel have not yet been trained, system testing should be delayed until the training has been completed. After operational use has begun, periodic testing should be conducted to ensure the continued proper and secure use of the systems or components. Comprehensive tests should also be scheduled periodically to ensure that the IT plans are reasonable, effective, and complete, and that personnel know what their roles and responsibilities are in the conduct of the plans. High personnel turnover might necessitate more frequent testing to maintain the level of preparedness the organization requires.

The scheduling of tests should also consider factors such as available resources and the potential impact on the organization. It is important that when a test is being scheduled, that senior managers are notified and the potential impact on operations assessed to determine the best time to conduct the test. For example, conducting a test that might affect the operations of the organization might not be wise during known peak operational periods. Although scheduling comprehensive tests when many employees are on vacation or on holiday breaks might have a minimal impact on operations, it also could limit the number of available personnel. Ensuring that senior leadership in the organization has agreed to the test, especially for comprehensive tests, is an essential step in the development of the test.

Testing schedules might also be affected by factors external to the organization itself. For example, compliance and regulatory issues might dictate that certain tests be performed on a periodic basis. Another example is environmental and safety issues; a test requiring that staff relocate operations from one facility to another might be better accomplished when the weather is not unpredictable so that employees do not have to travel under extreme conditions, such as a blizzard.

Once the need to conduct a test has been established, the TT&E program coordinator should create a test design team²¹ to design each specific test. Several factors can have a significant bearing on the design of the test, including the level of the test (component, system, or comprehensive), the organizational entities involved, and the scope of the test. These factors can affect the lead time required to develop the test, the level of complexity for the test, and the length of time the test will take. At an early stage in the design process, the personnel who will participate in the test should be identified and the senior managers for these affected areas should be contacted. Sections 6.2.1 through 6.2.6 describe the major steps in the event design process.

The scope of the test should be determined based on current system or security requirements and any potential compliance or regulatory requirements. The scope of the test is directly shaped by the type of test. Component tests are more focused and generally involve fewer individuals and organizational entities. System tests are broader in scope and include more personnel and multiple components.

Comprehensive tests involve much larger portions of the organization, potentially all personnel, and require more extensive coordination and planning.

The design team should define the tests that will be conducted and specify the expected results or outcomes. The test plan could consist of a series of smaller individual tests each designed to examine a part of the component, system, or group of components and systems being tested. The objectives for each test should be to measure, check, or verify whether the component, system, or group of components and systems satisfies its intended purpose and functions adequately. Where possible, the expected results or outcomes should be expressed in an objective and measurable manner, with subjective measurements being minimized. The results should be quantifiable and repeatable to the extent reasonably possible.

Tests are often performed as part of standard operational activities, such as restoring a backup, moving a server from one room to another, upgrading or patching operating systems or applications, or changing hardware components (e.g., swapping hard drives, replacing a failed power supply). Combining tests with operational activities is generally more efficient than performing them separately and is also less likely to negatively impact operations.

Other examples of common tests are activating call tree cascades and determining if they can be executed within prescribed time limits, and removing power from a system or system component.

The design team should specify the assessment tools and procedures needed to accomplish the test. The specific tools needed may vary greatly depending upon the scope of the test. Tools might range from specialized software or hardware tools (e.g., network sniffers, vulnerability scanners) to measurement and recording devices (e.g., stopwatches, cameras, video recorders) to checklists used to measure adherence to defined processes and procedures. Tools might also include items needed by the test team for logistical support (e.g., radios, cell phones, badges).

The participants in a test vary based on the scope of the test to be performed. Participation in testing events can be thought of in two levels:

The design team should attempt to identify both levels of participants, although for larger tests the affected individuals might have to be identified in groups instead of individually. Individuals in the first level of participants should be notified well in advance of the test through an e-mail or memorandum. Individuals and groups in the second level of participants should be notified before the test occurs; examples of this include an announcement to the participants of possible disruptions to the systems being tested and a message on a help desk phone number stating that disruptions may occur to the systems being tested between certain hours.

The design team usually designates a test director, who is responsible for all aspects of the test, including staffing, development, conduct, logistics, and oversight of the design team. The test director designates one or more data collectors, who monitor and record the results of the test. The test director and data collectors should meet before the event to discuss the details surrounding the test, including its scope and objectives. At this time, the test director and data collectors review the results from previous exercises and tests, if applicable, to heighten their awareness of potential issues before the event.

The design team also often includes one or more individuals that are subject matter experts in the areas being tested. These individuals can help develop the test plan and identify the necessary testing tools. Because they are aware of the details of the test, these individuals should not be participants in the test; instead, they can be test observers, facilitators, data collectors, or controllers.

The design team should begin to coordinate the logistical support far enough in advance to ensure the successful completion of the test. The time required for coordination also depends on the scope, and typically varies from a month in advance for component testing to several months for a comprehensive test of components and systems for a large IT plan such as a disaster recovery plan or incident response plan. The checklist in Table 6-1 contains examples of possible logistics actions that might need to be performed. Although specific logistical elements are identified during the test design phase, it is imperative that the required list of logistical components be updated frequently, especially after the test is fully developed.

Table 6-1. Sample Logistics Checklist for Test Events

Once the event is designed, the design team needs to create test documentation. The magnitude of this work depends on the scope of the test. This documentation could include the following:

When developing a comprehensive test for an IT plan, there might be significant overlap with what would occur when developing a functional exercise for the same plan. In fact, many comprehensive tests have test plans that include materials similar to those used for functional exercises, including a scenario, an MSEL, and message injects. For example, during a test, participants may be told that a particular backup tape has failed, a certain server is unusable, or a road to the backup facility is closed. The key difference between a functional exercise and a test is that a test is conducted in an operational environment whenever possible, and it is designed to validate the actual effectiveness of both the processes and procedures for system recovery and restoration outlined in the plan, as well as the training that has been conducted, to ensure personnel understand their responsibilities and know how to react in a given situation.

The locations for tests vary based on the type of test being conducted and the test’s scope. For example, a small component test could be conducted in a single office, while a comprehensive test of components and systems for an IT plan could involve many different parts of an organization in various locations.

Safety and security are two elements that should be maintained during any test. The organization’s operational systems and networks need to be protected so that they do not sustain damage. The core function or mission of the organization should not be disrupted to the extent that the organization can no longer function and provide the services that it was created to provide. For these reasons, the test director should monitor all tests closely. At any sign of a possible catastrophic disruption, or in the event that the safety of an individual is at stake or the security of the organization or its data is in question, the test director and any member of the test team should have the ability to terminate the test immediately.

Following the conclusion of the test, the test director and data collectors should conduct an informal test debrief with participants, often referred to as a hotwash. The test director leads the hotwash and requests feedback from the participants and data collectors. Immediately following the test, the data collectors and participants should be asked to provide the test director with their notes or any forms completed during the course of the test and the hotwash session.

During the evaluation phase, a member of the design team or another selected staff member should develop an after action report that determines how well the tested systems or components functioned. The introduction to the after action report should document background information about the test such as the scope, objectives, and tests. The after action report should also document observations made by the test team during the test and recommendations for enhancing the IT plan that had its components or systems tested, along with associated procedures and components. The after action report should also include a list of test participants and may provide information from any participant surveys that were distributed during the hotwash to solicit feedback.

The after action report typically takes a few days to prepare. In the event that critical lapses in security or safety are noted, the test director should not wait until the final report is created to notify management. An informal report should be generated immediately with any critical changes that should be made, with the full report following in a reasonable amount of time. A formal debrief is sometimes conducted after a test so that the test director, plan coordinator, and other management staff can discuss the results of the test. Stakeholders typically review the report and may provide suggestions for minor wording changes, such as strengthening a recommendation to ensure that its importance is clear.

Following the development of the after action report, the plan coordinator might assign action items to select personnel to update the IT plan that had its components or systems tested. The plan coordinator should then update the plan, if appropriate, by implementing recommendations made in the after action report. It may also be necessary to brief additional managers on the results of the test, update other security-related documents, and perform other actions based on the test.

Tests are evaluation tools that use quantifiable metrics or expected outcomes to validate the operability of one or more IT systems or system components that are identified as critical in an IT plan. Tests can take several forms, including component testing (testing individual hardware or software components, or groups of related components), system testing (testing complete systems), and comprehensive testing (testing all systems and components that support an IT plan). A test is conducted in as close to an operational environment as possible, which means that the test should be conducted in a manner that resembles the everyday work environment in which the system or component is found. If possible, an actual test of the components or systems used to conduct daily operations for the organization should be used. Tests can potentially be disruptive to an organization’s operations, so tests are sometimes performed on systems that mimic the actual operational systems, especially if there is not strong confidence that the tests will be completely successful.

One commonly used methodology for planning and performing test events has the following phases:

This page has been left blank intentionally.

Appendix A provides the following sample documentation:

This sample documentation is designed to be used as a template by those responsible for designing and developing tabletop exercise documentation. In addition to the documentation described in this appendix, a briefing containing the agenda and logistics information should be developed and projected at the beginning of the exercise.

[INSERT ORGANIZATION NAME] [INSERT TABLETOP EXERCISE TITLE]

FACILITATOR GUIDE

[Insert Tabletop Location]

[Insert Tabletop Date]

[Insert table of contents]

In an effort to validate [insert organization name] [insert name of plan being exercised²²], [insert organization name] will conduct a tabletop exercise to examine processes and procedures associated with the implementation of the [insert plan name]. This discussion-based exercise will be a [insert number of hours]-hour event that will begin at [insert start time] and will last until [insert end time].

The exercise is designed to facilitate communication among select personnel regarding the implementation of recovery operations at [insert organization name] following an event causing the outage of mission critical systems that are housed in the [insert facility name]. This exercise is designed to improve the readiness of the [insert organization name] and help validate existing [insert plan name] procedures.

Participants should come to the exercise prepared to discuss high-level issues related to the recovery of mission critical systems at the [insert facility name]. To achieve the exercise’s stated objectives, discussion will focus on the following [insert facility name] contingency planning elements:

Participants may choose to bring back-up reference material that will aid in answering the above questions.

A tabletop exercise is a discussion-based event in which participants meet in a “classroom” setting to address the actions they would take in response to an emergency situation. Tabletops are an effective initial step for personnel to discuss the full range of issues related to a crisis scenario. These exercises provide an excellent forum to examine roles and responsibilities, unearth interdependencies, and evaluate plans.

Participants will be presented with a scenario affecting the [insert facility name]. A facilitator will help guide discussion by asking questions designed to address the exercise’s objectives. The facilitator may choose to inject modifications to the scenario to further stimulate discussion. Participants will also be encouraged to ask one another questions.

SAMPLE OBJECTIVES

The exercise objectives are as follows:

SAMPLE AGENDA

Date:

[insert date]

Location:

[insert address]

9:00 a.m.-9:15 a.m.

Welcoming Remarks and Introductions

9:15 a.m.-9:45 a.m.

Exercise Briefing (Objectives, Rules of Engagement, etc.)

9:45 a.m.-11:30 a.m.

Scenario Discussion

11:30 a.m.-12:00 p.m.

Debrief/Hotwash

SAMPLE SCENARIO

At [insert time] on [insert date], an electrical fire in the [insert facility name] caused extensive damage and the termination of operations in the data center. The [insert plan name] was fully activated in response to this incident, and operations will be conducted at the [insert alternate facility name] for the foreseeable future. [Insert organization name] employees will be displaced from the building until smoke, water, and other health hazards are removed. Despite the problem at the [insert facility name], Directors and Administrators show no sign of altering their agendas and expect a seamless transition of IT operations to the [insert alternate facility name].

SAMPLE FACILITATOR QUESTIONS

The following questions are designed to be used by the facilitator to guide the discussion and ensure the pre-defined objectives are met; depending on the flow of the exercise, the facilitator may elect to use these questions or other questions to ensure participants meet the objectives through the discussion:

SAMPLE DEBRIEF/HOTWASH QUESTIONS

An after action report identifying strengths and areas where improvements might be made will be provided after the exercise. The following questions are designed to obtain input into the after action report from participants.

A.2 Sample Tabletop Exercise Participant Guide

[INSERT ORGANIZATION NAME] [INSERT TABLETOP EXERCISE TITLE]

PARTICIPANT GUIDE

[Insert Tabletop Location]

[Insert Tabletop Date]

[Insert table of contents]

SAMPLE INTRODUCTION

In an effort to validate [insert organization name] [insert name of plan being exercised²³], [insert organization name] will conduct a tabletop exercise to examine processes and procedures associated with the implementation of the [insert plan name]. This discussion-based exercise will be a [insert number of hours]-hour event that will begin at [insert start time] and will last until [insert end time].

The exercise is designed to facilitate communication among select personnel regarding the implementation of recovery operations at [insert organization name] following an event causing the outage of mission critical systems that are housed in the [insert facility name]. This exercise is designed to improve the readiness of the [insert organization name] and help validate existing [insert plan name] procedures.

Participants should come to the exercise prepared to discuss high-level issues related to the recovery of mission critical systems at the [insert facility name]. To achieve the exercise’s stated objectives, discussion will focus on the following [insert facility name] contingency planning elements:

Participants may choose to bring back-up reference material that will aid in answering the above questions.

SAMPLE CONCEPT OF OPERATIONS

A tabletop exercise is a discussion-based event in which participants meet in a “classroom” setting to address the actions they would take in response to an emergency situation. Tabletops are an effective initial step for personnel to discuss the full range of issues related to a crisis scenario. These exercises provide an excellent forum to examine roles and responsibilities, unearth interdependencies, and evaluate plans.

Participants will be presented with a scenario affecting the [insert facility name]. A facilitator will help guide discussion by asking questions designed to address the exercise’s objectives.

SAMPLE OBJECTIVES

The exercise objectives are as follows:

SAMPLE AGENDA

Date:

[insert date]

Location:

[insert address]

9:00 a.m.-9:15 a.m.

Welcoming Remarks and Introductions

9:15 a.m.-9:45 a.m.

Exercise Briefing (Objectives, Rules of Engagement, etc.)

9:45 a.m.-11:30 a.m.

Scenario Discussion

11:30 a.m.-12:00 p.m.

Debrief/Hotwash

SAMPLE SCENARIO

At [insert time] on [insert date], an electrical fire in the [insert facility name] caused extensive damage and the termination of operations in the data center. The [insert plan name] was fully activated in response to this incident, and operations will be conducted at the [insert alternate facility name] for the foreseeable future. [Insert organization name] employees will be displaced from the building until smoke, water, and other health hazards are removed. Despite the problem at the [insert facility name], Directors and Administrators show no sign of altering their agendas and expect a seamless transition of IT operations to the [insert alternate facility name].

SAMPLE PARTICIPANT QUESTIONS

The following questions sample questions that might appear in the Participant Guide.

SAMPLE DEBRIEF/HOTWASH QUESTIONS

An after action report identifying strengths and areas where improvements might be made will be provided after the exercise. The following questions are designed to obtain input into the after action report from participants.

A.3 Sample Tabletop Exercise After Action Report

[INSERT ORGANIZATION NAME] [INSERT TABLETOP EXERCISE TITLE]

AFTER ACTION REPORT

[Insert Tabletop Location]

[Insert Tabletop Date]

[Insert table of contents]

SAMPLE INTRODUCTION

On [insert date], [insert organization name] participated in [insert duration of exercise]-hour tabletop exercise designed to validate their understanding of the [insert plan name].

SAMPLE OBJECTIVES

The exercise objectives are as follows:

SAMPLE AGENDA

Date:

[insert date]

Location:

[insert address]

9:00 a.m.-9:15 a.m.

Welcoming Remarks and Introductions

9:15 a.m.-9:45 a.m.

Exercise Briefing (Objectives, Rules of Engagement, etc.)

9:45 a.m.-11:30 a.m.

Scenario Discussion

11:30 a.m.-12:00 p.m.

Debrief/Hotwash

SAMPLE DISCUSSION FINDINGS

The [insert exercise name] provided information on [insert relevant information]. An important benefit of the exercise was the opportunity for participants to raise important questions, concerns, and issues. At the conclusion of the exercise, participants were asked to complete an evaluation form regarding the information provided, additional information needed, and their thoughts on the event and topics, to be included in the after action report. A sample evaluation form can be found on page C-16.

The discussion findings from the exercise along with any necessary recommended actions are as follows:

General Findings

The exercise provided an excellent opportunity for participants to [insert relevant information]. As a result of the exercise, participants left with a heightened awareness of [insert relevant information].

Specific Findings

Specific observations made during the exercise, and recommendations for enhancement of the plan, are as follows:

Observation 1. [Insert general topic area]

[Insert observation]

Recommendations

[Insert recommendations]

Observation 2. [Insert general topic area]

[Insert observation]

Recommendations

[Insert recommendations]

Example Observations and Recommendations:

Observation 1. Communications

A plan identifying standardized systems for communicating with contingency plan members does not exist.

Recommendations

Observation 2. Flyaway Kits

Essential personnel have not been issued flyaway kits, containing personal items and/or those items needed to perform their operations, to carry to relocation facilities in the event of an emergency.

Recommendation

SAMPLE EVALUATION FORM

Please take a moment to fill out the evaluation form.

Information

Exercise Evaluation Form

Survivability Program

Please take a few moments to answer the following questions about the exercise.

NAME

1). Did you have available to you all of the information and resources needed to fulfill your responsibilities?

2). Did you feel that there was an adequate level of training to support the response effort at the relocation site?

3). Was the structure of the exercise realistic?

4). Please provide comments regarding what you believe worked and didn't work during the exercise.

5). Do you believe you are sufficiently prepared to conduct extended emergency operations from the relocation facility? Please Circle One

Not Prepared Slightly Prepared Prepared Extremely Prepared

6.) Please rate the overall exercise. Please Circle One

Needs Improvement Fair

Good Very Good

SAMPLE EVALUATION RESULTS

Following the [insert tabletop exercise name], on [insert date], participants were given an evaluation form on which to record their impressions of the exercise. These forms allowed participants to rate presentations on a numerical scale and to provide additional comments for consideration in the after action report. Refer to Exhibit 1 for more detailed information regarding the participants’ responses. Each exhibit will reflect the evaluation forms for each individual event. If evaluation forms have a point scale, either a pie chart or bar graph will be depicted.

The questions covered whether participants thought additional issues should have been raised; whether participants thought the exercise was beneficial; what participants gained from the exercise; and what can be done to improve future exercises. [Insert percentage] of the participants completed the evaluation.

In response to the question regarding whether participants thought additional issues should have been raised, nearly [Insert percentage] of those who completed the evaluation indicated that all relevant issues were addressed. Other comments were [insert relevant information].

In response to the question regarding whether participants thought the exercise was beneficial, [Insert percentage] of those who completed the evaluation indicated that the exercise was beneficial. Comments ranged from [insert relevant information (i.e., "good start" to "extremely beneficial.")].

In response to the question about what participants gained from the exercise, nearly [Insert percentage] of those who completed the evaluation form remarked [insert relevant information].

EXHIBIT: PARTICIPANT RESPONSES

Overall, the feedback from the [insert tabletop exercise name] was [insert relevant information].

This page has been left blank intentionally.

Appendix B—Sample Functional Exercise Documentation

Appendix B provides the following sample documentation:

This sample documentation is designed to be used as a template by those responsible for designing and developing functional exercise documentation. In addition to the documentation described in this appendix, a briefing containing the agenda and logistics information should be developed and presented at the beginning of the exercise.

B.1 Sample Functional Exercise Scenario

[INSERT ORGANIZATION NAME] [INSERT FUNCTIONAL EXERCISE TITLE]

SCENARIO

[Insert Functional Exercise Location]

[Insert Functional Exercise Date]

The scenario is developed by the functional exercise team during the Development Phase. Scenario documentation may include a brief Scenario Background designed to provide participants a sense of the world/local situation in the weeks or months before the start of the exercise. This information will be provided to participants in briefings before the exercise or at the beginning of the exercise event. The scenario itself portrays the events that will occur during the conduct of the event. These events will also become a part of the Master Scenario Events List and will be introduced into exercise play in the form of Injects.

D-Day Minus 20

International tensions have dramatically risen overseas involving strategic interests of the United States. Despite attempts to resolve disputes diplomatically, troops from hostile countries deploy and appear poised to make a major military incursion against nations allied to the United States. U.S. intelligence agencies also detect documented attempts by hostile nations to destabilize the governments and economies of allies, which would have an adverse impact on U.S. military and economic interests in the region.

D-Day Minus 10

As tensions continued to build, hostile entities undertake small-scale military operations against allied and U.S. interests overseas. A U.S. reconnaissance plane is shot down and the bodies of the dead crew are displayed on television. An emergency Cabinet meeting is called and it is decided that the U.S. military will deploy to the region to protect allied governments and U.S. interests. It is anticipated that an initial operational capability to defend U.S. interests will not be complete until next month.

D-Day Minus 5

In response to the U.S. declaration to send troops and materiel to the region, hostile nations vow to take whatever actions are necessary to “strike a vicious blow against the American imperialists.” They state that any war that the U.S. provokes will also be fought on the American homeland. U.S. intelligence agencies soon detect an increase of cyber attacks against U.S. and threats to carry out terrorist attacks against the U.S. government. Intelligence also indicates that hostile foreign interests within the U.S. are increasingly active and terrorist cells in other countries have been activated to potentially carry out attacks against the U.S., both overseas and within U.S. borders. U.S. government officials suspect that the hostile nations hope to weaken U.S. public support and impede the military’s capability to deploy by engaging in actions that might include the use of weapons of mass destruction and cyberterrorism. Nevertheless, the U.S. military continues to deploy to the region.

D-Day

0900: The United States Computer Emergency Readiness Team (US-CERT) issues an alert indicating the presence of what is thought to be an advanced computer worm. US-CERT estimates that the worm has already infected over 500,000 computers worldwide in only 2 hours, prompting the center to warn that the worm’s spread has the potential to disrupt business and personal use of the Internet for applications such as electronic commerce, e-mail, and entertainment. U.S. intelligence agencies link the worm’s release as a response to U.S. military deployments and fear the attack may have been designed specifically to disrupt communications between agencies supporting the military deployment.

Intrusions into numerous Federal Web sites have been reported in recent hours. Hackers, who government officials believe to be associated with hostile nations, have successfully compromised the security of various U.S. Government information systems. Anti-government vandalism has also been reported on numerous department and agency Web sites.

[Insert organization/data center name] is currently in the process of responding to the effects of the worm and defending against further electronic intrusions.

1000: As a result of credible threats of imminent terrorism and the fear that ongoing electronic intrusions against Federal information systems may be part of a concerted information warfare attack, the Department of Homeland Security (DHS) raises the Homeland Security Advisory System threat condition from an Orange “High” to a Red “Severe” risk of terrorist attack.

1200: A freight train pulling numerous tanker cars passes slowly through the center of the city. Some of the tankers are carrying methyl isocyanate (MIC), a highly explosive chemical used in the production of pesticides. As the train passes the [insert name of organization’s facility], one of the tanker cars erupts in a violent explosion. The blast collapses a portion of the building and results in power outages in the immediate vicinity. Many personnel in the [insert name of organization’s facility] are killed or injured.

The [insert data center name] survived the blast and emergency power was engaged for critical IT systems. Several data center personnel had recently left the center for lunch and their well-being is unknown at this time. Management has directed that the data center implement contingency plans and relocate to their alternate computing facility. In addition to restoring essential data center operations at the alternate facility, management indicates that defending [insert organization name] from further electronic intrusions remains a priority.

D-Day Plus 1

1000: Intelligence and law enforcement agencies continue to track numerous threats against the United States. The Federal Bureau of Investigation (FBI) notifies Government agencies of documented threats of further attacks against Federal departments and agencies throughout the United States. One such notice to [insert organization name] includes a report of possible terrorist surveillance activities outside the organization’s alternate computing facility. As a result, management directs that data center personnel explore options to move operations to a backup alternate facility in the event that the threat is found to be credible.

[INSERT ORGANIZATION NAME] [INSERT FUNCTIONAL EXERCISE TITLE]

MASTER SCENARIO EVENTS LIST (MSEL)

[Insert Functional Exercise Location]

[Insert Functional Exercise Date]

The Master Scenario Events List (MSEL) is created by the functional exercise team during the Development Phase. The MSEL lists key scenario events, expected Injects that will build on the key events, and the objectives of the each MSEL item. Controllers, simulators, and data collectors will refer to the MSEL throughout the Conduct Phase of the exercise to ensure the exercise remains on track.

[INSERT ORGANIZATION NAME] [INSERT FUNCTIONAL EXERCISE TITLE]

[Insert Functional Exercise Location]

[Insert Functional Exercise Date]

Injects are created by the functional exercise team during the Development Phase. These messages are introduced by Controllers during the Conduct Phase and are provided to exercise participants via the means shown on the Inject form. In the case of the Sample Inject provide below, a Controller would play the role of the Chief Information Officer and would call the Team Chief to provide information and request follow-on action. Expected actions by the Team Chief or other exercise participants are documented in the “Notes to Control/Response Cell” at the bottom of this form to aid controllers, simulators, or data collectors in anticipating what actions will result from the Inject.

[Insert Name of Exercise] Implementers for [Insert Date]

EXAMPLE

#15 – [Insert inject title] (i.e., Development of Disaster Recovery Strategies for Alternate Facility)

原本PDFの抜粋画像

Inject Date/Time: [Insert date/time] (i.e., Day 2, 1045)

From:

[Insert by whom the message is delivered] (i.e., Chief Information Officer)

To:

[Insert for whom the message is intended] (i.e., Team Chief)

Inject Means:

[Insert the means by which the message is delivered] (i.e., Phone Call)]

[Insert message text]

Example

Now that we know the magnitude of the damage to the Building and our data center there, it is apparent that we will be operating out of the alternate facility (AF) for the foreseeable future. Given the continued threat of terrorist attacks, we need to develop contingency plans in the event of a major outage affecting the AF. What is our strategy to ensure continuity of mission critical systems at the AF? Which systems and applications are prioritized for recovery? How long will it take to develop a viable backup for those systems?

Note to Control/Response Cell:

原本PDFの抜粋画像

[Insert any type of information that the Control/Response Cell may need to consider to track, evaluate, or respond to exercise players]

Example

Expect the AF Team Chief to consult the AF Contingency Plan and coordinate with appropriate system and application engineers to develop a recovery strategy.

B.4 Sample Functional Exercise Inject Tracking Form

[INSERT ORGANIZATION NAME] [INSERT FUNCTIONAL EXERCISE TITLE]

INJECT TRACKING FORM

[Insert Functional Exercise Location]

[Insert Functional Exercise Date]

Portions of the Inject Tracking Form are developed by the functional exercise team during the Development Phase, and additional information is filled in by Controllers during the Conduct Phase. In the sample below, BOLDED text would have been created during the Development Phase and ITALICIZED text would have added to the form by Controllers during the Conduct Phase. The purpose of the form is to provide Controllers with a “play book” that states which injects are provided to participants at each given time. The form is then used by Controllers to document the time at which an Inject is introduced and a summary of activities taken by participants in response to the Inject.

B.5 Sample Functional Exercise After Action Report

[INSERT ORGANIZATION NAME] [INSERT FUNCTIONAL EXERCISE TITLE]

AFTER ACTION REPORT

[Insert Functional Exercise Location]

[Insert Functional Exercise Date]

The After Action Report is developed by Data Collectors during the Evaluation Phase. The After Action Report provides relevant background information about the event, scope of the exercise, objectives, scenario, and key findings and recommendations. In addition, the After Action Report lists the event’s participants and may provide relevant information from surveys completed by participants at the conclusion of the exercise.

[Insert table of contents]

On [insert date], [insert organization name] participated in [insert duration of exercise] functional exercise designed to validate their understanding of the Cyber Intrusion Response Plan and the Alternate Facility Contingency Plan. [Insert any other additional background information about the exercise that is relevant for the after action report.]

The exercise was designed to examine the [insert organization’s name] ability to respond to a concerted cyber attack campaign from the alternate computing facility. The event examined all aspects of the activation, operation, and reconstitution phases of both the Cyber Intrusion Response Plan and the Alternate Facility Contingency Plan. All personnel with operational responsibilities under the two plans participated in the event. Senior level decision-makers were not exercised in the exercise; however, a second exercise focused on management activities is planned in the coming months.

The exercise objectives are as follows:

[Insert scenario or high-level overview of the scenario.]

SAMPLE EXERCISE FINDINGS

The [insert exercise name] provided information on [insert relevant information]. An important benefit of the exercise was the opportunity for participants to receive hands-on training in responding to an emergency from the alternate facility. In addition, the exercise provided participants with an opportunity to raise important questions, concerns, and issues. At the conclusion of the exercise, participants were asked to complete an evaluation form regarding the information provided, additional information needed, and their thoughts on the event, to be included in the after action report. [A sample evaluation form can be found on page B-17.]

Findings from the exercise and recommended actions are as follows:

General Findings

The exercise provided an excellent opportunity for participants to [insert relevant information]. As a result of the exercise, participants left with a heightened awareness of [insert relevant information].

Specific Findings

Specific observations made during the exercise, and recommendations for enhancement of the plan, are as follows:

Observation 1. [Insert general topic area]

[Insert observation]

Recommendations

[Insert recommendations]

Observation 2. [Insert general topic area]

[Insert observation]

Recommendations

[Insert recommendations]

Example Observations and Recommendations:

Observation 1. Communications

A plan identifying standardized systems for communicating with contingency plan members does not exist.

Recommendations

Observation 2. Flyaway Kits

Essential personnel have not been issued flyaway kits, containing personal items and/or those items needed to perform their operations, to carry to relocation facilities in the event of an emergency.

Recommendation

Please take a moment to fill out the evaluation form.

Please take a few moments to answer the following questions about the exercise.

NAME

1). Did you have available to you all of the information and resources needed to fulfill your responsibilities?

2). Did you feel that there was an adequate level of training to support the response effort at the relocation site?

4). Please provide comments regarding what you believe worked and didn't work during the exercise.

Not Prepared Slightly Prepared Prepared Extremely Prepared

Needs Improvement Fair Good Very Good

Following the [insert functional exercise name], on [insert date(s)], participants were given an evaluation form on which to record their impressions of the exercise. These forms allowed participants to rate presentations on a numerical scale and to provide additional comments for consideration in the after action report. Refer to Exhibit 1 for more detailed information regarding the participants’ responses. [Each exhibit will reflect the evaluation forms for each individual event. If evaluation forms have a point scale, either a pie chart or bar graph will be depicted.]

The questions covered whether participants thought additional issues should have been raised; whether participants thought the exercise was beneficial; what participants gained from the exercise; and what can be done to improve future exercises. [Insert percentage] of the participants completed the evaluation.

In response to the question regarding whether participants thought additional issues should have been raised, nearly [insert percentage] of those who completed the evaluation indicated that all relevant issues were addressed. Other comments were [insert relevant information].

In response to the question regarding whether participants thought the exercise was beneficial, [insert percentage] of those who completed the evaluation indicated that the exercise was beneficial. Comments ranged from [insert relevant information (i.e., "good start" to "extremely beneficial.")].

In response to the question about what participants gained from the exercise, nearly [insert percentage] of those who completed the evaluation form remarked [insert relevant information].

EXHIBIT 1: PARTICIPANT RESPONSES

Overall, the feedback from the [insert functional exercise name] was [insert relevant information].

Appendix C provides the following sample documentation for three types of tests (component, system, and comprehensive):

This sample documentation is designed to be used as a template by those responsible for designing and developing test documentation.

C.1 Sample Component Test Documentation

An example of a component test is the occasional test of the nation’s Emergency Alert System, formerly known as the Emergency Broadcast Network.²⁴ During such a test, the equipment is tested, and a tone and announcement associated with an emergency are broadcasted on the radio or television. Although the test does not involve a simulated emergency with responders, the message tests a specific component of the system.

[Insert test type or name]

Component Test Plan

Date of Testing: [Insert date]

Time Period: [Insert time] to [insert time]

Frequency: [Insert frequency]

Test Focus: [Insert test focus]

Test Objectives: The objectives of this test are as follows:

Test Details: [Insert test details]

Participants: [Insert participants]

Training Staff: [Insert training staff]

Validation Staff: [Insert validation staff]

Evaluation Staff: [Insert evaluation staff]

Test Cancellation Procedures: [Insert test cancellation procedures]

Test Main Point of Contact: [Insert test main point of contact]

Test Approval Grantor: [Insert test approval grantor]

On [insert date] between [insert time] and [insert time], the [insert component] for [organization or policy] will be tested. Participants will be expected to perform the following tasks:

Test Cancellation Procedures: [Insert test cancellation procedures]

If you have any questions, please contact [insert point of contact].

Initiate the Emergency Alert System, which will perform the following functions:

A test of a component requires validation criteria to determine whether the component or system functioned as intended. Test validation should include the metrics by which the success of the component or system will be measured. It should also detail the expected outcome; in this case, the two-tone attention signal and message should be heard clearly.

[Insert test type or name]

Component Test Validation Worksheet

Date of Testing: [Insert date]

Time Period: [Insert time] to [insert time]

Test Focus: [Insert test focus]

Participants: [Insert participants]

Training Staff: [Insert training staff]

Validation Staff: [Insert validation staff]

Test Objectives: The objectives of this test are as follows:

Validation Methodology: [Insert validation methodology]

Was the test able to be validated? [Insert answer]

Comments: [Insert comments]

Were there any aspects of the test that could not be validated? [Insert answer]

Comments: [Insert comments]

Recommendations: [Insert recommendations]

Based on the objectives and validation metrics, the test should be evaluated to determine whether the system test and associated components and processes performed adequately. Possible improvements and recommendations are an important part of the evaluation process.

[Insert test type or name]

Component Test Evaluation Worksheet

Date of Testing: [Insert date]

Time Period: [Insert time] to [insert time]

Test Focus: [Insert test focus]

Participants: [Insert participants]

Training Staff: [Insert training staff]

Validation Staff: [Insert validation staff]

Evaluation Staff: [Insert evaluation staff]

Test Objectives: The objectives of this test are as follows:

Were the test objectives met? [Insert answer]

Comments: [Insert comments]

Test details: [Insert test details]

Was testing criterion adequate? [Insert answer]

Comments: [Insert comments]

Could the testing criterion be improved? [Insert answer]

Comments: [Insert comments]

Was the test validation criterion adequate? [Insert answer]

Could the test validation criterion be improved? [Insert answer]

Comments: [Insert comments]

Did the test perform as expected? [Insert answer]

Comments: [Insert comments]

Were there any failures during the test? [Insert answer]

Did the failure cause the test to fail? [Insert answer]

Comments: [Insert comments]

Recommendations: [Insert recommendations]

COMPONENT TEST AFTER ACTION REPORT

The component test after action report should consist of the following components:

COMPONENT TEST GENERAL FINDINGS

The General Findings section highlights the outcome of the test. It might consist of a statement along the lines of the following:

The component test provided an excellent opportunity for participants to test the [insert relevant information]. As a result of this component test, participants received a heightened awareness of the importance of [insert relevant information]. The [organization or department] as a whole learned [insert relevant information] as a result of this test.

COMPONENT TEST SPECIFIC FINDINGS

The Specific Findings section provides greater detail of the results of the test. It should provide sufficient detail so that a person knowledgeable of the technical aspects of the component could use the evaluation to improve the component or process. A possible outline for the specific observations section is as follows:

Specific observations made during the exercise, and recommendations for enhancement of the plan, are as follows:

Observation 1. [Insert general topic area]

[Insert observation]

Recommendations

[Insert recommendations]

Observation 2. [Insert general topic area]

[Insert observation]

Recommendations

[Insert recommendations]

SUPPORTING DATA

The Supporting Data section of the report includes the specific data that was collected during the test. These are often included as attachments with a brief explanation of how they were gathered.

C.2 Sample System Test Documentation

An example of a system test is the testing of an organization’s data backup and restoration system and procedures. During such a test, all aspects of the data backup equipment and personnel procedures for archiving and restoring data are tested.

SAMPLE SYSTEM TEST STRUCTURE:

System Test: Data Backup and Restoration

SAMPLE SYSTEM TEST PLAN

[Insert test type or name]

System Test Plan

Date of Testing: [Insert date]

Time Period: [Insert time] to [insert time]

Frequency: [Insert frequency]

Test Focus: [Insert test focus]

Test Objectives: [Insert test objectives]

Test Details: [Insert test details]

Test Components:

System Test Component 1: [Insert component]²⁵

System Test Main Point of Contact: [Insert system test main point of contact]

System Test Approval Grantor: [Insert system test approval grantor]

SAMPLE SYSTEM TEST BRIEFING FOR PARTICIPANTS

On [insert date] between [insert time] and [insert time], the [insert component or system] for [department or policy] will be tested. Each of the following components will be tested:

[insert component] Component Test in [insert system] System Test²⁶

On [insert date] between [insert time] and [insert time], the [insert component] for [department or policy] will be tested. Participants will be expected to perform the following tasks:

Test Cancellation Procedures: [Insert test cancellation procedures]

If you have any questions, please contact [insert point of contact].

SAMPLE SYSTEM TEST VALIDATION

A test of any system requires methods and criteria to evaluate whether the system test and associated component tests or processes worked as intended. Test validation should include the metrics by which the success of the component or process will be measured and detail the expected outcome. System test validation can be determined by the validation of each individual component test objectives and validation.

In this example, validation for this system test should verify the backup procedures’ accuracy and equipment functionality for backing up data for its associated type of systems; verify data integrity checking; validate local and offsite data storage and retrieval procedures; and verify system restoration function as expected for each associated type of system.

SAMPLE SYSTEM TEST VALIDATION WORKSHEET

[Insert test type or name]

System Test Validation Worksheet

Date of Testing: [Insert date]

Time Period: [Insert time] to [insert time]

Test Focus: [Insert test focus]

Participants: [Insert participants]

Training Staff: [Insert training staff]

Validation Staff: [Insert validation staff]

Test Objectives: The objectives of this test are as follows: