NIST SP 800-50 Rev. 1 日本語訳

Marian Merritt

Susan Hansche

Dr. Brenda Ellis

Kevin Sanchez-Cherry

Julie Nethery Snyder

Donald Walden

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-50r1

Marian Merritt, Applied Cybersecurity Division, Information Technology Laboratory

Susan Hansche, Cybersecurity and Infrastructure Security Agency, Department of Homeland Security

Dr. Brenda Ellis, National Aeronautics and Space Administration

Kevin Sanchez-Cherry, Office of the Chief Information Officer, Department of Transportation

Julie Nethery Snyder, MITRE

Donald Walden, Internal Revenue Service

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-50r1

September 2024

U.S. Department of Commerce Gina M. Raimondo, Secretary

National Institute of Standards and Technology Laurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology

Certain equipment, instruments, software, or materials, commercial or non-commercial, are identified in this paper in order to specify the experimental procedure adequately. Such identification does not imply recommendation or endorsement of any product or service by NIST, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose.

There may be references in this publication to other publications currently under development by NIST in accordance with its assigned statutory responsibilities. The information in this publication, including concepts and methodologies, may be used by federal agencies even before the completion of such companion publications. Thus, until each publication is completed, current requirements, guidelines, and procedures, where they exist, remain operative. For planning and transition purposes, federal agencies may wish to closely follow the development of these new publications by NIST.

Organizations are encouraged to review all draft publications during public comment periods and provide feedback to NIST. Many NIST cybersecurity publications, other than the ones noted above, are available at https://csrc.nist.gov/publications.

Authority This publication has been developed by NIST in accordance with its statutory responsibilities under the Federal Information Security Modernization Act (FISMA) of 2014, 44 U.S.C. § 3551 et seq., Public Law (P.L.) 113-283. NIST is responsible for developing information security standards and guidelines, including minimum requirements for federal information systems, but such standards and guidelines shall not apply to national security systems without the express approval of appropriate federal officials exercising policy authority over such systems. This guideline is consistent with the requirements of the Office of Management and Budget (OMB) Circular A-130.

Nothing in this publication should be taken to contradict the standards and guidelines made mandatory and binding on federal agencies by the Secretary of Commerce under statutory authority. Nor should these guidelines be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, Director of the OMB, or any other federal official. This publication may be used by nongovernmental organizations on a voluntary basis and is not subject to copyright in the United States. Attribution would, however, be appreciated by NIST.

NIST Technical Series Policies Copyright, Use, and Licensing Statements NIST Technical Series Publication Identifier Syntax

Publication History Approved by the NIST Editorial Review Board on 2024-08-09 Supersedes NIST SP 800-50 (October 2003) https://doi.org/10.6028/NIST.SP.800-50; Supersedes NIST SP 800-16 (April 1998) https://doi.org/10.6028/NIST.SP.800-16

How to Cite this NIST Technical Series Publication: Merritt M, Hansche S, Ellis B, Sanchez-Cherry K, Snyder JN, Walden D (2024) Building a Cybersecurity and Privacy Learning Program. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-50r1. https://doi.org/10.6028/NIST.SP.800-50r1

Marian Merritt: 0000-0002-2116-8959

Contact Information sp800-50-comments@nist.gov National Institute of Standards and Technology Attn: Applied Cybersecurity Division, Information Technology Laboratory 100 Bureau Drive (Mail Stop 2000) Gaithersburg, MD 20899-2000

Additional Information Additional information about this publication is available at https://csrc.nist.gov/pubs/sp/800/50/r1/final, including related content, potential updates, and document history.

All comments are subject to release under the Freedom of Information Act (FOIA).

This publication provides guidance for federal agencies and organizations to develop and manage a life cycle approach to building a Cybersecurity and Privacy Learning Program (CPLP). The approach is intended to address the needs of large and small organizations as well as those building an entirely new program. The information leverages broadly accepted standards, regulations, legislation, and best practices. The recommendations are customizable and may be implemented as part of an organization-wide process that manages awareness, training, and education programs for a diverse set of federal employee audiences. The program should encourage behavior change as part of risk management and lead to developing a privacy and security culture in the organization. The guidance also includes suggested metrics and evaluation methods to regularly improve and update the program as needs evolve.

awareness; behavior change; cybersecurity; education; learning program; privacy; privacy culture; role-based; security culture; training.

The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective cybersecurity and privacy of other than national security-related information in federal information systems. The Special Publication 800-series reports on ITL’s research, guidelines, and outreach efforts in information system security, and its collaborative activities with industry, government, and academic organizations.

Patent Disclosure Notice

NOTICE: ITL has requested that holders of patent claims whose use may be required for compliance with the guidance or requirements of this publication disclose such patent claims to ITL. However, holders of patents are not obligated to respond to ITL calls for patents and ITL has not undertaken a patent search in order to identify which, if any, patents may apply to this publication.

As of the date of publication and following call(s) for the identification of patent claims whose use may be required for compliance with the guidance or requirements of this publication, no such patent claims have been identified to ITL.

No representation is made or implied by ITL that licenses are not required to avoid patent infringement in the use of this publication.

This publication was developed through the efforts of a dedicated team of volunteer authors. We express our thanks to Jessica Dickson, National Institute of Standards and Technology (NIST); Susanne Furman, NIST; Julie Haney, NIST; Dan Jacobs, Office of Personnel Management; Jody Jacobs, NIST; Eric Gray, Department of Education; Sarah Moffatt, National Institutes of Health; Dylan Gilbert, NIST; Naomi Lefkovitz, NIST; Jeremy Licata, NIST; Rodney Petersen, NIST; Eduardo Takamura, NIST; and Victoria Yan Pillitteri, NIST.

Executive Summary

Ensuring that a federal organization’s workforce is aware of and prepared to respond appropriately and effectively to cybersecurity and privacy risks is an important effort that requires a strategic approach based on thoughtful planning, resource considerations, and leadership-driven decision-making. This long-awaited update to the 2003 NIST Special Publication (SP) 800-50, Building an Information Technology Security Awareness and Training Program, provides guidance to create and manage a program that includes cybersecurity and privacy awareness campaigns, role-based training, and other workforce education programs. These programs combine to create an overall Cybersecurity and Privacy Learning Program (CPLP) that is part of a larger organizational effort to reduce cybersecurity and privacy risks. The resulting CPLP supports federal requirements and incorporates industry-recognized best practices for risk management.

In addition to meeting statutory responsibilities under the Federal Information Security Management Act (FISMA) [2], this Special Publication supports the National Defense Authorization Act of 2021 (NDAA) [1] to “publish standards and guidelines for improving cybersecurity awareness of employees and contractors of Federal agencies.”¹ Including privacy as a foundational element in the CPLP reflects the guidance found in the 2016 update to the Office of Management and Budget (OMB) Circular A-130:

…it also emphasizes the role of both privacy and security in the federal information life cycle. Importantly, the inclusion of privacy represents a shift from viewing security and privacy requirements as merely compliance exercises to understanding security and privacy as crucial and related elements of a comprehensive, strategic, and continuous risk-based program at federal agencies. [3]

原本PDFの抜粋画像

Additionally, this update includes elements previously found in SP 800-16, Information Technology Security Training Requirements: A Role- and Performance-Based Model [4], which identified federal agency and organizational work roles that required specialized role-based training for cybersecurity tasks and skills. The relevant content from SP 800-16 has been incorporated into this publication or has been included in SP 800-181r1 [5]. As a result, SP 800-16 will be withdrawn upon the release of this publication.

Everyone in an organization plays a role in the success of an effective cybersecurity and privacy program. For those whose information technology, cybersecurity, or cybersecurity-related job responsibilities require additional or specific training, the NICE Workforce Framework for

Cybersecurity (NICE Framework)² [5] identifies the specific knowledge and skills necessary to perform tasks associated with work roles in these areas.³

Users of this publication will find guidance on the steps necessary to:

Throughout each section, there are recommendations to enable a program to continually evolve and improve, thereby minimizing privacy and security risks to the organization.

This document identifies the phases in the management of a CPLP and is organized as follows:

Reducing and managing cybersecurity and privacy risks require continuous attention from everyone in an organization. A key component of organizational cybersecurity and privacy plans is a learning program, which helps to build an understanding of risks and explain everyone’s role in identifying, responding to, and managing those risks. While learning programs vary in each federal organization, there are fundamental shared elements that can be utilized to create a Cybersecurity and Privacy Learning Program (CPLP) strategy and establish support for implementation, evaluation, and reporting activities. The CPLP effort sits within a larger federal organizational program to address risks, including efforts to reduce risks from physical privacy and security threats, insider threats, and other related human risk management concerns. Such efforts are outside of the scope of this document.

For ease of use, the remainder of this document will use the term “CPLP” to refer to all elements of cybersecurity and privacy awareness, training, and educational activities to include other programs with titles such as awareness training, practical exercises (e.g., tabletop exercises, role-playing simulations, cyber ranges, or phishing campaigns), topic-based training, role-based training, and educational programs. It is not necessary for organizations to rename their programs as CPLP; this term is applied generically.

The previous version of SP 800-50 defined awareness, training, and education as separate elements in a learning continuum. Research efforts [6][7] conducted with Federal Government training managers have shown that these terms have different meanings and can lead to confusion when describing the broader purpose of building a CPLP. While some managers may refer to programs as “awareness and training” or “awareness training,” the terms are applied inconsistently across organizations. Regardless of what an organization calls its program, the overarching goal of a CPLP is to provide opportunities for learning at all levels or stages of one’s career. It is about creating programs where learning can take place.

This Special Publication also uses the term “learner” to describe individuals in any of the learning program audience segments. The term “learner” can be found in SP 800-181r1 [5], where it refers to an individual who is acquiring specialized knowledge or developing a skill. This definition is useful here since it describes all participants in a “learning program.” This document will refer to the program as a CPLP or CPLPs, as some organizations may require multiple programs to fulfill the learning requirements of the entire workforce.

This document provides guidelines for building and maintaining comprehensive CPLPs for federal organizations and includes awareness activities and campaigns, awareness training, practical exercises, topic-based training, role-based training, and educational programs. The document includes guidance on how an organization can create a strategic program plan and ensure that there are appropriate resources to meet the organization’s learning goals.

This publication is intended to serve a diverse audience, including:

Individuals associated with designing, developing, implementing, assessing, o operating, managing, and improving CPLPs for federal agencies and organizations

Individuals responsible for overseeing federal human resources, talent o management, contractors, and training programs

Individuals responsible for federal CPLPs, learning professionals, and managers, o such as Chief Learning Officers (CLOs) and curriculum developers

Individuals responsible for meeting staff learning needs, prioritizing the use of o learning resources, identifying learning gaps, and evaluating learning effectiveness within the workspace

Individuals with information system oversight or governance responsibilities, o such as senior leaders, risk executives, authorizing officials, Chief Information Officers (CIOs), Chief Information Security Officers (CISOs), Data Management Officers (DMOs), and Chief Privacy Officers (CPOs)

Individuals with cybersecurity and privacy management responsibilities (e.g., o managing programs and projects and ensuring that staff members have the appropriate knowledge and skills to perform their work roles), including program and project managers, cybersecurity managers, and security operation managers

Workforce members who are responsible for assisting in the identification of o CPLP activities as a subject-matter expert (SME), meeting the requirements of the roles or job functions, identifying learning and/or competency gaps and needs within the organization’s cybersecurity and privacy program, determining necessary customizations, and developing a compliance baseline for the organization

Key considerations for managers of federal Cybersecurity and Privacy Learning Programs⁴

原本PDFの抜粋画像

This guide describes how a federal organization can design, develop, implement, and maintain a CPLP as part of an enterprise cybersecurity and privacy program. This includes identifying the learning needs of the organization’s personnel, including federal full-time or part-time employees, interns, retirees, contractors, external partners, consultants, researchers, and any users who access the enterprise system or data in support of the organizational mission.

CPLPs are inclusive of various other programs, such as awareness programs, social engineering campaigns, new hire training, regular training, technical training, role-based training, and other relevant learning activities conducted within the organization or through external resources (e.g., courses, certificates, and advanced programs). Additionally, CPLPs sit within a larger federal organizational effort to address and reduce risks from physical privacy and security threats, insider threats, and other related human risk management concerns. Those efforts are outside of the scope of this document.

CPLPs must have actively managed plans throughout their life cycles, which requires attention and adjustment over time. CPLP managers should carefully and thoughtfully outline, discuss, review, and document the CPLP’s goals and available options. When the owners of the organization’s CPLP adopt an effective strategy and develop a proper planning approach with measurement and feedback throughout the year, the entire organization remains connected to the CPLP’s objectives. Figure 1 shows the key phases of building and managing a learning program: Plan and Strategy, Analysis and Design, Development and Implementation, and Assessment and Improvement.

Fig. 1. Cybersecurity and Privacy Learning Program life cycle

These phases can occur in sequence or simultaneously. At any time during the life cycle, CPLP managers and their teams can develop curriculum, evaluate instructor feedback, send out practical exercise email quizzes, design posters for awareness, or develop a presentation for senior leadership. Consider this diagram a reminder of the breadth of work.

In a broad sense, CPLPs are valued elements of the organization’s learning culture. To be effective, the CPLPs must be linked to organizational goals and viewed as adaptive, continuous, and evolving. In a learning organization, personnel can expand and enhance their current capabilities to understand and meet new mission requirements. Personnel are respected for their ability to create and inspire others and are active in creating life-long learning achievements. If an organization offers other learning programs (e.g., career development, leadership, and executive development), CPLPs need to be similarly integrated into the enterprise-wide learning structure.

Developing a cybersecurity and privacy culture is an important component of establishing a successful CPLP. The culture of the organization should emphasize, reinforce, and drive its desired behaviors toward cybersecurity and privacy. Risks increase when the dominant culture fails to recognize the value of engaging individuals in the efforts to support cybersecurity and privacy best practices. There are two risk sectors that the CPLP can address:

When a CPLP is valued in the organization’s culture, the ability to address both types of risk increases. The organization’s leaders establish a CPLP as a significant component of managing risks by supporting, championing, and participating in learning activities, from awareness campaigns to role-based training. CPLP learning managers create rigorous measurement systems to evaluate the workforce’s attitudes, engagement, and support for the program. CPLP activities contribute to and help measure the organization’s cybersecurity and privacy culture. Human resource leaders reinforce those cultural norms by ensuring that CPLP messaging is part of onboarding and other workplace materials. Together, these efforts set the tone for the entire organization.

People are an organization’s greatest asset. A cybersecurity and privacy culture supports an environment in which the entire workforce is well-versed in cybersecurity and privacy risk management needs, expectations, and values. An organization, in turn, supports an effective cybersecurity and privacy culture when it understands the needs of the workforce and provides education and training to help employees and contractors learn expected cybersecurity and privacy behaviors. Any effective learning activity can be incorporated into a CPLP when it is respectful, inclusive, and helps learners understand their roles in the organization. The content should indicate to the learner that they are a valued participant in helping the organization manage risks. The workforce appreciates that they will contribute to the organization’s positive cybersecurity and privacy culture with the knowledge and skills they acquire by participating in the CPLP.

Organizations and system owners must develop a CPLP approach that champions every user’s responsibility to protect information and assets. New technologies and risks will continue to require an organization-wide approach to managing cybersecurity and privacy risks. The NIST Cybersecurity Framework (CSF) [8], Privacy Framework [9], and Risk Management Framework (RMF) [10] highlight the importance of awareness and training for personnel as well as monitoring and improving risk management practices.

While cybersecurity and privacy are independent and separate disciplines, some of their objectives overlap. Cybersecurity programs are responsible for protecting information, information systems, and operational technologies from unauthorized access, use, disclosure, disruption, modification, or destruction (i.e., unauthorized system activity or behavior) in order to provide confidentiality, integrity, availability, and safety. Privacy programs are responsible for managing risks to individuals that are associated with data processing throughout the information life cycle⁵ in order to provide predictability, manageability, and disassociability, as well as ensure compliance with applicable privacy requirements. Managing cybersecurity risks contributes to managing privacy risks. However, managing cybersecurity risks alone is not sufficient, as privacy risks can also arise by means unrelated to cybersecurity incidents, as illustrated in Fig. 2 [9].

Fig. 2. Cybersecurity and privacy risk relationship

For example, an organization may combine de-identified data with a new data set during data processing, resulting in re-identification. Similarly, a de-identified data set may be publicly released but result in re-identification when combined with other public information. The data processing activities do not include a cybersecurity incident or breach but still introduce privacy risks.

Teaching the workforce about different cybersecurity and privacy risks enables them to effectively address the risks they encounter in their daily activities. For example, all members of the workforce will need training that helps them understand what a privacy event is and how one might occur within their organization. They should be able to identify and report a privacy event. Incident response professionals will need training that helps them determine when a cybersecurity incident may also be a privacy event, which often requires additional procedures when responding (e.g., determining if an unsecured site resulted in an actual privacy breach).

Organizations can benefit from a coordinated and flexible approach to developing CPLPs that effectively meet the organization’s needs.⁶

Once an organization understands the relationship between cybersecurity and privacy, it can determine its approach to developing both integrated and cybersecurity- or privacy-specific learning activities based on the relevant topics and workforce roles in its environment. For example, the organization can determine how to effectively:

Members of the workforce with roles that can impact privacy must also have a clear understanding of how to identify and address privacy risks that may arise. The NIST Privacy Framework [9] provides a common language for understanding, managing, and communicating about privacy risks. Just as the workforce considers the risks associated with security events, they must also consider privacy events — the potential problems that could arise from system, product, or service operations with digital or non-digital data through a complete life cycle, from data collection to disposal. Privacy problems can arise from an individual’s direct use of a product. Some problems can also arise simply from individuals’ interactions with systems, products, and services, even when the data being processed is not directly linked to identifiable individuals. The problems that individuals can experience as a result of data processing can be expressed in various ways. The NIST Privacy Framework describes them as ranging from dignity-type effects (e.g., embarrassment or stigmas) to more tangible harms (e.g., discrimination, economic loss, or physical harm) [9].⁷

Individuals may experience privacy problems that arise from data processing, and the organization may in turn experience impacts, such as noncompliance costs, revenue loss from customer abandonment of products and services, or harm to its external brand reputation or internal culture. Organizations commonly manage these types of impacts at the enterprise risk management level. By connecting problematic data actions (i.e., data processing activities that can cause problems) that individuals experience to these well-understood organizational impacts, organizations can bring privacy risks into parity with other risks that they manage in their broader portfolio and drive more informed decision-making about resource allocation to strengthen privacy programs. Figure 3 illustrates the relationship between privacy risks and organizational risks [9].

Fig. 3. Relationship between privacy risks and organizational risks

CPLPs are most effective when they help the workforce understand both the direct impacts of privacy events on individuals and the resulting impacts that privacy risks can have on the organization. For example, a CPLP can address the types of impacts that an individual may experience from the secondary use of data (e.g., for marketing purposes) and the resulting consequences to the organization (e.g., costs associated with non-compliance, loss of trust in the organization, or a shift in mission priorities while addressing the impact).

An organization’s CPLP should coordinate with existing cybersecurity and privacy programs. With limited resources, duplicating efforts will negatively affect one or both programs. In cases where there is an integrated cybersecurity and privacy program, this is less likely to be an issue.

While it is important to understand the policies that require agencies to develop and implement CPLPs, it is also crucial for organizations to understand who is responsible for cybersecurity and privacy education. This section identifies and describes those in an organization who are responsible for ensuring that the workforce has access to and completes their cybersecurity and privacy training.

SP 800-37r2 (Revision 2), Risk Management Framework for Information Systems and Organizations: A System Life Cycle Approach for Security and Privacy [10], identifies the typical roles associated with these programs. Since terminology may vary by organization, it can be useful to refer to the NICE Framework as a complementary tool for identifying those with responsibilities for managing the CPLP and those who require additional training. The NIST Privacy Workforce Public Working Group (PWWG)⁸ also has information about the tasks, knowledge, and skills required of privacy professionals in an organization.

The size, maturity, and resources of CPLPs can vary widely, even within the same organization. The roles and responsibilities for key positions in CPLPs should be documented to help ensure the most effective use of resources and enable the programs to mature to their desired state. As such, each organization needs to consider how to assign these roles. Whether there is a single person serving multiple functions or multiple people collaborating across the organization, be mindful of the possibility of overlapping responsibilities.

The head of the organization is responsible and accountable for information security protections and must prioritize the development of effective CPLPs. This includes implementing a viable cybersecurity and privacy program with a strong learning component. As noted in SP 800-37r2, the head of the organization ensures that “the organization has adequately trained personnel to assist in complying with the security and privacy requirements in legislation, Executive Orders, policies, directives, instructions, standards, and guidelines” [10]. The head of the organization should:

FISMA [2], OMB Circular A-130 [3], and various other regulations designate the responsibility for ensuring CPLPs to certain senior official positions. Titles for these positions include Executive Director, Associate Director, Assistant Director, Division Chief, Chief Information Officer, Chief Privacy Officer, Chief Information Security Officer, Chief Data Officer, Chief Artificial Intelligence Officer, Information Owner, System Owner, Mission or Business Owner, Senior Accountable Official for Risk Management, and System Security or Privacy Officer. These roles are tasked with setting strategic direction, identifying and managing risks, ensuring that resources are available, and overseeing personnel with significant responsibilities for cybersecurity and privacy, including the roles found in appropriate program funding and management.

In addition, senior leaders must champion workforce requirements by:

Agencies and organizations should form a Senior Leadership Committee that meets regularly with CPLP managers to discuss strategy and provide resource support. The CPLP managers will provide the Senior Leadership Committee with regular reports on the CPLP’s performance throughout the year. If there is an emerging risk or new threat that needs to be addressed in the CPLP, the Senior Leadership Committee can connect the learning team with operational teams and SMEs who can provide learning content and resources. Organizations may also include employee representatives as stakeholders in the Senior Leadership Committee to ensure that they have a voice in decision-making.

CPLP managers have tactical-level responsibilities for the CPLP. In this role, the CPLP manager should work with the curriculum development professionals and instruction team, security operations team, information system security officers and managers, CISOs, human resources (HR), procurement, and risk executive teams to:

The term “managers” includes supervisors and those with organizational responsibilities to ensure that the personnel who report to them comply with cybersecurity and privacy learning requirements. Managers should:

The system user is an individual authorized to access information and information systems to perform assigned duties. As noted in SP 800-37r2,

System user responsibilities include, but are not limited to, adhering to organizational policies that govern acceptable use of organizational systems, using the organization-provided information technology resources for defined purposes only; and reporting anomalous or suspicious system behavior. [10]

原本PDFの抜粋画像

A CPLP strategic plan provides an organization-wide view of the current state of cybersecurity and privacy learning, where the organization wants or needs to be, and how to address the gap between those two states (e.g., resources, staffing.) The CPLP strategic plan helps CPLP managers ensure that the organization’s personnel are ready to meet the challenges of the cybersecurity and privacy risks associated with their work.

OMB Circular A-130 [3] establishes a general policy for the planning, budgeting, governance, acquisition, and management of federal information, personnel, equipment, funds, IT resources, and supporting infrastructure and services. Each federal agency is required to develop, maintain, and implement a comprehensive CPLP to meet its mission needs. To develop a robust program with a variety of materials, including offering learners engaging opportunities to stay current on relevant cybersecurity and privacy risks to their organization, the CPLP must have an effective strategy for development, implementation, and continual improvement.

This section discusses the steps involved in building a CPLP strategic plan that takes the organization’s objectives, unique requirements, audience types, and program scope into consideration. The planning stages will also help the organization evaluate its priorities, budget, resources, and communication plans.

The CPLP strategic plan must intersect with the organization’s strategic plan for continual workforce development. The owner of the CPLP should understand the structure and mission of the organization to determine where the organization’s strategy originates. Some agencies are organized with a top-down approach, where a headquarters owns the mission and provides guidance on the CPLP strategy. Other organizations develop CPLPs in various business functions or combine both approaches. Documenting the CPLP strategy and how it supports the goals of the organization’s risk management strategy shows executive leadership why the CPLP is needed.

A well-developed CPLP strategic plan describes how an organization’s risk management and cybersecurity and privacy culture enable all personnel to assess risks with their every action and decision. With agencies of varying sizes, a program that works for one will not necessarily work for another. Each agency must develop a program that will work best for them. The CPLP strategy should always be clearly stated and will most likely be reviewed by the Senior Leadership Committee and agreed upon before any funding is approved.

The CPLP strategic plan should address key items, including:

The CPLP strategic plan should also:

Existing CPLPs may also benefit from a gap analysis or current program assessment to clearly distinguish between the current and target states and enable the program leadership to shape their approach accordingly.

CPLP policies and procedures work together to express what the organization wants to do and how to do it. Policies are clear and simple statements, rules, or assertions that specify the correct or expected behavior of an entity. They provide the guiding principles for meeting the mission and conducting operations, and they can ensure consistent and effective training and awareness methods. Procedures describe how policies will be implemented or enacted. They are written to include who will do what, the steps or phases of the action, defined criteria or implementation levels, and related documentation.

For both cybersecurity and privacy business operations, policies and procedures identify acceptable practices and expectations, as well as guidance for how to train personnel on those requirements and expectations. CPLP policies and procedures should align with broader organizational policies and clearly describe the expectations of the learning programs.

The benefits of establishing policies and procedures include:

The following example policy statements provide context on what is important when establishing, reviewing, or updating CPLP policies.

SP 800-53r5 [11] control AT-1 describes the policies and procedures for cybersecurity and privacy awareness and training (learning) programs.

Organizations can utilize a variety of techniques to identify and describe the steps needed to implement a program. One method is to begin by identifying the organization’s goals, the objectives to meet those goals, and the operational tactics to meet those objectives. Each goal should have objectives that will often include measurable targets, such as identifying who needs role-based training or training a percentage of the organization by a specified date. It is a best practice to choose “SMART” goals: specific, measurable, achievable, relevant, and time-bound. Each program objective will have tactics associated with them. Tactics are tools, methods, or mechanisms that enable the program to pursue the objective identified in the plan’s strategy. Ultimately, every individual item in the plan — down to the most detailed tactical level — can be traced back to where it originates in the overall strategy. Every activity should support the overall CPLP strategy. Managing the steps to implement CPLPs and ensure that the program meets organizational learning needs requires discipline on the part of the team.

Table 1 outlines a model for the strategy, including goals, objectives, and tactics.

Table 1. Elements of a CPLP strategy

The following two scenarios demonstrate each of the implementation steps.

A physical security review of an area in the organization where sensitive data is routinely handled by many employees finds that basic steps are not being taken to maintain a “clean desk.” The privacy policy requires files that contain sensitive data to be kept in folders in locked cabinets. During the review, printed files that contained sensitive data were located in paper stacks and folders that were loosely placed on top of the desks.

The CPLP manager seeks to improve the handling of printed sensitive data and ensure that personnel follow the protection requirements. They determine that a fresh, eye-catching awareness flier or poster might encourage better employee adherence to the policy and reduce this risk.

An executive offers available funding dedicated to producing printed materials. The CPLP manager can utilize that funding to print “Keep It Clean” stickers to attach to work folders and provide a case of such folders to each member of the workforce in the area that handles sensitive data.

In this example, the CPLP manager continually monitored the workplace and risks, coordinated the budget, planned the printing of the stickers, and worked with management to deliver the materials. Because this is a one-time issue, the planning steps were streamlined under the existing program.

A new regulation requires all cybersecurity professionals to implement a specific procedure in their daily routines.

The CPLP manager works with the cybersecurity policy owners to understand and interpret the regulation. Together, they define a strategy with goals, objectives, and a set of program tactics to provide new training to all members of the workforce and meet the new requirements with specific new procedures. The CPLP manager decides to work with organizational training staff to create an online experience that would also enable remote workers to fully participate.

As the course is being completed, the CPLP manager works with organizational leaders and management to identify expected measures of completion and success and to ensure that all necessary members of the workforce are identified and trained. As an element of continuous monitoring, the CPLP manager works with the learning office and leadership to test the completion and success of the training.

Program measurements and metrics are the key drivers for the Assessment and Improvement phase of the CPLP life cycle (see Sec. 1.3). CPLP measurements enable an organization to describe and quantify the learning program, show the effectiveness and impact of the program, understand where changes are required for success, meet budgetary and resource requirements, and make data-driven decisions. Through the development and monitoring of CPLP measurements, an organization can better address their learning goals and resources.

For CPLP purposes, it is necessary to differentiate between quantitative and qualitative measurements. A quantitative measurement is assessed by assigning a number or category to an object to describe an attribute of that object, such as using a 1-5 scale to assess the effectiveness of learning content. For example, a post-learning program assessment instruction might state: “Rate your knowledge level of social engineering exploits on a scale of 1-5, where 1 is a low level of knowledge and 5 is a high level of knowledge.” Such a measurement enables the learning content to be evaluated with an average or mean score.

A qualitative measurement is based on descriptive data, for example, data collected through observations, interviews, focus groups, or open-ended text fields in surveys.

Developing and establishing CPLP measurements requires careful consideration and should be driven by the program’s goals and objectives as well as any regulatory requirements. CPLP managers should be prepared to answer some common questions, such as:

The CPLP manager should identify how the measurements will be collected, how frequently, who should have access to the measurements, and how the reports will be documented and shared. While laws, regulations, and policies often set specific measurable requirements, CPLP measurements must go beyond simply achieving compliance. The CPLP’s impact on workforce capabilities, attitude, and behavioral changes must also be measured.

The measurements and metrics should be tied directly to the learning goals, objectives, and outcomes of the CPLP program and organization. While these terms are often used interchangeably, they each have a specific meaning with regard to the designation and collection of measurements and metrics. Whether it is at the organizational, course, module, or lesson level, these terms retain their distinct meanings. For clarification, this document provides the following high-level definitions for these terms and describes how they are used in the learning community:

After participating in the learning activity, learners will be able to (insert o measurable verb) and (insert learning statement).

Table 2 shows examples of learning goals, objectives, and outcomes and how their differences impact the generation of measurements for a program.

Table 2. Examples of learning goals, objectives, and outcomes

CPLP managers should build programs with efficient data gathering techniques to provide effective reporting information. This will likely include collecting data on employees that may carry a heightened sensitivity due to context (e.g., training records are often part of employment or contract records and can be tied to performance evaluations or result in consequences for failure to take required training). CPLP managers must identify and manage the cybersecurity and privacy risks associated with processing learning data, including risks associated with learning management systems (LMSs) and reporting practices.

Quantitative measurements are expressed in numerical form and are usually referred to as objective data. The combination of quantitative and qualitative measurements provides a holistic understanding of the CPLP.

Some common examples of quantitative learning program data include:

Qualitative measurements are based on descriptive data, for example, data collected through observations, interviews, focus groups, or open-ended text fields in surveys and can provide insight into the learning experience.

Some common examples of qualitative learning program data include:

Developing metrics can be one of the most important yet challenging parts of the CPLP effort. How the organization defines and collects measurements will lead to the definition and analysis of the metrics. An effective set of metrics can help improve content quality and learner engagement, obtain support from the organization, increase funding, reveal impacts on the cybersecurity and privacy risk management program, and demonstrate returns on investment. In recent learning program research efforts by NIST, participants reported [6] that despite best intentions, their organizations often used a limited number of metrics that did not provide a complete view of program effectiveness.

The following examples show how measurements provide the data for metrics:

Course Attendance

Course Completion

Engagement During Courses

Course Cost per Participant

Behavior Change per Participant

CPLP managers rely on CPLP measurements and metrics to assess the effectiveness of their learning programs and ensure that the learning outcomes are achieved. SP 800-55v1 [12] provides guidance on the selection and aggregation of information security measurements and the development of an information security measurement program.

When considering the organizational personnel whose learning objectives will be addressed with the CPLP, it is recommended to view each individual as belonging to one or more learning program audience segment. Every person who participates in the CPLP is counted in the “all users” group. The second group includes privileged access account holders who have organization-approved access to restricted systems or data that requires special care. These individuals will receive additional cybersecurity and privacy training to minimize risks to systems and data. The third group includes employees and contractors with significant cybersecurity and/or privacy responsibilities who require an individualized program of role-based training to ensure that their knowledge and skills are sufficient to execute the tasks required for their work. There will likely be some overlap between the privileged access account holders and those with significant cybersecurity and/or privacy responsibilities. Figure 4 shows the groups and how they overlap.

Fig. 4. CPLP learning program audience segments

In a typical scenario, all of the organization’s personnel will participate in the CPLP, agree to abide by the acceptable use policy or standards of behavior, complete the recurring¹⁰ learning program training, and attend, complete, view, and receive the other various ongoing program elements.

SP 800-53r5 [11] control AT-2, Awareness and Training, refers to “all user” training as cybersecurity and privacy “literacy” training. As part of or after completing the recurring training, users will sign a rules of behavior policy that defines the behaviors required to gain and keep system access. SP 800-53r5 also indicates that the training will need to be updated for any system changes or following any organization-defined events:

Subsequent literacy training may be satisfied by one or more short ad hoc sessions and include topical information on recent attack schemes, changes to organizational security and privacy policies, revised security and privacy expectations, or a subset of topics from the initial training. Updating literacy training and awareness content on a regular basis helps to ensure that the content remains relevant. Events that may precipitate an update to literacy training and awareness content include, but are not limited to, assessment or audit findings, security incidents or breaches, or changes in applicable laws, Executive Orders, directives, regulations, policies, standards, and guidelines.

原本PDFの抜粋画像

All users in the workforce (commonly referred to as “system users” and “general users”) are critical to reducing unintentional errors and vulnerabilities. The organization’s personnel may include full-time and part-time employees, contractors, consultants, foreign or domestic guest researchers, visitors, guests, other agency personnel, and other collaborators or associates who require access. All users must:

Examples of topics that the CPLP may address include:

The organization may also be able to identify those who are subject to or who pose greater human risk due to their role, the applications they use, or simply because they have not yet committed to adopting best practices for cybersecurity and privacy. The CPLP manager — in consultation with the individual’s manager and the IT department — should develop additional mechanisms for identifying risky behaviors and metrics for evaluating participation in the “all user” learning program and consider ways to incentivize behavioral change in positive, non-punitive ways.

Individuals with privileged access accounts are trusted with additional access or responsibilities to perform functions that ordinary users are not authorized to perform, such as configuring network management and granting system access (e.g., system administration privileges). Due to their ability to access critical resources, privileged access account holders require additional training to ensure that they understand their account access privileges and do not accidently cause or exploit vulnerabilities or misuse data. For each type of privileged access account, the CPLP manager must coordinate training with the account holder’s manager or supervisor, human capital officer, and training managers to ensure that training is delivered and kept current.

Personnel with significant cybersecurity and/or privacy responsibilities — including some privileged access account holders (e.g., “super users”) — have rights or access to sensitive or critical systems and will require additional training. For example, a manager of an HR environment who ensures that the system is properly configured and available or a Chief Human Capital Officer (CHCO) who collaborates on setting policies for the system will have significant cybersecurity and privacy responsibilities. Network and IT administrators who manage the system and network access are privileged access account holders. If their work roles include monitoring for data loss or other privacy and cybersecurity issues, then they also have significant cybersecurity and/or privacy responsibilities that require additional role-based training. This access can be rescinded when a holder’s work role changes.

SP 800-53r5 [11] control AT-3, Role-Based Training, provides a definition for the training required:

Comprehensive role-based training addresses management, operational, and technical roles and responsibilities covering physical, personnel, and technical controls. Role-based training also includes policies, procedures, tools, methods, and artifacts for the security and privacy roles defined. Organizations provide the training necessary for individuals to fulfill their responsibilities related to operations and supply chain risk management within the context of organizational security and privacy programs. Role-based training also applies to contractors who provide services to federal agencies.

原本PDFの抜粋画像

This training is typically associated with job duties determined by organizational leaders (e.g., the agency’s CIO, CPO, or CISO) and the employee’s manager or supervisor and is documented in the employee’s performance plan. Personnel in these work roles may require professional development to maintain their status or memberships, such as annual certifications or courses. Examples of typical role-based training recipients include the CISO, privacy officers, cybersecurity managers, cybersecurity and privacy analysts, and incident responders. Cybersecurity work roles and competency areas are explored in SP 800-181r1 [5], which describes the knowledge, skills, and tasks associated with cybersecurity-related work.

FISMA [2] requires personnel with significant cybersecurity and/or privacy responsibilities to receive role-based training. Additional guidance can be found in SP 800-37r2 [10], SP 800-53r5 [11], and SP 800-181r1 [5]. In combination, these documents assist with the identification of roles and functions in the cybersecurity workforce that require role-based training.

Determining who in the organization will participate in role-based training is a multi-step process that begins with defining the significant cybersecurity and privacy work roles in the organization and identifying the staff who are aligned with the designated work roles. Often, the determination begins with senior leadership and direction from the office of the CIO, CISO, or CPO in partnership with HR. CPLP managers should participate closely in this effort to identify those with significant cybersecurity and privacy responsibilities.

These work roles should also be included in position descriptions, hierarchy charts, and responsibilities to show how the work required to achieve a particular objective has been identified. Individuals may assume additional work roles based on their particular skills, organizational policies regarding cross-training, and organizational staffing levels. SP 800-181r1 [5] identifies work roles for cybersecurity and is a detailed lexicon for understanding the related knowledge and skills that are typical for such roles.

The ultimate goal of a CPLP is to reduce cybersecurity and privacy risks to the organization through cultural and behavioral changes, not simply to achieve compliance.

The complexity of the material must be determined before development begins and be commensurate with the role of the person who will undergo the learning effort. Material should be developed based on two important criteria: 1) the target attendee’s role and 2) the cybersecurity and privacy responsibilities required for that role. This will require coordination with HR and CLOs (or equivalent). Individuals who receive training will appreciate the effort made to ensure that they understand the material in a manner appropriate to their learning needs and the nature of the work that they do.

A typical CPLP includes a variety of learning program elements that are delivered to diverse audiences through various platforms and methods. CPLP managers will identify the necessary and most effective types of program elements for each audience type per learning goal and adjust their selections to match their available budget and schedule considerations.

The typical CPLP elements are:

Employees may initially believe that the CPLP is the annual or regularly scheduled training course or event delivered to all users, which may include informal department programs, a mandatory presentation delivered in an auditorium, or an online course. Other learning program elements are targeted for those with significant cybersecurity and/or privacy responsibilities and those who are privileged access account holders. A CPLP program will consist of the mandatory elements (required by policy and learning objectives for all CPLP learning participants) and the many other activities implemented throughout the CPLP life cycle to reinforce these messages.

Regardless of the learning activity, the learning goal for these events is to ensure that personnel are aware of their roles and responsibilities for protecting information and assets and are able to take appropriate action to respond to a variety of cybersecurity and privacy risks.

Cybersecurity and privacy awareness learning activities should be conducted on an ongoing basis throughout the year to ensure that employees are aware of their roles within the organization and the appropriate steps they must take to protect information, assets, and individuals’ privacy. Activities can be campaign-oriented or ad hoc based on the subject matter, threats, or vulnerabilities or take place during seasonal events.

Examples of awareness activities that are appropriate for all users include:

Experiential learning activities or practical exercises are specific learning scenarios that simulate events and incidents. The practical exercises can include social engineering efforts (e.g., phishing, smishing, and vishing) in the form of simulations, learning games, quizzes on identifying and processing data, tabletop exercises, hands-on virtual lab exercises, contingency plan and disaster recovery scenarios, and attack or defend scenarios conducted in cyber ranges.

For example, in an organization-wide all-user phishing exercise, a “tricky” email is sent to users to see whether they can spot a phishing attempt or if they can be tricked into clicking on a link to a malicious website or opening an infected attachment. Phishing exercises can also target learning for those in specific roles, such as leadership or known administrators. Begin by determining the current click rates for a specific group of users or by work roles to understand if there is a need for more targeted training. The NIST Phish Scale can be helpful in identifying and measuring behaviors for phishing threats.¹¹ These exercises offer opportunities to collect metrics and measurements, which are usually referred to as “click-through” or reported measurements. These types of measurements indicate whether the user reported the email as a phishing attempt or whether they clicked on a link or opened an attachment. Some organizations include “report phishing” capabilities on their email platform (e.g., a button on the platform’s menu) to encourage best practices.

The organization’s legal team should be included in the design and review of planned phishing exercises to avoid negative impacts, such as using legitimate brands or naming federal organizations in the phishing “bait,” which could result in emails or calls to those entities. In addition, since employees may not like being tricked, it is important to tell employees that the organization is conducting phishing exercises on a random basis and that the results will be used to guide future learning activities. These activities should not be punitive, nor should any employee be called out for their response. When viewed as learning opportunities, the phishing exercises can provide important data on vulnerabilities and which employees may need additional learning support.

Other experiential learning exercises may be better suited for those with significant cybersecurity or privacy responsibilities (e.g., role-based training), such as tabletop exercises and contingency plan scenarios. Additional examples may be found in NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities [13].

Training is a broad term that includes the learning program content designed to increase or improve job-related knowledge and skills. Some of the techniques that an organization can employ include:

Blending various training delivery techniques can be an effective way to present material and hold an audience’s attention. For example, showing videos during an instructor-led session allows the audience to focus on a different source of information. The video can also reinforce what the instructor has been presenting.

During the planning phase of the CPLP life cycle, evaluate the organization’s critical risk factors to determine the learning priorities. If a phased approach is necessary due to budget constraints or resource availability, some factors to consider are:

The CPLP plan defines sufficiently detailed program elements that support the strategy for each activity or campaign. Many organizations utilize standard program plan templates that provide baselines for organizational expectations. The exact level of detail within the plan will vary, depending on organizational and program requirements and resources. As the program matures, the CPLP manager should conduct recurring reviews (i.e., at least annually) of the plan, along with the stakeholders and individuals who will support and manage the program.

An important element of developing the CPLP strategy is to determine what currently exists within the organization and what resources are dedicated to existing programs. If the learning program does not yet exist or requires significant redesign or updates, refer to the program strategy process (outlined in Sec. 2.1) to review the most important program elements for inclusion. Resources are typically defined as any asset that is required to meet goals and objectives, such as people, materials, equipment, and technology. An important consideration in obtaining resources is establishing a CPLP budget.

Once the CPLP strategy has been approved by senior leadership (identified in Sec. 1.6.2) and priorities have been established, funding requirements must be added to the plan. A determination must be made regarding the extent of funding support to be allocated based on the strategic goals. Senior leadership should help the CPLP manager understand or establish their budget. While each program will have different funding needs, some typical costs include:

Organizations with a limited (or even no additional) budget for cybersecurity and privacy initiatives can still establish an effective CPLP by leveraging existing resources, implementing cost-effective monitoring solutions, and prioritizing cybersecurity and privacy as a shared responsibility across the organization. Some materials may be available from other federal agencies, partner organizations, or online vendor resources. Some materials may already exist in-house and should be inventoried and evaluated to determine whether they are current and meet the existing learning goals. The implementation timeline will help indicate when additional funding may be required to support tools, major curriculum and content deliverables, new staffing requirements, and other learning program elements and activities.

The following are example questions that can help guide the development of budget requirements:

CPLP managers must work with senior leadership to advocate for the program against competing priorities, budget constraints, and staff time costs and to develop a strategy to address any shortfall in funding that may impact the organization’s ability to meet its learning goals. This may require adjusting the learning strategy to be more in line with the available budget, advocating for additional funding, or reallocating current resources. It may also mean that the program plan needs to be phased in over some predefined time period as funding becomes available.

Those who have managed federal CPLPs report that training the workforce requires a combination of technical knowledge and professional attributes, such as communication, creativity, and interpersonal skills [7]. If the organization does not have the budget for CPLP course developers, research what other agencies or organizations of similar size have done to meet their own needs. Some organizations may have in-house instructional designers, curriculum developers, instructors, web developers, communications experts, and graphic designers. Other organizations may need to include these professional costs in the budget for a new project and identify qualified contractors or external courses.

Different information requires different methods of delivery. Some program elements will be appropriate to deliver via online learning, while others will necessitate both instructors and physical classroom locations. Determining these requirements in the planning stage will allow for appropriate resource allocation (e.g., rooms to be reserved, computers and projectors secured, etc.). Even posters and flyers require space considerations, as they will need to be displayed in a sufficiently prominent area to have a learning impact on personnel.

One of the most important aspects of executing the CPLP strategic plan is collaborating with the learning team, key stakeholders, senior leadership, and personnel. Involving stakeholders and employees during the planning process can lead to greater success as the program begins and as each program element is implemented. Determining what to communicate should focus on:

Obtaining early and continual support for the strategic plan is important to keep the momentum for the CPLP strong and inspire engagement and satisfaction with the plan. A solid communication strategy will address those needs. Consider whether the organization has a centralized communications department or whether communications decisions will be made at the business unit level. Then develop a communications plan (or incorporate these elements into an existing communications plan) to share information about the new or updated CPLP. Keep it simple and tailored to internal stakeholders. The CPLP manager may choose to create a custom version of the strategic plan that includes different information for different audiences.

Some important elements to share include information about what the CPLP is and who manages it. Funding issues and gaps may also need to be identified and addressed. For example, agency leaders and managers need to know whether the cost to implement the CPLP activities will be funded by the CIO, CISO, CLO, or another program budget or whether their budgets will be impacted to cover a portion of the expense. In addition, schedules and completion requirements must be communicated.

The CPLP communications plan should include:

Everyone involved in the implementation of the program must understand their roles and responsibilities. Most organizations may find it helpful to tailor their messaging based on the audience. A few examples of audiences and their roles include:

While there are many theories and models for the design and development of training programs, this document focuses on the ADDIE model¹² — a traditional instructional systems design model with five distinct phases:

During the analysis phase, the CPLP manager identifies the organizational and learning needs or gaps to determine which audiences will need training and their existing levels of knowledge and skill. It may be necessary to evaluate various work roles for competency gaps so that relevant learning programs can be customized and created based on the specific learning and skill needs for that work role. During the design phase, these gaps are translated into learning objectives, which are the focus of the learning material. Tying the learning objectives to identified knowledge and skills gaps ensures that the end result is relevant and will succeed in addressing the identified learning needs.

The analysis phase is the process during which the CPLP manager determines the organization’s learning and performance needs. In this context, the needs (i.e., gaps) are the difference between the current learning goals (or activities) and the desired state. To determine their learning needs, organizations may conduct a formal or informal needs assessment (also referred to as a needs analysis). The primary benefit of the analysis phase is to identify learning needs for the organization and learning audience. Additional benefits include having information that clearly defines the learning needs, support for and prioritization of resources, and the alignment of learning goals to organizational mission goals.

In the beginning of the analysis phase, it may be helpful to identify the primary members of the analysis team, including several additional constituent groups. This may include:

development of an organizational culture focused on cybersecurity and privacy protections.

There are many reasons why the analysis phase is rushed or skipped entirely. For example, organizations may think it will take too much time, personnel may be unavailable, or the necessary funding may be lacking. Most often, organizations believe they already know what they need. However, critical problems can arise by skipping the analysis phase, such as:

It may be tempting to skip the time needed to properly analyze the organization’s needs. Even if the only option is to conduct an informal discussion and review with a few individuals, it is still important to have the conversation and document what is needed. The analysis phase establishes a clear vision for the next steps of the CPLP’s development.

While there are many ways that CPLP managers can evaluate the learning needs of the organization, the process for identifying learning program needs from a strategic point of view tends to be a repeatable process, regardless of the specific learning goal or audience. The steps are:

The most important step in initiating a new phase in the CPLP is to establish the learning needs. For example, the organization may be prepared to introduce new technology, legislation may have been passed that requires personnel to acquire new knowledge or skills, or a new privacy or cybersecurity risk may have emerged that requires the organization to introduce a new learning module. Identifying and prioritizing learning needs will allow the CPLP managers to focus their attention on the issues of greatest importance to the organization.

The following techniques can help define the learning needs:

During the analysis phase, the learning program manager will identify and define the audiences to be trained on the learning goals. It may be helpful for supervisors to coordinate with the organization’s cybersecurity and privacy learning function to determine whether personnel need additional training.

Potential learning audiences for the CPLP include:

The following examples show how personnel can be assigned to multiple learning programs:

The primary knowledge and skillset for the “all users” learning audience segment is the ability to recognize cybersecurity and privacy risks, take appropriate actions to reduce harm to the organization, and report any incidents or events, when appropriate. All users must be empowered and skilled in adhering to the organization’s rules of behavior and acceptable use policies, which include guidance on how to use organization-provided devices and access network resources.

Privileged access account holders must be able to judge risks appropriately and use systems that they have been given access to without introducing additional risks or harm to the organization. The NICE Framework [5] can be a useful resource for identifying the necessary role-based knowledge and skills for those with significant cybersecurity or privacy responsibilities. An organizational job analysis will also be useful in determining the learning objectives for program participants. For those with significant privacy responsibilities, the CPLP manager should consult with managers, SMEs, and the privacy senior leadership of the organization (i.e., CPO or SAOP) for additional guidance on the knowledge and skills required of individuals.¹³

There are existing models for evaluating the tasks necessary for a particular person’s role, such as considering the complexity or difficulty of the task, its importance, and how frequently the task is performed. This is sometimes referred to as the “DIF model” for considering the relative difficulty, importance, and frequency of the task. It can be helpful for identifying the knowledge and skills that the CPLP should focus on when training those with significant cybersecurity or privacy responsibilities.

The next step in the analysis phase is to determine what the audience segment already knows about the topic and what skills they possess while keeping the learning goal in mind. The CPLP should focus on providing the learner with the requisite amount of new knowledge and skills while reinforcing existing knowledge and skills. There are several methods for determining the existing knowledge and skill set:

These methods can also identify whether new training is needed for a role or whether existing training needs to be updated or modified.

Thus far, the analysis measures personnel’s existing knowledge and skills with an overview of each audience segment. The difference between that and the ideal state of knowledge and skills for the learning goal is referred to as “the learning gap.” During the design phase, the CPLP manager will use information about each learning gap (per learning goal, learning audience, etc.) to design a program specific enough to address each need.

At the beginning of the design phase, consider what knowledge and skills the audience needs to learn or develop and what gaps the learning material will close. This will drive the creation of the learning objectives and the process for achieving them. The design process should end with a systematic blueprint of the approach needed for the CPLP to address the identified knowledge and skills gaps.

The CPLP manager begins a formal design phase for the CPLP or a new element in an ongoing CPLP by creating a Design Document that outlines the requirements. Then it will be necessary to determine whether they need to build or buy learning materials to satisfy those requirements. The steps in the design phase are:

The Design Document provides a blueprint for the development and implementation of the learning program elements. It is usually created by the CPLP manager and reviewed by key stakeholders (when necessary for funding and other approvals) before moving to the development phase.

Typical elements of a Design Document include:

Based on its resources, the organization will determine whether it can build, have built, or utilize existing government or commercial-off-the-shelf (COTS) learning content, which is discussed further in Sec. 4.

CPLP managers will need to determine what training materials have previously been used in their organization and are still available and appropriate for use. Additionally, there may be materials and programs available from elsewhere in the organization, agency, or partner agencies. Federal resources may have materials, presentations, and even speakers available to satisfy a variety of learning goals. An important result of the survey effort will be insight into what is currently being done to meet learning requirements in the organization and the gap in needed program materials. The content should be adaptable to fit the learning program participants’ needs and the organizational context with reasonable effort or costs.

There is a variety of external sources of CPLP materials available. Some possible sources include:

Sources of timely material may include:

CPLP managers can build new partnerships or reinforce existing ones with the organization’s functional managers who coordinate or conduct their own learning programs. Functional training developed in-house (e.g., financial applications or personnel management) often lacks adequate discussion of related cybersecurity and privacy issues. Through these cross-departmental partnerships, CPLP managers can review existing references to topic areas in the materials, check for completeness and accuracy, and assist the functional manager by developing a learning module for any material that previously had no cybersecurity or privacy component.

CPLP managers consolidate what they have found in available materials to identify the learning objectives for the CPLP. Whether the CPLP manager is working on the entire plan, designing a few new elements, or updating existing elements, this stage can be very useful in ensuring that the effort is closely aligned with identified organizational needs.

Consider the following examples of identified training gaps and their associated learning objectives.

Scenario 1: A recent analysis indicated that on-site, remote, and teleworking employees — including employees with privileged access accounts — are using single-factor authentication (i.e., a password). The CIO has approved the implementation of a multi-factor authentication token system starting with privileged access accounts in the first quarter and all other accounts in the second quarter. The CPLP manager has been tasked with helping employees understand their roles in utilizing this new multi-factor authentication system.

Analysis Phase: Identify Knowledge and Skills Gaps

Since this is a new authentication method, the “all users” and privileged access account holders learning audience segments need information and training on the new policies, processes, and procedures for accessing the system. In addition, they need to know why this effort is important or how it protects the information and assets on enterprise systems. Privileged access account holders will also need additional information focused on the specific privileges they will have once authenticated to the system.

Design Phase: Create Learning Objectives Based on Knowledge and Skills Gaps

The learning goals and objectives for the program must then be established. In this example, the goals and objectives for the learning program involve enabling employees to:

Scenario 2: A recent external audit of the organization’s system privacy policies and practices found that the financial office employees were not adequately protecting the privacy of employee bank information when processing the employees’ travel costs.

Analysis Phase: Identify Knowledge and Skills Gaps

During the analysis phase, the learning program manager determined that financial office employees are designated as employees with significant cybersecurity and privacy responsibilities and receive an annual one-hour self-paced training course. Based on the analysis, it was determined that the financial office employees lacked a basic understanding of the policies and procedures for protecting sensitive and privacy-related information. Since this could have immediate and damaging consequences, this lack of knowledge will be addressed with a customized training solution delivered immediately and by including the topic in updates to the annual one-hour self-paced training course.

Design Phase: Create Learning Objectives Based on Knowledge and Skills Gaps

A webinar was scheduled with the financial office employees to provide immediate direction on corrective actions for the protection of sensitive employee bank information. The learning goals and objectives are:

Before moving to the development phase, the CPLP manager must consolidate the requirements for development using the results of the analysis and design phases. It should be possible to fully articulate the competency gaps being targeted by the learning program participant segment and the related learning objectives.

The following CPLP requirements are also important to consider:

In the development phase, each audience’s requirements are evaluated, budgeted, and provided for separately. The previously identified audience requirements and learning objectives are the guiding factors for developing the content. Some organizations may have existing and mature awareness, training, and education programs that can provide content or influence the development and implementation phases. Established best practices and lessons learned should be considered when developing the learning content.

The development process will involve various personnel, including:

Once the baseline requirements of the program have been solidified, a feedback strategy can be designed and implemented to ensure that materials continue to support the CPLP strategy and address identified training needs.

After the CPLP managers complete their analysis and design reviews, they will have a comprehensive set of design documents to guide the development of new materials. These documents are useful when allocating budgets and personnel for the creation of new materials or program elements. However, additional information will be needed to guide the content creators in their work.

The CPLP manager will need to create a requirements document if it is necessary to create or source new CPLP content, curricula, or other program elements. The requirements document incorporates the information from the design document as well as any additional and necessary information to provide to the training and curricula developers, editors, and designers, whether they are in-house or vendors. The requirements document will also be useful for the organization’s acquisition and budget functions.

The requirements document generally provides detailed and specific criteria related to the content needed to meet the learning objectives. Typical prompts or questions to review when creating the requirements document for any new CPLP material include:

The learning program criteria for the requirements document may include the following considerations:

The “all user” learning program elements (see Sec. 2.5.1) are delivered throughout the year. However, there may be necessary updates to, and iterations of the content based on events and organizational requirements. Ensure that the budget is allocated to update the content or amend the materials with other delivery methods (e.g., to video training) if making actual content changes is cost prohibitive.

Developing a dynamic and effective “all user” learning program — and particularly, the cybersecurity and privacy presentation — is challenging, especially when learners arrive with an expectation that they only need to do the bare minimum to fulfill their training requirement. However, given the ever-changing nature of cybersecurity and privacy risks, there will almost always be new and crucial content for them to understand.

Consider how key messages will be reinforced throughout the “all user” learning program. When necessary, refresher messages may be sent to reinforce a change in behaviors. For example, if the results of a phishing campaign show higher than usual click-through rates, it may be necessary to repeat the campaign using different mediums to ensure that the audience is aware of expected behaviors on the system. Repeated messages become retained messages. Use the awareness program materials to keep the “all user” learning program topical without becoming monotonous or intrusive. This is a tricky balance to achieve and requires a variety of delivery formats and messages. Consider varying the awareness program techniques, such as sending out cybersecurity or privacy topic emails on a monthly basis, adding a campaign message to everyone’s official organization signature block for Cybersecurity Awareness Month in October or Data Privacy Week in January, or place posters in the agency’s lunchroom all year round.

Choosing techniques for disseminating cybersecurity and privacy awareness messages throughout an organization depends on available resources and the complexity of the messages. Some techniques that are appropriate for a single message include posters, screensavers, warning banners, organization-wide emails, brown bag seminars, and awards programs. Techniques that can more easily include several messages or themes include “do and don’t” lists, email newsletters, web-based sessions, teleconferencing sessions, in-person instructor-led sessions, and email signature messaging. Examples of awareness material can be viewed on the Federal Information Security Educators (FISSEA) website¹⁹ under “FISSEA Security Awareness and Training Contest.”

Additional considerations when developing the “all user” learning program include:

self-assessments, and spot-checks can also help learning program managers identify additional topics to address.

Developing new materials or elements for privileged access account holders is similar to developing them for the “all users” segment. Create a requirements document that aligns learning goals for this audience segment with available funding and organizational requirements.

Additional considerations for developing new material for the privileged access account holders learning program include:

Some starting points include: understanding the rights and privileges allotted to this learning program audience segment; reviewing the risks related to privileged access accounts or the systems or applications associated with privileged access; reviewing these issues with the offices of the CIO, CISO, and/or CPO; and aligning learning goals for these risks to the available budget for impacted personnel and departments. Evaluating organizational policies, program reviews, internal audits, self-assessments, and spot-checks can also help CPLP managers identify additional topics to address.

The customized and individualized nature of ongoing skills development and training for personnel with significant cybersecurity and/or privacy responsibilities requires a more detailed and nuanced learning program approach. For example, multiple requirements documents may be needed to develop new learning program elements and identify training that will satisfy learning objectives. The CPLP manager should partner and coordinate these efforts with the organization’s human capital office, CLO, training and curriculum developers, and the individual managers and supervisors for the personnel in this learning program audience segment.

CPLP managers should ensure that the complexity of the training is commensurate with the role and needs of the people who will undergo the learning effort. Cybersecurity and privacy role-based training materials can be developed at a beginning level for a person who is just learning a discipline. Materials can be developed at an intermediate level for someone who has more experience and, therefore, more responsibility in their workplace. Advanced materials can be developed for agency SMEs whose jobs incorporate the highest level of trust and an accompanying high level of cybersecurity or privacy responsibilities.

CPLP managers should evaluate available materials from a variety of external sources, including federal programs and repositories, industry vendors, and academic institutions. Some of these courses or learning elements may be low cost or even free.

Where budget permits, CPLP managers may choose to acquire (or license) a vendor-provided library of courses and align, curate, and recommend a set of those courses for the learning program audiences or certain work roles. Acquisitions of CPLP materials should be guided by the same learning criteria identified for the creation of new elements (see Sec. 3.3.1). CPLP managers should also consult with their organization’s agency contracting office.

Include a learner testing (“user testing”) phase for all new CPLP elements prior to implementation. Content should be assessed for each learning program audience segment to ensure that it meets their needs and is appropriate for their skill level. Additional learner testing might include evaluating the intended element’s delivery method, the appropriateness of the language, the value to the learner, and overall acceptance of the new element. Feedback from learner testing should be iterative and incorporated at every step of the design effort, not just in the form of evaluations after implementation.

Implementation refers to the actual distribution and delivery of the CPLP materials. This phase focuses on the connection between the learner and the content. Once the plan for implementing the CPLP has been communicated to and accepted by management (see Sec. 2.11), the implementation phase can begin. Use a life cycle process when implementing the program to avoid a “one and done” scenario, and periodically review the program for updates and corrections.

CPLP managers should implement a new learning program or a single element with the same repeatable steps. It is of the utmost importance that all of those involved in the implementation phase be included in a well-designed communications effort. This ensures that personnel and their managers or supervisors are well-informed about any upcoming CPLP opportunities that are relevant to their required learning plans. The implementation phase is also the time to confirm that the required reporting and metrics can be satisfied in later program phases. Steps to consider before initiating the implementation phase include:

Communication is a large part of developing an organization’s shared culture of supporting the learning program efforts. CPLP managers should develop a communications plan for each phase of the program element implementation and include the organization’s communications team. CPLP managers should also determine the appropriate timing to inform managers, supervisors, and possibly the personnel involved about upcoming and required learning program elements, as well as the frequency with which to send out reminders and other forms of communication that encourage cooperation from the organization.

Each individual CPLP element (e.g., presentation, course, or tabletop exercise) requires a separate and more detailed form of communication to inform learners and their managers of important details regarding the learning event. This includes items such as:

The communications plan should include a clear explanation of why the learning program is mandated (if applicable) and the benefits of attending (i.e., the learner outcomes). Applicable federal legislation, regulations, and agency or organizational policies should be referenced.

Each learner audience must be specified for the training assigned. For example, if the organization’s policy states that all system users must complete a particular training to gain or maintain access to enterprise systems, the communications plan must include this notice. For those with significant cybersecurity or privacy responsibilities, identify which training is assigned to a specific work role, individual, or department.

Employees must know the consequences of failing to complete the learning activity according to the organization’s policy. This should be explained in the learning program communications plan and noted in the course description in the learning plan within the LMS.

Other considerations for CPLP communications include:

As noted in Sec. 2.4, CPLP managers should strive to ensure that the implementation of all new CPLP elements (e.g., courses, training, posters, practical exercises, etc.) will allow for performance metrics and measurements to be established and collected. Review these requirements during the developmental phase to ensure that the measurements still meet expectations, such as regulatory and annual reporting requirements, learning objectives, and learner outcomes. The goal is not simply to meet compliance requirements but to enable an ongoing development effort for the CPLP. As previously mentioned in Sec. 3.2.2, the measurements and metrics are included in the design plan requirements used by curriculum and content developers.

Some considerations for reporting measurements and metrics include:

As a final step, the CPLP manager will meet with the Senior Leadership Committee to review the performance of the program, address new organizational risks or concerns to include in the training program content, and identify any areas for significant improvement. Section 5 provides more information on how measurements and metrics help to ensure that the CPLP life cycle is an ongoing and continual effort.

Establish a primary calendar for CPLP activities, which may be automated using an LMS. Enable organization-wide access so that personnel can find elements that are applicable to each audience segment (e.g., by date, learning objective, etc.). Align this calendar with organizational availability, such as observed holidays, planned events, major IT releases, and other considerations that might cause learner conflicts with the CPLP schedule. As part of the communications plan, send out reminders to ensure that instructors and materials are identified and allocated well in advance.

Once any CPLP element has been delivered or implemented, the CPLP manager should initiate post-implementation activities that fuel assessment and improvements.²⁰ Some awareness elements, such as measuring learning program participant engagement, may be difficult, especially for passive items like posters or email signatures. Nevertheless, it is possible to measure impact. One method could include surveying a sample of learners to discuss their familiarity with the messaging or whether they have practiced any of the tips.

Some key items to consider during the implementation phase are:

An effective CPLP meets the needs of learners and the organization by measuring and evaluating program performance on a continual basis. This process requires up-to-date knowledge, awareness, and understanding of the legal and regulatory compliance requirements for the organization and cybersecurity and privacy risks. CPLP managers work with organizational leaders, training staff, and learners to share performance reporting and decision-making throughout all phases of the CPLP. Analyzing organizational risks (e.g., decrease in the number and frequency of common issues that the learning program has targeted over time) and reviewing the efficacy of materials (e.g., learner feedback responses to courses) are important in the continual improvement of a CPLP in an evolving threat landscape. Remember to choose measurements that can be quantified, such as SMART goals (i.e., specific, measurable, achievable, relevant, and time-bound).

The process for assessing and improving the CPLP may vary by organization and available resources. Consider the following steps as part of evaluating the CPLP’s performance, whether for the entire CPLP, for each audience segment, or for a single CPLP element.

At the end of a campaign and regularly throughout the year as agreed upon with the Senior Leadership Committee, CPLP managers should create a summary document to review with senior leadership. This report will provide an analysis of attendance, feedback, measurements, and other metrics and help to identify action items, areas of improvement, and next steps. It should be tailored for senior leadership using appropriate language and framing (e.g., avoid using technical jargon without explanation).

A CPLP assessment report includes:

Instructor evaluation (Sec. 5.2.3.1) o Content evaluation (Sec. 5.2.3.2) o Learner, supervisor, and organizational feedback (Sec. 5.2.3.3) o

Measurements and metrics are the key drivers for the assessment and improvement phase of the CPLP life cycle. The process of developing and establishing CPLP measurements requires careful consideration and should be driven by the program goals and objectives, as well as any regulatory requirements.

How the organization defines and collects its measurements will lead to the definition and analysis of its metrics. Metrics monitor the accomplishment of the program goals and objectives by quantifying the level of implementation, effectiveness, and efficiency of the program while identifying possible improvements. An effective set of metrics can help improve content quality and learner engagement, obtain support from the organization, increase funding, reveal impacts on the cybersecurity and privacy risk management program, and demonstrate returns on investment. Include results from both quantitative and qualitative measurement instruments.

CPLP managers should be aware of and prepared to participate in all CPLP-related compliance regulations for the organization, including reporting requirements. In some organizations, compliance reporting may be handled by an assigned single individual or group. For those organizations in which the duties are separated, it is critical to maintain collaborative communication to ensure that the program meets compliance.

A fully developed and integrated CPLP may become a useful tool for supporting enterprise risk management, although many are initially developed to address compliance requirements in laws, regulations, policies, or standards. Meeting these compliance obligations is often the primary focus of higher level leadership but should only be the starting point for a robust CPLP program. Examples of common quantifiable metrics to demonstrate CPLP compliance include training a certain percentage of the workforce and the results of practical exercises. Organizations should determine which compliance measurements they must achieve and consider those inputs when developing the CPLP.

CPLP managers should work with policy owners to determine the methods that the CPLP will use to meet reporting needs and ensure that the results of the learning efforts satisfy compliance requirements. Examples of questions to ask to help identify those needs include:

CPLP managers should periodically evaluate the overall effectiveness of the CPLP and report the results to leadership to encourage continued support for the program. At this step of the program life cycle, the application of metrics and measurements (see Sec. 2.4) should be documented.

In addition to ensuring compliance with regulatory requirements, the program itself needs to be able to meet its own goals. Two major components of the program to measure and assess are the instructors and the content. CPLP managers should review the performance of instructors to ensure that the material is delivered effectively. The success of the content can then be determined by incorporating feedback from instructors, students, and leadership. The following subsections describe measurements and metrics needs for assessment and evaluation.

The instructor is responsible for creating an environment in which learning can take place by facilitating discussions, providing knowledge and expertise, and motivating and inspiring the participants to learn new skills. An instructor evaluation includes both an assessment of the instructor and feedback from what the instructor observed or experienced.

CPLP managers and the Senior Leadership Committee work together to find the right instructors for their personnel and their CPLP’s learning objectives by determining whether organizational resources can support a dedicated in-house team of instructors or external resources are needed to implement instructor-led learning content. In some organizations, the cybersecurity learning program manager is also the privacy learning program manager and the lead instructor.

Since instructors are a key factor in creating a supportive and effective learning environment, it is important to monitor the performance of instructors via observation and other forms of feedback. Some effective instructor qualities include how well the instructor communicates the material and key concepts for the audience level, engages with learners in a positive and respectful manner, encourages active participation, uses interactive teaching methods, and responds to learner questions and concerns.

Instructor feedback can be obtained via the learner’s course feedback, as well as from direct annual observations of the instructor. Some tips for providing learner feedback are:

Some tips for instructor observations are:

It is important to create a fair and supportive instructor evaluation process to ensure that learning goals are achieved and there is continuous improvement. Combining multiple evaluation methods and a transparent process provides a solid foundation for constructive evaluation. Most importantly, instructors will know that they are appreciated for their important role in the learning program.

Instructors also provide important feedback on the learning material and how it was received by the learners. CPLP managers should work with instructors to review their comments and observations and, when necessary, support the adjustment of material for greater effectiveness. Instructors can provide feedback on:

Considering multiple types of feedback can help CPLP managers develop a thorough understanding of what is working and where improvement is needed in a CPLP. Feedback mechanisms can be used to evaluate learners’ reactions, learnings, behaviors, and results — the four levels of learner impact in the Kirkpatrick Model [15], as shown in Table 3.

Table 3. Four levels of the Kirkpatrick Model for training evaluation

Level 1 is sometimes referred to as the “smile sheet” because a simple questionnaire with visual prompts for ratings can be used to ask the learner for their immediate thoughts on the learning program. The results from Level 1 reactions can be used to assess the immediate impact of the learning program element.

Level 1 reaction measurements are influenced by:

Level 2 evaluates how well the participant absorbed the learning content and acquired new knowledge based on the learner outcomes, which are written in a format that is measurable, achievable, and outcome-based. One of the most common techniques for measuring learner performance is an end-of-course assessment, though there are other innovative and engaging techniques that should be considered and utilized.

Level 2 learning best practices include:

Level 3 considers whether the learner was able to apply new knowledge or skills to their work responsibilities and duties. Obtaining feedback depends largely on asking questions of learners, their peers, or their supervisor.

Level 3 behavior best practices include:

How have you utilized the knowledge gained from the learning program in the o workplace? How has your approach to [job activity] changed after attending the learning program? Describe a situation at work in which you were able to apply your new knowledge or skills.

What additional learning content support or reinforcement or job aid would help o you utilize new knowledge and skills in the workplace?

Was the cybersecurity lab activity on troubleshooting rogue IP addresses o adequate to help you do this activity on the job? In what ways, if any, was the small group activity on identifying roles and responsibilities helpful for your workplace?

In what ways, if any, have you observed or noticed changes in the learner’s [job o activity] since they attended the learning program. Have you observed the learner sharing their new knowledge or skills with others?

What new processes or procedures need to be included in the learning o materials, if any? What new focus areas or changes in the workplace, if any, need to be included in the learning materials?

How could job aids, awareness materials, or learning support help reinforce the o new knowledge or skills?

Level 4 feedback determines the impact that the learning program had on the organization, sometimes referred to as a return on investment (ROI). CPLP managers analyze the results of the activity from an enterprise-wide perspective. For example, the organization may decide to implement a new technology, such as artificial intelligence (AI), in its business processes. Training provides the policies and procedures for how the learner can use AI technologies in their work role. The program requires learners to attend the training but also requires supervisor approval and justification for its use. The organization must then determine whether the training on policies and procedures or supervisory control produced the results. To utilize this assessment method correctly, the requirements, objectives, overall outcomes, and metrics must be clearly defined.

Learning objectives and learner outcomes should be aligned with the organization’s strategic and business objectives and goals during the design process. An organizational goal aligned to learning objectives should be quantifiable and detailed (e.g., “SMART” goals). An example of aligning an organizational goal, learning goals, learner outcomes, and organizational results is:

A metric to determine whether the organizational goal is achieved is a trend o analysis of time needed by staff to mitigate known exploited vulnerabilities. Establish a baseline before the learning program, and track changes following the learning program.