NIST SP 800-55 Vol. 1 日本語訳

Volume 1 — Identifying and Selecting Measures

Katherine Schroeder Hung Trinh Victoria Yan Pillitteri

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-55v1

Volume 1 — Identifying and Selecting Measures

Katherine Schroeder

Hung Trinh

Victoria Yan Pillitteri

Computer Security Division Information Technology Laboratory

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-55v1

December 2024

U.S. Department of Commerce Gina M. Raimondo, Secretary

National Institute of Standards and Technology

Laurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology

Certain equipment, instruments, software, or materials, commercial or non-commercial, are identified in this paper in order to specify the experimental procedure adequately. Such identification does not imply recommendation or endorsement of any product or service by NIST, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose.

There may be references in this publication to other publications currently under development by NIST in accordance with its assigned statutory responsibilities. The information in this publication, including concepts and methodologies, may be used by federal agencies even before the completion of such companion publications. Thus, until each publication is completed, current requirements, guidelines, and procedures, where they exist, remain operative. For planning and transition purposes, federal agencies may wish to closely follow the development of these new publications by NIST.

Organizations are encouraged to review all draft publications during public comment periods and provide feedback to NIST. Many NIST cybersecurity publications, other than the ones noted above, are available at https://csrc.nist.gov/publications.

Authority This publication has been developed by NIST in accordance with its statutory responsibilities under the Federal Information Security Modernization Act (FISMA) of 2014, 44 U.S.C. § 3551 et seq., Public Law (P.L.) 113-283. NIST is responsible for developing information security standards and guidelines, including minimum requirements for federal information systems, but such standards and guidelines shall not apply to national security systems without the express approval of appropriate federal officials exercising policy authority over such systems. This guideline is consistent with the requirements of the Office of Management and Budget (OMB) Circular A-130.

Nothing in this publication should be taken to contradict the standards and guidelines made mandatory and binding on federal agencies by the Secretary of Commerce under statutory authority. Nor should these guidelines be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, Director of the OMB, or any other federal official. This publication may be used by nongovernmental organizations on a voluntary basis and is not subject to copyright in the United States. Attribution would, however, be appreciated by NIST.

NIST Technical Series Policies Copyright, Use, and Licensing Statements NIST Technical Series Publication Identifier Syntax

Publication History Approved by the NIST Editorial Review Board on 2024-11-04 Supersedes NIST SP 800-55 Rev. 1 (July 2008) https://doi.org/10.6028/NIST.SP.800-55r1

How to Cite this NIST Technical Series Publication: Schroeder K, Trinh H, Pillitteri VY (2024) Measurement Guide for Information Security: Volume 1 — Identifying and Selecting Measures. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-55v1. https://doi.org/10.6028/NIST.SP.800-55v1

Katherine Schroeder: 0000-0002-4129-9243

Hung Trinh: 0000-0002-3323-0836

Victoria Yan Pillitteri: 0000-0002-7446-7506

cyber-measures@list.nist.gov

National Institute of Standards and Technology Attn: Computer Security Division, Information Technology Laboratory 100 Bureau Drive (Mail Stop 8930) Gaithersburg, MD 20899-8930

Additional Information Additional information about this publication is available at https://csrc.nist.gov/pubs/sp/800/55/v1/final, including related content, potential updates, and document history.

All comments are subject to release under the Freedom of Information Act (FOIA).

This document provides guidance on how an organization can develop information security measures to identify the adequacy of in-place security policies, procedures, and controls. It explains the measures prioritization process and how to evaluate measures.

assessment; information security; measurement; measures; metrics; performance; qualitative; quantitative; reports; security controls.

The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective security and privacy of other than national security-related information in federal information systems. The Special Publication 800-series reports on ITL’s research, guidelines, and outreach efforts in information system security, and its collaborative activities with industry, government, and academic organizations.

Audience

This guide is written primarily for users with responsibilities or interest in information security measurement and assessment. Government and industry can use the concepts, processes, and candidate measures presented in this guide.

Patent Disclosure Notice

NOTICE: ITL has requested that holders of patent claims whose use may be required for compliance with the guidance or requirements of this publication disclose such patent claims to ITL. However, holders of patents are not obligated to respond to ITL calls for patents and ITL has not undertaken a patent search in order to identify which, if any, patents may apply to this publication.

As of the date of publication and following call(s) for the identification of patent claims whose use may be required for compliance with the guidance or requirements of this publication, no such patent claims have been identified to ITL.

No representation is made or implied by ITL that licenses are not required to avoid patent infringement in the use of this publication.

Information security measurement enables organizations to describe and quantify information security, allocate finite resources, and make informed and data-driven decisions for improved outcomes. However, organizations first need to know what policies, procedures, and controls they have in place at any given time; whether those policies and procedures are having the desired results; and how the organization and its risks are impacted. By developing and monitoring measurements that evaluate what an organization has in place for information security risk management and how well those efforts are working, an organization can better address their goals and direct resources.

NIST Special Publication (SP) 800-55v1 (Volume 1) is a flexible guide for developing and selecting information security measures at the organization, mission/business, and system levels to identify the success of in-place policies, procedures, and controls.¹ This document expands on previous NIST work on information security measures and measurements by focusing on quantitative assessments² and addressing organizational and program maturity.

The SP 800-55v2 [23] provides a methodology for implementing an information security measurement program. Additionally, while many of the principles of information security measurement may apply to privacy, privacy is out of scope for this document.

This document is intended to provide considerations for measuring the information security program activities described in other NIST publications, including:

The remaining sections of this document discuss the following:

In the context of this document, the following terms are defined as follows:

This document discusses concepts that are similar to the Stevens Scale of Measurement, as shown in Table 1, but takes a different view on what is and is not a measurement. For the purposes of this document, a nominal scale is considered a form of data gathering, and an ordinal scale is considered a ranking system. Both interval and ratio scales use variables that represent true numbers and can be used in a quantitative assessment, so they are considered measurement [19].

Table 1. Stevens Scale of Measurement

The terms assessment and measurement are often used interchangeably in the information security field, as both aid in risk management and security posture analysis. This document provides a lexicon for key terminology and an overview of foundational concepts for measuring and assessing information security risk and clarifies the distinction between assessment and measurement. As described in Sec. 1.4, assessment refers to the process of evaluating, estimating, or judging against defined criteria, and measurement is the process of obtaining quantitative values. Hence, assessment is a broader concept that also includes measurement.

Organizations perform multiple kinds of assessment when evaluating information security risk, such as risk assessments, program assessments, and control assessments. Risk assessments are used to identify the risks that an organization faces and can support decision-making [9]. Program-level assessments are used for decision-making about the strategies, policies, procedures, and operations that determine the security posture of an information security program. In control assessments, organizations evaluate whether specific controls are performing the way they were intended and achieving the desired results. Both program assessments and control assessments are in and of themselves a form of risk assessment and provide a different lens for viewing information security risk. SP 800-55 is intentionally agnostic on specific risk assessment models. Many of these models may help identify areas of threat, likelihood, vulnerability, and impact that require further assessment.⁴

Measures are numerically expressed data that are gathered through the process of measurement.⁵ Measures can be derived from any operations or systems that can be measured with numbers. Quantitative assessments judge measures data against a set criteria or target and can be used to analyze information security risks using frequency, rates, financial impacts, and other numeric indicators.

Using quantitative assessments requires a knowledge of measurement techniques and data analysis processes. One challenge of measurement is using the right measures and quantity of measures to perform useful analysis. A single measure alone may not provide sufficient data to make risk-based decisions, but organizations may also have restraints on resources that prevent them from employing and analyzing every potential measure. An organization finds the number of measures and depth of analysis that work best for their needs.

The ability to measure information security risks relies on data availability. Methods for collecting information security data may include experimentation, observation, or sampling. The NIST Engineering Statistics Handbook [18] offers detailed information on choosing a sampling scheme, including the following methods:

Once the data from measurement is procured, the outputs of quantitative analysis can be used in a quantitative assessment to determine whether the organization is meeting its information security goals and support risk-based decision-making. Data analysis methods⁶ are largely based on the type of questions that the organization is asking about their information security risks, program, and controls. The NIST Engineering Statistics Handbook [18] identifies three popular approaches to data analysis:

Table 2 shows examples of quantitative analysis across risk assessment, program-level assessment, and control-level assessment.

Table 2. Data analysis examples

Organizations that are early in the process of assessing their information security risks, program, or systems may rely heavily on qualitative assessments that present nonnumerical information in place of measurement. These nonnumerical methods can help show context, examine labels, and look at behavior. A prominent example of qualitative assessment featured in many information security measurement programs is the risk matrix — a table that uses colored rating scales to show the impact and likelihood of various risks. As organizations gain the ability to record and track information security data, they can move toward the increased precision and reduced bias of quantitative assessments.

There are three types of assessment:

Quantitative assessments (i.e., measurements) can provide objective data that allows for tracking and shows changes. However, they can be difficult to produce in early stages of measurement since they require more data and resources than nonnumerical and categorified qualitative assessments. In contrast, simple nonnumerical and categorified qualitative assessments may be more commonly used and easier to conduct, but their results can also be subjective and require everyone to have an equal understanding of the scale used.

It is important for organizations to consider their motivations for measuring information security risks before determining whether a quantitative or qualitative assessment is appropriate. For example, an organization motivated primarily by compliance with an industry certification or international standard has different measurement needs than an organization motivated by cost reduction. An organization could have multiple, competing motivations that drive the identification and selection of measures.

Some organizational motivations may benefit from quantitative assessments, such as trying to determine whether the organization is patching known vulnerabilities in an acceptable amount of time. Knowing the mean time to remediate a vulnerability provides more precise insight into patching efficiency than simply knowing the number of vulnerabilities patched in a year. Because the question of mean time to remediate a vulnerability deals with attainable non-zero numbers, a measurement can be taken, and a mathematically derived answer can be given.

When real and attainable numbers based on gathered data can be found and analyzed, a quantitative assessment may be the appropriate action. If there are proposed questions that do not have measurable numbers attached to them but still need to be addressed, a qualitative assessment may be the best option.

Commonly used qualitative methods include color scales that represent risk levels or number scales that show rankings. For the purposes of this document, qualitative and semi-quantitative assessments are not considered measurements, and the values produced by these types of assessments are not considered measures. Most organizations will use a mixture of quantitative, semi-quantitative, and qualitative assessments. Ultimately, some or all of the assessment results will be used to determine success.

Developing and establishing measurements to capture and provide meaningful data at all levels of an organization requires careful consideration. Meaningful measures take organizational information security goals and objectives into account and are obtainable, repeatable, and feasible to measure. Information security measurement enables organizations to quantify improvements or gaps in securing systems and demonstrate quantifiable progress in accomplishing strategic goals and objectives (i.e., security posture). Well-designed measurements can provide information on the implementation, effectiveness, efficiency, and business impacts of controls, such as the results of information security activities, events (e.g., incident data, revenue lost to cyber attacks), and information security investments.

Measurement provides data that can enable an organization to examine the impacts of implementing information security programs, specific controls, and associated policies and procedures. Such data is integral when making risk-based decisions, weighing performance against designated metrics, and demonstrating compliance. Measurement can also increase accountability and strengthen governance by providing data that can facilitating the identification of the personnel responsible for controls implemented within specific organizational components or systems and support an environment that allows for continuous analysis and improvement.

In addition to measurements, organizations also utilize metrics to track progress, facilitate decision-making, and improve performance. Information gained from measurements may be used to identify and define new metrics. Metrics can be applied at the system level, program level,⁷ and organization level. System-level metrics, such as the frequency of third-party access to a system or the number of communication ports open, can facilitate tactical decision-making and support program-level metrics. Program-level metrics, such as the number of security incidents in a year or the cost per incident, may be helpful when making organizational strategic decisions. Both system- and program-level metrics can also support risk management-informed decision-making.

Metrics are designed to track progress, facilitate decision-making, and improve performance with respect to a set target. Metrics leverage measures to provide insight into how well an organization is performing at the program or system level and whether the organization is reducing their information security risk. As with measures, the characteristics of meaningful metrics include the value being objective, accurate, precise, tied to a fixed reference or point in time, replicable, and comparable to previous measurements. Metrics are set with organizational goals in mind and drive subsequent assessments whose results then inform the metrics going forward. Figure 1 shows a notional process for the definition, collection, and analysis of metrics.

Fig. 1. Notional process for the definition, collection, and analysis of metrics

Knowing why measurements are being taken can help when selecting which potential measurements or metrics to focus on. The purpose and output of measurements must be unambiguous and easily understood, and the chosen metrics tell a meaningful story about organization-, program-, or system-level information security. For example, when evaluating cybersecurity awareness training, consider completion rates and the results of review quizzes instead of marking participation as “low, medium, or high.”

When initially selecting quantitative metrics, it is natural to want to adopt any available data points, perhaps with the intent to use them to narrow down focus on critical elements. Poorly selected quantitative metrics can undermine the overall quality of reporting and erode confidence in the work product. When considering a quantitative metric, consider whether the data point directly illustrates the performance of a valid control or tracks the presence of a material risk to the organization’s objective. If not, it may be best to exclude the data item in question. With quantitative metrics - initially at least - less is often more.

By keeping metrics consistent over time, organizations can evaluate long-term trends and expected ranges. A new metric may provide important insights, but tracking the measurements related to metrics over a continuous period (e.g., quarter to quarter, year to year) will give more information about the success of organization-, program-, and system-level information security plans, policies, procedures, and goals. Some metrics may be gathered because of external guidance or regulations.

Key risk indicators (KRIs) and key performance indicators (KPIs) are examples of metrics, though not all metrics fall into these categories. Organizations may find that a wide variety of metrics fit their needs. For example, appropriate measures at the organization level may include the cost per security incident as part of the budget allocation process, whereas measurements at the system level may include the frequency of virus scans across individual systems.

Because measurement can involve large amounts of data, having a plan for data handling is critical to ensuring that factors such as documentation, data management, data quality, and uncertainty are all considered. An organized and repeatable process that allows for the consistent assessment of collected data provides much-needed context for measurements.

Information security measurements can be scoped to a variety of environments and needs. Assets, controls, vulnerabilities, and security incidents can all be measured. Measures can be applied to organizational units, sites, processes, and other constructs. Since the high volume of measurements may be too numerous to track, organizations may want to aggregate data to help track security trends across an organization. Organizations will carefully define the scope of measures based on specific stakeholder needs, strategic goals and objectives, operating environments, risk priorities, and resources. As an organization develops measurements, they may want to consider options to make the data collection process easier, such as automation.

Information security measures can be applied at the system level to provide quantifiable data regarding the implementation, effectiveness, efficiency, and impact of required or desired security controls. System-level measures can be used to determine the system’s security posture, demonstrate compliance with organizational requirements, and identify areas of improvement.

Measurements can be used to monitor organizational information security activities at the program and organization levels. These measurements may be derived by aggregating multiple system-level measures or developed by using the entire enterprise as the scope. Organization-level measurements require that the processes on which the measures depend are consistent and repeatable and ensure the availability of data across the organization.

Perfectly measuring information security is challenging due to the gap between mathematical models and practical implementations [21]. Instead, experimenting as possible with relative metrics, models, and approaches over time is the best way to identify the most effective performance indicators.

Organizations document their measures in a standard format to ensure the repeatability of measures development, collection, and reporting activities. By keeping a consistent record of what is being measured, where the data comes from, what formulas and calculations are being used, and who interacts with the data, it becomes easier to trace data and ensure continuity of the process.

Organizations can tailor their standard format to their unique environments and requirements based on internal practices and procedures. However, the following fields offer a common starting point:

Information owner — Identify the organizational component and the individual who owns the required information.

Information collector — Identify the organizational component and the individual responsible for collecting the data.⁸

Information customer — Identify the organizational component and the individual who will review the data.

If a measure is reported to stakeholders without proper context, there can be unintended consequences or indecision. Therefore, a report could include elements such as:

These examples are not meant to provide a complete list of elements to be included in measurement reporting but highlight how measurement reporting can support organizational decision-making by providing context around measurements and findings. More information about measurement reporting and communication about measurements can be found in SP 800-55v2, Measurement Guide for Information Security: Volume 2 — Program.

Although substantial amounts of information security data may be collected, not all data will be useful for the information security measurement program. Any data collected specifically for information security measures are as non-intrusive as possible and of maximum usefulness to ensure that available resources are primarily used to correct problems rather than collect data.

Information security data repositories represent a significant collection of operational and vulnerability data. Due to the sensitivity of this data, they are protected in accordance with applicable laws, regulations, policies, and procedures.

Data collection methods and the data repositories used for measures data collection and reporting (either directly or as data sources) are clearly defined to ascertain the quality and validity of the data. This also helps ensure that testing is repeatable and can show changes over time.

Data validity is suspect if the primary data source is an incident-reporting database that only stores information reported by a few organizational elements or if reporting processes between organizations are inconsistent. The importance of standardizing reporting processes cannot be overemphasized. When organizations are developing and implementing processes that may serve as inputs into an information security measurement program, ensuring that data gathering and reporting are clearly defined helps facilitate valid data collection. Having a validation process in place to check the integrity, accuracy, and structure of the data provides a way to address potential errors before any analysis is done. By setting a standard process to validate data, an organization can have a repeatable way to look at the data and ensure its quality.

Even when measurements are intended to be precise and accurate, random and systemic errors can still occur. While there is no guaranteed way to measure uncertainty in all measurements, statistical information calculated from the data (e.g., standard deviation, standard error of mean, and confidence intervals) can provide more insight.

Uncertainty can be reduced by using data cleaning methods, such as validation, normalization, transformation, and imputation, as shown in Table 3.

Table 3. Data cleaning methods for reducing uncertainty

In addition to making the data itself more useable, data analysis methods can address uncertainty within the data. Organizations often make quantitative projections using statistical methods, such as regression, time series analysis, and machine learning methods. When looking at projections, it is helpful to consider that future events and other unknown factors can cause unforeseen changes.

The reliability and effectiveness of security measures require their adherence to a set of defined characteristics. These characteristics act as a foundation for the development, implementation, and assessment of information security measures:

These characteristics collectively define the foundation of robust security measurement frameworks. Adherence to these principles ensures the reliability, consistency, and replicability of security measure developed to safeguard assets and information.

This document separates measures/assessment results into four types:

Implementation measures demonstrate the progress of specific controls. Monitoring implementation may include assessment results, such as a tally of known systems or a binary “yes/no” about which systems have up-to-date patches.⁹ Implementation measures look at quantitative outputs and are usually demonstrated in percentages. Examples of implementation measures related to information security programs include the percentage of systems with approved system security plans and the percentage of systems with password policies that are configured as required. Implementation measures can also examine system-level areas, such as the percentage of servers in a system with a standard configuration.

By gathering this data, an organization can understand how its goals are being implemented and what tasks still need to be accomplished. Organizations never fully retire implementation measures because they are a record of what exists and what needs improvement. However, once implementation measures are completed, the emphasis and resources of the measurement program broaden to include effectiveness, efficiency, and impact measures.

Effectiveness measures evaluate how well implementation processes and controls are working and whether they are meeting desired outcomes. An effectiveness assessment can either concentrate on the evidence and results of a quantitative analysis of measures or be applied in a qualitative “yes/no” paradigm. Effectiveness measures may require multiple data points that quantify the degree to which information controls are implemented and their effects on the organization’s information security posture.

Efficiency measures examine the timeliness of controls by determining the speed at which they give useful feedback and how quickly those issues are addressed. An efficiency assessment concentrates on the evidence and results of quantitative measures analysis.

Effectiveness and efficiency together are often referred to as Program Results.

Impact measures articulate the impact of information security on an organization’s unique mission, goals, and objectives by quantifying the following:

These measures combine the results of control implementation with a variety of information about resources. They can provide the most direct insight into the value of information security to the organization and are sought by executives. While impact measures are largely organization-specific, a general example would be the percentage of the organization’s budget devoted to information security controls. After implementation measures are gathered, further analysis — not dissimilar to normalization — is required before the results can be expressed to a non-technical audience.

Developing and selecting information security measures consists of four major activities:

This document focuses on the development of measures related to information security risk management, which is part of a larger implementation process of information security measurement.¹⁰ The identification and definition of the existing information security program are important to the development of measures.

Identification and definition include:

Knowing what controls are implemented in an organization is foundational to quantitative assessment. The system- and program-level controls that need to be tracked must be understood before an organization can evaluate what kinds of measurements to take or the process of prioritizing potential measures. This creates a structure for determining what measurements need to be taken and what metrics are used for evaluation.

When developing measures, an organization needs to know the attributes of the asset it is trying to protect. For example, there is no organizational gain in trying to measure the confidentiality of a Domain Name System (DNS) registry when the critical characteristics are its integrity and availability. As an organization cannot take and evaluate infinite measures, the most successful measures consider stated priorities that inform the organization’s information security strategy. For example, if the organization considers the resource availability of a public website to be of critical importance, then selecting “the average number of port scans per day” as a measure might yield an accurate and informative result but not one that is relevant to the stated goals of the organization.

Qualitative and semi-quantitative assessments may also be useful or even necessary to assess implementation, effectiveness, efficiency, and impact, as shown in Table 4.

Table 4. Examples of measures and types of qualitative and semi-quantitative assessment results

After implementation measures are gathered, further analysis, not dissimilar to data normalization, is required before the results can be expressed to a non-technical audience. Successful effectiveness and efficiency assessments of individual phenomenon may require additional analysis using multiple measures. For example, when looking for “delayed vulnerability remediations,” simply checking the mean time to incident response is unlikely to provide all of the information necessary to determine the causes of an issue.

After implementation measures are in place, organizations prioritize which efficiency, effectiveness, and impact measures to implement. Prioritization can be driven by a variety of factors, including an organization’s risk management strategy, mission and business objectives, the availability of data collection for target assets, the value of the asset, the cost of the controls required to prevent adverse outcomes, the potential availability of imperfect or partial controls, information from risk assessments, policies, and legal, regulatory, or other requirements.

Likelihood and impact modeling are meant to work in tandem as part of a larger risk assessment process.¹¹ Simply knowing either the likelihood or the potential impact of an event is not enough information to determine the importance of a potential measure to an organization.

Identifying existing data for use in likelihood and impact modeling typically involves working with stakeholders from across the organizational structure. When possible, data from existing risk assessments can be utilized to reduce redundancy and enable decision-making (e.g., using existing modeled data to help decide what measurements to prioritize). Organizations may also have useful data from audits, interviews, surveys, or studies and external data on likelihood and impact. Annual reports can provide information on threat landscapes and the financial impacts of information security incidents that can be used to create models. A wide range of event likelihood models can be used to assess the likelihood of adverse events when determining which systems and controls to measure.

Organizations can also compare impact models with event probability models (e.g., expected loss and statistical analysis of historical market trends) to determine their measurement priorities. Controls or systems with higher likelihoods of incident or higher potential impacts if affected could then be prioritized when organizations decide how to allocate measurement resources. Where possible, leverage existing event likelihood and impact models (e.g., risk registers¹²) to avoid a duplication of efforts. More information on quantitative likelihood and impact models can be found in Appendix C.

In addition to using historical information for likelihood and impact modeling, current trends may provide useful datapoints when prioritizing and selecting measures. Staying updated on current threats allows for more effective continuous measurement and assessment. At the same time, recency bias¹³ about current events often influences choices when determining courses of action and resource allocation. Outliers and unexampled events may occur over time. An organization can prepare for these issues using horizon scanning, stress tests, and system resilience.¹⁴

Information gained from modeling likelihood and impact can be combined with knowledge about organizational goals and existing controls to create a customized weighing scale to prioritize potential measures. Using a weighing scale with set parameters ensures consistency when prioritizing and selecting measures, even those that are unrelated to information security.

Measures that are ultimately selected are useful for:

After an organization selects its measures, the collected data is evaluated. Evaluation may look different depending on the types of measures being analyzed. Quantitative data analysis methods, like those in Sec. 2.3, can be used to evaluate measures.

For implementation measures, evaluation may be as simple as comparing the percentage of controls implemented with the goal percentage of implementation. Effectiveness, efficiency, and impact measures will likely be more complicated to evaluate. Both effectiveness and efficiency measures often begin by establishing average data output and evaluating acceptable ranges against output going forward. For example, an organization may want to know if the volume of data being transferred on the network has an anomaly. To monitor for changes, the average volume of data transferred is established. An organization may also set an acceptable range based on a standard deviation from this average. This may mean looking for outliers in the data or monitoring for changes over time. Evaluating impact measures will likely include outcomes outside of information security, such as financial outcomes or even public perception.

Various indicators and inputs can be useful to track the effectiveness and efficiency of an information security program by monitoring performance and security over time, such as:

Access to average outputs, acceptable ranges, and long-term data makes effectiveness and efficiency measures more accurate and beneficial by enabling organizations to track changes over time. Even if processes are not yet consistent, average outputs and acceptable ranges help organizations set metrics. Some metrics are directly related to established averages, while others are set by other sources, and established ranges may not have any effect on organizational goals. While inconsistent processes will not provide meaningful data, measurements may still be used to establish average outputs and acceptable ranges for future analysis. Data analysis for finding average outputs and acceptable ranges will typically include historical data and a forecast of what that trend may continue to look like in the future if all variables stay the same.

Some measures also have the potential to give misleading information. Inputs such as phishing test success rates or the number of known vulnerabilities depend heavily on the quality of work behind them. A poorly designed phishing test might show a better success rate while giving less information about the preparedness of the workforce to recognize a well-designed phishing email. This does not mean that organizations need to avoid these measures altogether, but numbers alone may not always show the whole story.

References

[1] National Institute of Standards and Technology (2024) The NIST Cybersecurity Framework (CSF) 2.0. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Cybersecurity White Paper (CSWP) NIST CSWP 29. https://doi.org/10.6028/NIST.CSWP.29
[2] National Institute of Standards and Technology (2006) Minimum Security Requirements for Federal Information and Information Systems. (U.S. Department of Commerce, Washington, DC), Federal Information Processing Standards Publication (FIPS) 200. https://doi.org/10.6028/NIST.FIPS.200

[3] Bowen P, Kissel RL (2007) Program Review for Information Security Management Assistance (PRISMA). (National Institute of Standards and Technology, Gaithersburg, MD), NIST Interagency or Internal Report (IR) NIST IR 7358. https://doi.org/10.6028/NIST.IR.7358

[4] Stine KM, Quinn SD, Witte GA, Gardner RK (2020) Integrating Cybersecurity and Enterprise Risk Management (ERM). (National Institute of Standards and Technology, Gaithersburg, MD), NIST Interagency or Internal Report (IR) NIST IR 8286. https://doi.org/10.6028/NIST.IR.8286

[5] Taylor BN (2011) The current SI seen from the perspective of the proposed new SI. Journal of Research of the National Institute of Standards and Technology 116(6):797. https://doi.org/10.6028/jres.116.022
[6] Software Quality Group (2021) Metrics and Measures. (National Institute of Standards and Technology, Gaithersburg, MD). Available at https://www.nist.gov/itl/ssd/software-quality-group/metrics-and-measures
[7] Thomas D (2019). Monte Carlo Tool. (National Institute of Standards and Technology, Gaithersburg, MD). Available at https://www.nist.gov/services-resources/software/monte-carlo-tool

[8] ASTM International (2018) ASTM C1012/C1012M-18a – Standard Test Method for Length Change of Hydraulic-Cement Mortars Exposed to a Sulfate Solution (ASTM International, West Conshohocken, PA). https://doi.org/10.1520/C1012_C1012M-18A
[9] Joint Task Force Transformation Initiative (2012) Guide for Conducting Risk Assessments. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-30, Rev. 1. https://doi.org/10.6028/NIST.SP.800-30r1
[10] Joint Task Force (2018) Risk Management Framework for Information Systems and Organizations: A System Life Cycle Approach for Security and Privacy. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-37, Rev. 2. https://doi.org/10.6028/NIST.SP.800-37r2

[11] Chew E, Swanson MA, Stine KM, Bartol N, Brown A, Robinson W (2008) Performance Measurement Guide for Information Security. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-55, Rev. 1. https://doi.org/10.6028/NIST.SP.800-55r1

[12] Grance T, Nolan T, Burke K, Dudley R, White G, Good T (2006) Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-84. https://doi.org/10.6028/NIST.SP.800-84
[13] Kent K, Souppaya M (2006) Guide to Security Log Management. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-92. https://doi.org/10.6028/NIST.SP.800-92
[14] Dempsey KL, Pillitteri VY, Baer C, Niemeyer R, Rudman R, Urban S (2020) Assessing Information Security Continuous Monitoring (ISCM) Programs: Developing an ISCM Program Assessment. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-137A. https://doi.org/10.6028/NIST.SP.800-137A
[15] Barker E, Smid M, Branstad D (2015) A Profile for U.S. Federal Cryptographic Key Management Systems. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-152. https://doi.org/10.6028/NIST.SP.800-152
[16] Ross R, Winstead M, McEvilley, M (2022) Engineering Trustworthy Secure Systems. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-160v1r1. https://doi.org/10.6028/NIST.SP.800-160v1r1
[17] Boyens JM, Smith AM, Bartol N, Winkler K, Holbrook A, Fallon M (2022) Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-161r1-upd1, Includes updates as of November 1, 2024. https://doi.org/10.6028/NIST.SP.800-161r1-upd1

[18] Heckert N, Filliben J, Croarkin C, Hembree B, Guthrie W, Tobias P, Prinz J (2012) NIST/SEMATECH e-Handbook of Statistical Methods. (National Institute of Standards and Technology, Gaithersburg, MD). https://doi.org/10.18434/M32189

[19] Stevens SS (1946) On the Scales of Measurement. Science 103(2684):677-680. http://www.jstor.org/stable/1671815

[20] Turan MS, Barker E, Kelsey J, McKay KA, Baish ML, Boyle M (2018) Recommendation for the Entropy Sources Used for Random Bit Generation. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-90B. https://doi.org/10.6028/NIST.SP.800-90B

[21] Stolfo S, Bellovin S, Evans D (2011) Measuring Security. IEEE Security & Privacy 9(3):60-65. https://doi.org/10.1109/MSP.2011.56

[22] Ross R, Pillitteri V, Graubart R, Bodeau D, McQuaid R (2021) Developing Cyber-Resilient Systems: A Systems Security Engineering Approach. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-160v2r1. https://doi.org/10.6028/NIST.SP.800-160v2r1

[23] Schroeder K, Trinh H, Pillitteri VY (2024) Measurement Guide for Information Security: Volume 2 — Developing an Information Security Measurement Program. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-55v2. https://doi.org/10.6028/NIST.SP.800-55v2

Appendix A. Glossary

assessment The action of evaluating, estimating, or judging against defined criteria. Different types of assessment (i.e., qualitative, quantitative, and semi-quantitative) are used to assess risk. Some types of assessment yield results.

assessment results The output or outcome of an assessment.

Bayesian methodology Statistical approach to data analysis based on Bayes’ theorem where uncertainty is quantified by combining existing information with new information to create forecast models. [18, adapted]

classical data analysis A data analysis technique where data collection is followed by the imposition of a model, and the analysis, estimation, and testing that follow focus on the parameters of that model. [18, adapted]

data validation The process of determining that data or a process for collecting data is acceptable according to a predefined set of tests and the results of those tests. [15]

experimentation A systematic approach to the process of testing new ideas, methods, or activities that applies principles and techniques at the data collection stage to ensure the generation of valid, defensible, and supportable conclusions.

exploratory data analysis A data analysis technique where data collection is immediately followed by analysis with the goal of inferring what model would be appropriate. [18, adapted]

false positive An erroneous acceptance of the hypothesis that a statistically significant event has been observed. [20]

imputation The replacement of unknown, unmeasured, or missing data with a particular value. The simplest form of imputation is to replace all missing values with the average of that variable. [18, adapted]

information security The protection of information and systems from unauthorized access, use, disclosure, disruption, modification, or destruction to provide confidentiality, integrity, and availability. [2]

interval scale From the Stevens Scale of Measurement, a quantitative measurement scale using variables with equal values and no true zero, such as time and temperature. [19, adapted]

key performance indicator A metric of progress toward intended results.

key risk indicator A metric used to measure risk.

lagging indicator A metric that tracks the outcome of events or trends.

leading indicator A predictive metric that tracks events or behaviors that precede incidents. machine learning The development and use of computer systems that adapt and learn from data with the goal of improving accuracy.

mean The sum of the data points divided by the number of data points. Commonly referred to as the average. [18, adapted]

mean time to detect A metric that tracks the average amount of time that a problem exists before it is found.

mean time to recovery A metric that tracks the average amount of time that it takes to recover from a product or system failure.

measurement The process of obtaining quantitative values using quantitative methods.

measures Quantifiable and objective values that result from measurement.

median The value of the point that has half the data smaller than that point and half the data larger than that point. [18]

metrics Measures and assessment results designed to track progress, facilitate decision-making, and improve performance with respect to a set target.

mode The value of the random sample that occurs with the greatest frequency. This value is not necessarily unique. [18]

Monte Carlo analysis A probabilistic sensitivity analysis used to account for uncertainty. [7]

nominal scale From the Stevens Scale of Measurement, a scale that labels named variables into classifications. [19, adapted]

normalization The conversion of information into consistent representations and categorization. [4]

observational data Data captured through the observation of an activity or behavior without the direct involvement of the subject.

ordinal scale From the Stevens Scale of Measurement, a scale that orders and ranks data without establishing a degree of variation between ranks. [19, adapted]

outliers An observation that lies an abnormal distance from other values in a random sample from a population. [18]

qualitative assessment The use of a set of methods, principles, or rules for assessing risk based on nonnumerical categories or levels. [6]

quantitative assessment The use of a set of methods, principles, or rules for assessing risk based on numbers where the meanings and proportionality of values are maintained inside and outside of the context of the assessment. [6] random sampling A method of sampling where each sample has an equal chance of selection in hopes of gathering an unbiased representation. [18, adapted]

ratio scale From the Stevens Scale of Measurement, a quantitative measurement scale with a true zero using variables that can be compared to find differences or intervals. [19]

regression A statistical technique used to predict the value of a variable based on the relationship between explanatory variables.

sampling The process of taking samples of something for the purpose of analysis.

semi-quantitative assessment The use of a set of methods, principles, or rules for assessing risk based on bins, scales, or representative numbers whose values and meanings are not maintained in other contexts. [9]

stratified sampling The process of segmenting a population across levels of some factors to minimize variability within those segments. [18]

systematic stratified sampling A method of sampling where samples are taken at a regular interval. [18, adapted]

time series analysis The analysis of an ordered sequence of values of a variable at equally spaced time intervals. [18, adapted]

transformation The conversion of one state or format into another state or format.

Appendix B. Data Analysis Dictionary

The following information is found in the NIST Engineering Statistics Handbook [18].

B.1. Bayesian Methodology

Bayesian or predictive methodology consists of formally combining the prior distribution on the parameters and the collected data to jointly make inferences and/or test assumptions about the model of parameters.

B.2. Classical Data Analysis

Classical data analysis is when data collection is followed by a model, and the subsequent analysis, estimation, and testing focus on the parameters of that model. Classical data analysis includes deterministic and probabilistic models, such as regression and ANOVA. Some of the more common relevant classical quantitative models include:

Location

Scale (or variability or spread)

Skewness and Kurtosis

Randomness

Distributional Measures

Outliers

2-Level Factorial Designs

B.3. Exploratory Data Analysis

Exploratory data analysis emphasizes graphical techniques and inferring different analytic models in order to determine what model would be appropriate. Some common exploratory data analysis graphical techniques include:

Appendix C. Modeling Impact and Likelihood

This appendix is intended to provide a high-level overview of complex statistical concepts. The successful application of these concepts will require further training and understanding on the part of practitioners.

C.1. Bayesian Methodology

Bayesian or predictive methodology is applied when there is previous knowledge of the conditions associated with an event. It can provide conditional probability estimates quickly and without using significant resources. Because Bayesian methodology relies on prior information, it is important to note that the use of either inaccurate or a different selection of prior information may lead to results that do not provide significant insight.

C.2. Monte Carlo Methodology

The Monte Carlo method is a multiple probability simulation used to predict possible outcomes of an uncertain event. The Monte Carlo method uses randomly generated outcomes within a set range, and the frequencies of different outcomes generated form a normal distribution.

The Monte Carlo method allows for repeated modeling and can be performed using spreadsheet editors or programing languages for statistical computing. When using the Monte Carlo method, it is important to note that these simulations show an estimated probability and not an inevitable outcome.

C.3. Time Series Analysis

Time series analysis shows the level, trend, seasonality, or noise within a series of data points in a time series. Time series data is often found when monitoring a process over a period. Time series analysis considers the potential for an internal structure, such as trends or seasonal variations to data.

Time series regression models are primarily used for forecasting. Time series decomposition exhibits patterns within time series data and can be useful when setting the expected range or use of processes or systems.

C.4. Value at Risk

Value at risk (VaR) is a statistical analysis technique that builds a model that measures the risk of loss, primarily using a probability density function. The three key elements of building a VaR model are a fixed time period, a specific level of loss in value, and a confidence interval.

Calculating VaR can be helpful when making decisions about investments and resources. Like all predictive models, VaR relies heavily on the quality of inputs and cannot effectively estimate all scenarios.

Appendix D. Change Log

In December 2024, the following changes were made to this Special Publication: