Internet Engineering Task Force (IETF)                        S. Bradner
Request for Comments: 6815                            Harvard University
Updates: 2544                                                  K. Dubray
Category: Informational                                 Juniper Networks
ISSN: 2070-1721                                               J. McQuaid
                                                            Turnip Video
                                                               A. Morton
                                                               AT&T Labs
                                                           November 2012

Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful

RFC 2544の適用性ステートメント:有害と見なされる実稼働ネットワークでの使用



The Benchmarking Methodology Working Group (BMWG) has been developing key performance metrics and laboratory test methods since 1990, and continues this work at present. The methods described in RFC 2544 are intended to generate traffic that overloads network device resources in order to assess their capacity. Overload of shared resources would likely be harmful to user traffic performance on a production network, and there are further negative consequences identified with production application of the methods. This memo clarifies the scope of RFC 2544 and other IETF BMWG benchmarking work for isolated test environments only, and it encourages new standards activity for measurement methods applicable outside that scope.

Benchmarking Methodology Working Group(BMWG)は、1990年以来、主要なパフォーマンス測定基準と実験室試験方法を開発しており、現在もこの作業を続けています。 RFC 2544で説明されている方法は、ネットワークデバイスのリソースをオーバーロードしてその容量を評価するトラフィックを生成することを目的としています。共有リソースの過負荷は、本番ネットワークのユーザートラフィックパフォーマンスに悪影響を与える可能性があり、メソッドの本番アプリケーションに特定されたさらに悪い結果があります。このメモは、RFC 2544およびその他のIETF BMWGベンチマーク作業の範囲を分離されたテスト環境のみで明確にし、その範囲外で適用可能な測定方法の新しい標準化活動を奨励します。

Status of This Memo


This document is not an Internet Standards Track specification; it is published for informational purposes.

このドキュメントはInternet Standards Trackの仕様ではありません。情報提供を目的として公開されています。

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 5741.

このドキュメントは、IETF(Internet Engineering Task Force)の製品です。これは、IETFコミュニティのコンセンサスを表しています。公開レビューを受け、インターネットエンジニアリングステアリンググループ(IESG)による公開が承認されました。 IESGによって承認されたすべてのドキュメントが、あらゆるレベルのインターネット標準の候補になるわけではありません。 RFC 5741のセクション2をご覧ください。

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at


Copyright Notice


Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved.

Copyright(c)2012 IETF Trustおよびドキュメントの作成者として特定された人物。全著作権所有。

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents ( in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

この文書は、BCP 78およびこの文書の発行日に有効なIETF文書に関するIETFトラストの法的規定(の対象となります。これらのドキュメントは、このドキュメントに関するあなたの権利と制限を説明しているため、注意深く確認してください。このドキュメントから抽出されたコードコンポーネントには、Trust Legal Provisionsのセクション4.eに記載されているSimplified BSD Licenseのテキストが含まれている必要があり、Simplified BSD Licenseに記載されているように保証なしで提供されます。

Table of Contents


   1. Introduction ....................................................3
      1.1. Requirements Language ......................................4
   2. Scope and Goals .................................................4
   3. The Concept of an Isolated Test Environment .....................4
   4. Why the Methods of RFC 2544 Are Intended Only for ITE ...........4
      4.1. Experimental Control and Accuracy ..........................4
      4.2. Containing Damage ..........................................5
   5. Advisory on RFC 2544 Methods in Production Networks .............5
   6. Considering Performance Testing in Production Networks ..........6
   7. Security Considerations .........................................7
   8. Acknowledgements ................................................7
   9. References ......................................................8
      9.1. Normative References .......................................8
      9.2. Informative References .....................................8
   Appendix A. Example of RFC 2544 Method Failure in Production
               Network Measurement ....................................9
1. Introduction
1. はじめに

This memo clarifies the scope and use of IETF Benchmarking Methodology Working Group (BMWG) tests including [RFC2544], which discusses and defines several tests that may be used to characterize the performance of a network interconnecting device. All readers of this memo must read and fully understand [RFC2544].


Benchmarking methodologies (beginning with [RFC2544]) have always relied on test conditions that can only be produced and replicated reliably in the laboratory. These methodologies are not appropriate for inclusion in wider specifications such as:


1. Validation of telecommunication service configuration, such as the Committed Information Rate (CIR).

1. 認定情報レート(CIR)などの通信サービス構成の検証。

2. Validation of performance metrics in a telecommunication Service Level Agreement (SLA), such as frame loss and latency.

2. フレームの損失や遅延など、通信サービスレベル契約(SLA)のパフォーマンスメトリックの検証。

3. Telecommunication service activation testing, where traffic that shares network resources with the test might be adversely affected.

3. テストとネットワークリソースを共有するトラフィックが悪影響を受ける可能性があるテレコミュニケーションサービスアクティベーションテスト。

Above, we distinguish "telecommunication service" (where a network service provider contracts with a customer to transfer information between specified interfaces at different geographic locations) from the generic term "service". Below, we use the adjective "production" to refer to networks carrying live user traffic. [RFC2544] used the term "real-world" to refer to production networks and to differentiate them from test networks.

上記では、「テレコミュニケーションサービス」(ネットワークサービスプロバイダーが顧客と契約して、地理的に異なる場所にある指定されたインターフェイス間で情報を転送するサービス)を一般的な用語の「サービス」と区別しています。以下では、形容詞「本番」を使用して、ライブユーザートラフィックを伝送するネットワークを指します。 [RFC2544]は「実世界」という用語を使用して、実稼働ネットワークを指し、テストネットワークと区別しました。

Although [RFC2544] has been held up as the standard reference for the testing listed above, we believe that the actual methods used vary from [RFC2544] in significant ways. Since the only citation is to [RFC2544], the modifications are opaque to the standards community and to users in general.


Since applying the test traffic and methods described in [RFC2544] on a production network risks causing overload in shared resources, there is direct risk of harming user traffic if the methods are misused in this way. Therefore, the IETF BMWG developed this Applicability Statement for [RFC2544] to directly address the situation.

[RFC2544]で説明されているテストトラフィックとメソッドを運用ネットワークに適用すると、共有リソースで過負荷が発生するリスクがあるため、メソッドをこのように誤用すると、ユーザートラフィックに悪影響を与える直接的なリスクがあります。したがって、IETF BMWGは[RFC2544]の状況に直接対処するためにこの適用性声明を作成しました。

1.1. Requirements Language
1.1. 要件言語

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

このドキュメントのキーワード「MUST」、「MUST NOT」、「REQUIRED」、「SHALL」、「SHALL NOT」、「SHOULD」、「SHOULD NOT」、「RECOMMENDED」、「MAY」、および「OPTIONAL」は、 RFC 2119 [RFC2119]で説明されているように解釈されます。

2. Scope and Goals
2. 範囲と目標

This memo clarifies the scope of [RFC2544] with the goal of providing guidance to the industry on its applicability, which is limited to laboratory testing.


3. The Concept of an Isolated Test Environment
3. 分離されたテスト環境の概念

An Isolated Test Environment (ITE) used with the methods of [RFC2544] (as illustrated in Figures 1 through 3 of [RFC2544]) has the ability to:


o contain the test streams to paths within the desired setup

o 目的のセットアップ内のパスへのテストストリームを含める

o prevent non-test traffic from traversing the test setup

o 非テストトラフィックがテストセットアップを通過しないようにする

These features allow unfettered experimentation, while at the same time protecting lab equipment management/control LANs and other production networks from the unwanted effects of the test traffic.


4. Why the Methods of RFC 2544 Are Intended Only for ITE
4. RFC 2544のメソッドがITEのみを対象としている理由

The following sections discuss some of the reasons why [RFC2544] methods are applicable only for isolated laboratory use, and the consequences of applying these methods outside the lab environment.


4.1. Experimental Control and Accuracy
4.1. 実験制御と精度

All of the tests described in RFC 2544 require that the tester and device under test are the only devices on the networks that are transmitting data. The presence of other traffic (unwanted on the ITE network) would mean that the specified test conditions have not been achieved and flawed results are a likely consequence.

RFC 2544で説明されているすべてのテストでは、テスターとテスト対象のデバイスが、データを送信しているネットワーク上の唯一のデバイスである必要があります。他のトラフィック(ITEネットワークでは望ましくない)の存在は、指定されたテスト条件が達成されておらず、結果に欠陥がある可能性が高い結果であることを意味します。

If any other traffic appears and the amount varies over time, the repeatability of any test result will likely depend to some degree on the amount and variation of the other traffic.


The presence of other traffic makes accurate, repeatable, and consistent measurements of the performance of the device under test very unlikely, since the complete details of test conditions will not be reported.


For example, the RFC 2544 Throughput Test attempts to characterize a maximum reliable load; thus, there will be testing above the maximum that causes packet/frame loss. Any other sources of traffic on the network will cause packet loss to occur at a tester data rate lower than the rate that would be achieved without the extra traffic.

たとえば、RFC 2544スループットテストは、信頼できる最大の負荷を特徴付けることを試みます。したがって、パケット/フレームの損失を引き起こす最大値を超えるテストが行​​われます。ネットワーク上の他のトラフィックソースがあると、テスターのデータレートで、余分なトラフィックなしで達成されるレートよりも低いパケット損失が発生します。

4.2. Containing Damage
4.2. ダメージあり

[RFC2544] methods, specifically to determine Throughput as defined in [RFC1242] and other benchmarks, may overload the resources of the device under test, and they may cause failure modes in the device under test. Since failures can become the root cause of more widespread failure, it is clearly desirable to contain all test traffic within the ITE.


In addition, such testing can have a negative effect on any traffic that shares resources with the test stream(s) since, in most cases, the traffic load will be close to the capacity of the network links.


Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private IPv4 address range for testing:


"...The network addresses through have been assigned to the BMWG by the IANA for this purpose. This assignment was made to minimize the chance of conflict in case a testing device were to be accidentally connected to part of the Internet. The specific use of the addresses is detailed below."


In other words, devices operating on the Internet may be configured to discard any traffic they observe in this address range, as it is intended for laboratory ITE use only. Thus, if testers using the assigned testing address ranges are connected to the Internet and test packets are forwarded across the Internet, it is likely that the packets will be discarded and the test will not work.


We note that a range of IPv6 addresses has been assigned to BMWG for laboratory test purposes, in [RFC5180] (as amended by errata).


See the Security Considerations section below for further considerations on containing damage.


5. Advisory on RFC 2544 Methods in Production Networks
5. 本番ネットワークにおけるRFC 2544メソッドに関する勧告

The tests in [RFC2544] were designed to measure the performance of network devices, not of networks, and certainly not production networks carrying user traffic on shared resources. There will be undesirable consequences when applying these methods outside the isolated test environment.


One negative consequence stems from reliance on frame loss as an indicator of resource exhaustion in [RFC2544] methods. In practice, link-layer and physical-layer errors prevent production networks from operating loss-free. The [RFC2544] methods will not correctly assess Throughput when loss from uncontrolled sources is present. Frame loss occurring at the SLA levels of some networks could affect every iteration of Throughput testing (when each step includes sufficient packets to experience facility-related loss). Flawed results waste the time and resources of the testing service user and of the service provider when called to dispute the measurement. These are additional examples of harm that compliance with this advisory should help to avoid. See Appendix A for an example.

1つのマイナスの結果は、[RFC2544]メソッドにおけるリソース枯渇の指標としてのフレーム損失への依存から生じます。実際には、リンク層および物理層のエラーにより、実稼働ネットワークが無損失で動作することが妨げられます。 [RFC2544]メソッドは、制御されていないソースからの損失が存在する場合、スループットを正しく評価しません。一部のネットワークのSLAレベルで発生するフレーム損失は、スループットテストのすべての反復に影響を与える可能性があります(各ステップにファシリティ関連の損失を経験するのに十分なパケットが含まれている場合)。欠陥のある結果は、測定に異議を唱えるために呼び出されたときに、テストサービスユーザーとサービスプロバイダーの時間とリソースを浪費します。これらは、このアドバイザリへの準拠が回避に役立つはずの害の追加の例です。例については、付録Aを参照してください。

The methods described in [RFC2544] are intended to generate traffic that overloads network device resources in order to assess their capacity. Overload of shared resources would likely be harmful to user traffic performance on a production network. These tests MUST NOT be used on production networks and as discussed above. The tests will not produce a reliable or accurate benchmarking result on a production network.


[RFC2544] methods have never been validated on a network path, even when that path is not part of a production network and carrying no other traffic. It is unknown whether the tests can be used to measure valid and reliable performance of a multi-device, multi-network path. It is possible that some of the tests may prove valid in some path scenarios, but that work has not been done or has not been shared with the IETF community. Thus, such testing is contraindicated by the BMWG.


6. Considering Performance Testing in Production Networks
6. 本番ネットワークでのパフォーマンステストの検討

The IETF has addressed the problem of production network performance measurement by chartering a different working group: IP Performance Metrics (IPPM). This working group has developed a set of standard metrics to assess the quality, performance, and reliability of Internet packet transfer services. These metrics can be measured by network operators, end users, or independent testing groups. We note that some IPPM metrics differ from RFC 2544 metrics with similar names, and there is likely to be confusion if the details are ignored.

IETFは、別のワーキンググループであるIPパフォーマンスメトリック(IPPM)をチャーターすることにより、本番ネットワークのパフォーマンス測定の問題に対処しています。このワーキンググループは、インターネットパケット転送サービスの品質、パフォーマンス、および信頼性を評価するための一連の標準メトリックを開発しました。これらのメトリックは、ネットワークオペレーター、エンドユーザー、または独立したテストグループによって測定できます。一部のIPPMメトリックは、類似した名前のRFC 2544メトリックとは異なり、詳細を無視すると混乱が生じる可能性があることに注意してください。

IPPM has not yet standardized methods for raw capacity measurement of Internet paths. Such testing needs to adequately consider the strong possibility for degradation to any other traffic that may be present due to congestion. There are no specific methods proposed for activation of a packet transfer service in IPPM at this time. Thus, individuals who need to conduct capacity tests on production networks should actively participate in standards development to ensure their methods receive appropriate industry review and agreement, in the IETF or in alternate standards development organizations.


Other standards may help to fill gaps in telecommunication service testing. For example, the IETF has many standards intended to assist with network Operations, Administration, and Maintenance (OAM). ITU-T Study Group 12 has a Recommendation on service activation test methodology [Y.1564].

他の規格は、通信サービステストのギャップを埋めるのに役立ちます。たとえば、IETFには、ネットワークの運用、管理、および保守(OAM)を支援することを目的とした多くの標準があります。 ITU-Tスタディグループ12には、サービスアクティベーションテスト方法に関する推奨事項があります[Y.1564]。

The world will not spin off axis while waiting for appropriate and standardized methods to emerge from the consensus process.


7. Security Considerations
7. セキュリティに関する考慮事項

This Applicability Statement intends to help preserve the security of the Internet by clarifying that the scope of [RFC2544] and other BMWG memos are all limited to testing in a laboratory ITE, thus avoiding accidental Denial-of-Service attacks or congestion due to high traffic volume test streams.


All benchmarking activities are limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the other constraints [RFC2544].


The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network.


Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the device under test/ system under test (DUT/SUT).

さらに、ベンチマークは「ブラックボックス」ベースで実行され、被試験デバイス/被試験システム(DUT / SUT)の外部で観測可能な測定のみに依存します。

Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks.

特別な機能は、特にベンチマークの目的でDUT / SUTに存在すべきではありません。 DUT / SUTから生じるネットワークセキュリティへの影響は、ラボと実稼働ネットワークで同じである必要があります。

8. Acknowledgements
8. 謝辞

Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis Villamizar, David Newman, and Adrian Farrel for suggesting improvements to this memo.

このメモの改善を提案してくれたMatt Zekauskas、Bill Cerveny、Barry Constantine、Curtis Villamizar、David Newman、Adrian Farrelに感謝します。

Specifically, Al Morton would like to thank his coauthors, who constitute the complete set of Chairmen-Emeritus of the BMWG, for returning from other pursuits to develop this statement and see it through to approval. This has been a rare privilege; one that likely will not be matched in the IETF again:

具体的には、Al Mortonは、他の追求からこの声明を発展させ、承認に至るまで見てくれたことについて、BMWGの議長と名誉の完全なセットを構成する彼の共著者に感謝します。これはまれな特権でした。 IETFで再び一致しない可能性が高いもの:

Scott Bradner served as Chairman from 1990 to 1993 Jim McQuaid served as Chairman from 1993 to 1995 Kevin Dubray served as Chairman from 1995 to 2006


It's all about the band.


9. References
9. 参考文献
9.1. Normative References
9.1. 引用文献

[RFC1242] Bradner, S., "Benchmarking terminology for network interconnection devices", RFC 1242, July 1991.

[RFC1242] Bradner、S。、「ネットワーク相互接続デバイスのベンチマーク用語」、RFC 1242、1991年7月。

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2119] Bradner、S。、「要件レベルを示すためにRFCで使用するキーワード」、BCP 14、RFC 2119、1997年3月。

[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999.

[RFC2544] Bradner、S.およびJ. McQuaid、「ネットワーク相互接続デバイスのベンチマーク手法」、RFC 2544、1999年3月。

[RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. Dugatkin, "IPv6 Benchmarking Methodology for Network Interconnect Devices", RFC 5180, May 2008.

[RFC5180] Popoviciu、C.、Hamza、A.、Van de Velde、G。、およびD. Dugatkin、「IPv6 Benchmarking Methodology for Network Interconnect Devices」、RFC 5180、2008年5月。

9.2. Informative References
9.2. 参考引用

[Bryant] Bonica, R. and S. Bryant, "RFC2544 Testing in Production Network", Work in Progress, October 2012.

[ブライアント]ボニカ、R.、S。ブライアント、 "RFC2544 Testing in Production Network"、Work in Progress、2012年10月。

[Y.1564] ITU-T Recommendation Y.1564, "Ethernet Service Activation Test Methodology", March 2011.

[Y.1564] ITU-T勧告Y.1564、「Ethernet Service Activation Test Methodology」、2011年3月。

Appendix A. Example of RFC 2544 Method Failure in Production Network Measurement

付録A.実稼働ネットワーク測定におけるRFC 2544メソッドの失敗の例

This Appendix provides an example illustrating how [RFC2544] methods applied on production networks can easily produce a form of harm from flawed and misleading results.


The [RFC2544] Throughput benchmarking method usually includes the following steps:


a. Set the offered traffic level, less than max of the ingress link(s).

a. 入力リンクの最大値よりも小さい、提供されるトラフィックレベルを設定します。

b. Send the test traffic through the device under test (DUT) and count all frames successfully transferred.

b. テスト対象デバイス(DUT)を介してテストトラフィックを送信し、正常に転送されたすべてのフレームをカウントします。

c. If all frames are received, increment traffic level and repeat step b.

c. すべてのフレームを受信した場合は、トラフィックレベルを上げて、ステップbを繰り返します。

d. If one or more frames are lost, the level is in the DUT-overload region (step b may be repeated at a reduced traffic level to more exactly determine the maximum rate at which none of the frames are dropped by the DUT, defined as the Throughput [RFC1242]).

d. 1つ以上のフレームが失われた場合、レベルはDUT過負荷領域にあります(ステップbをトラフィックレベルを下げて繰り返すと、DUTによってフレームがドロップされない最大レートをより正確に決定できます。スループット[RFC1242])。

e. Report the Throughput values, the x-y of graph of frame size and Throughput, and other information in accordance with [RFC2544].

e. [RFC2544]に従って、スループット値、フレームサイズとスループットのグラフのx-y、およびその他の情報を報告します。

In this method, frame loss is the sole indicator of overload and therefore the determining factor in the measurement of Throughput using the [RFC2544] methodology (even though the results may not report frame loss per se).


Frame loss is subject to many factors in addition to operating above the Throughput traffic level. These factors include optical interference (which may be due to dirty interfaces, crossover from other signals, fiber bend and temperature, etc.) and electrical interference (caused by local sources of radio signals, electrical spikes, solar particles, etc.). In the laboratory environment many of these issues can be carefully controlled through cleaning and isolation. Since [RFC2544] methodologies are primarily intended to test devices and not paths, the total length of path, the number of interfaces, and compound risk of random frame loss can be kept to a minimum.

フレーム損失は、スループットトラフィックレベルを超える動作に加えて、多くの要因の影響を受けます。これらの要因には、光学干渉(他の信号からのクロスオーバー、ファイバーの曲がりや温度などが原因である可能性があります)と電気干渉(無線信号のローカルソース、電気スパイク、太陽粒子などが原因)が含まれます。実験室環境では、これらの問題の多くは、洗浄と分離によって注意深く制御できます。 [RFC2544]の方法論は主にパスではなくデバイスをテストすることを目的としているため、パスの全長、インターフェイスの数、ランダムフレーム損失の複合リスクを最小限に抑えることができます。

In a production network, however, there will be many interfaces and many kilometers of path under test. This considerably increases the risk of random frame loss.


The risk of frame loss caused by outside effects is significantly higher in production networks, and significantly higher with long paths (both those with long physical path lengths, and those with large numbers of interfaces in the path). Thus, the risk of falsely low reported Throughput using an [RFC2544] methodology test is considerably increased in a production network.


Therefore, to successfully conduct tests with similar objectives to those in [RFC2544] in a production network, it will be necessary to develop modifications to the methodologies defined in [RFC2544] and standards to describe them. See [Bryant] for an in-progress effort and [Y.1564] for an approved method adapted to production service activation.


Authors' Addresses


Scott Bradner Harvard University 1350 Mass. Ave., Room 760 Cambridge, MA 02138 USA

スコットブラドナーハーバード大学1350 Mass。Ave.、Room 760 Cambridge、MA 02138 USA

   Phone: +1 617 495 3864

Kevin Dubray Juniper Networks


Jim McQuaid Turnip Video 6 Cobbleridge Court Durham, North Carolina 27713 USA

Jim McQuaid Turnip Video 6 Cobbleridge Court Durham、North Carolina、27713 USA

   Phone: +1 919-619-3220

Al Morton AT&T Labs 200 Laurel Avenue South Middletown,, NJ 07748 USA

Al Morton AT&T Labs 200 Laurel Avenue South Middletown ,, NJ 07748 USA

   Phone: +1 732 420 1571
   Fax:   +1 732 368 1192