Brussels / 30 & 31 January 2016


Measure Twice, Code Once

Understanding Network Benchmarks

The networking subsystems of any operating system have grown in complexity as the set of protocols and features supported has grown since the birth of the Internet. Firewalls, Virtual Private Networking, and IPv6 are just a few of the features present in the kernel that were not even envisioned when the original UNIX releases were developed over 30 years ago. Advances in networking hardware, with 10Gbps NIC cards being available for only a few hundred dollars, have far outstripped the speeds for which the kernel’s network software was originally written. As with the increasing speed of processors over the last 30 years, systems developers and integrators have always depended on the next generation of hardware to solve the current generation’s performance bottlenecks, often without resorting to any coherent form of measurement. This presentation shows developers and systems integrators at all proficiency levels how to benchmark networking systems, with specific examples. Common pitfalls are called out and addressed and a set of representative tests are given, all using open source software.

The technical challenges in developing a benchmark aren’t limited to avoiding lying to the reader. Figuring out how to measure something reliably and repeatably in a dynamic system requires a deep understanding of all of the system components and how interactions among them can conspire to create false measurements. A non-networked system, where the interconnections and interactions among the various components are, or should be, visible, is still complex enough to trip up many developers. A networked system is far harder to measure for several reasons, including: asynchrony, visibility and the lack of well synchronized clocks.

In this talk we will present our experiences measuring various components of the FreeBSD network subsystems, including generic forwarding, IPSEC, TCP as well as both of the commonly used firewalls (IPFW and PF). We present not only our measurements but the lessons learned in finding the right measurements and how to reliably reproduce our results. Common pitfalls are called out and addressed and a set of representative tests are given. A secondary outcome of this work is a simples, open source, system for network test coordination, Conductor, which is also described. All of our code, scripts, configurations, and test results are open source and hosted on github. This work is part of an ongoing longitudinal study of kernel network performance which continues to evolve over time.


Photo of George Neville-Neil George Neville-Neil