The technical challenges in developing a benchmark aren’t limited to avoiding lying to the reader. Figuring out how to measure something reliably and repeatably in a dynamic system requires a deep understanding of all of the system components and how interactions among them can conspire to create false measurements. A non-networked system, where the interconnections and interactions among the various components are, or should be, visible, is still complex enough to trip up many developers. A networked system is far harder to measure for several reasons, including: asynchrony, visibility and the lack of well synchronized clocks.

In this talk we will present our experiences measuring various components of the FreeBSD network subsystems, including generic forwarding, IPSEC, TCP as well as both of the commonly used firewalls (IPFW and PF). We present not only our measurements but the lessons learned in finding the right measurements and how to reliably reproduce our results. Common pitfalls are called out and addressed and a set of representative tests are given. A secondary outcome of this work is a simples, open source, system for network test coordination, Conductor, which is also described. All of our code, scripts, configurations, and test results are open source and hosted on github. This work is part of an ongoing longitudinal study of kernel network performance which continues to evolve over time.