With LinuxBoot we became vendors of our own system firmware. In order to go to production we need a reliable quality assurance process, and firmware testing was a necessity. In this talk we are presenting ConTest (short for Continuous Testing), a modular framework aimed at automating system testing workflows, like firmware validation and provisioning. ConTest has several goals in mind: being open source and community-driven; validate as much as possible at compile time and at job submission time, to minimize unnecessary operations and run-time failures; being lightweight and infrastructure-agnostic, so it can run in Facebook’s datacenters as well as on a Raspberry Pi; being composable, thanks to an interface-and-plugins architecture; being user-oriented so that it’s not necessary to know the internals to use it effectively; and being metrics and events driven, so that users can gain valuable insights about their jobs, more than just success rate (e.g. micro-benchmarking and trend analysis).

ConTest is aimed at anyone who need to automate system-level testing. Various plugins are provided out of the box, with examples on how to use them. The users can combine them like building blocks using a simple job description format based on JSON, and test scenarios of variable complexity. When default plugins are not enough, for example in order to talk to a custom service, users can develop new plugins, and plug them just like if they were part of the core framework. Open-sourcing your own plugins is always appreciated!