The recent surge of CI systems has created an interesting new problem, where a failure occurs in a specific test environment but does not appear in the familiar environment used by the developer.

This problem is compounded by the batch nature of such systems, where a developer can merely push additional patches to some branch to trigger an asynchronous execution process.

During the development of Ubuntu Core operating system, this problem was amplified by the fact building and testing a full OS image is a time-consuming process, leading to cycles that spanned hours and lead to frustration. Snapd developers created the spread program to solve this, among other, problem.

The SAAS solution became a thin wrapper around spread, which allocates, provisions, uses and finally discards the test environment. Crucially, almost all errors can be reproduced locally, as spread runs as a standalone tool, using QEMU, LXD, Google Compute Engine or Linode as executors.

This allows anyone to run spread locally, in interactive mode, and explore the problem without putting additional load on the centralized CI system, greatly improving the debugging process.