Successful CI initiatives such as KernelCI and MesaCI have been leveraging the Collabora LAVA lab for their automated upstream testing and development workflows, with over a hundred thousand test jobs running monthly on over 100 devices. Maintaining and scaling up on such a large lab comes with its own sets of requirements and challenges. As coverage increases in the project pipelines, the impact of any unreliability or failure in the hardware pool multiplies. Continuous monitoring and recovery mechanisms are crucial to ensure good device availability and reliability.

We will discuss the lifecycle of devices inside the lab from the bring-up to the day-to-day care, as well as the ongoing efforts to make the Collabora LAVA lab more robust as it grows.