Maintaining software systems for a long time is hard. Backporting kernel patches is a complex and expensive overhead. The Linux project's position is - quite rightly - to instruct down-streams to upgrade to the latest release.

In practice, upgrading is difficult, scary and sometimes avoided. It takes time and effort before there is enough confidence that new releases will work in context: full system testing in embedded environments can be arduous. What if we could have continuous, automated full system tests, from from UI, to OS, to kernel, on hardware?

This talk will provide an overview of how this can be achieved with a combination of OpenQA, LAVA and Continuous Integration pipelines. The talk will cover: - How the same tests can be used in both kernel space and user space testing - How the same tests can run in both virtualisation (with OpenQA and QEMU, developed originally for GNOME-OS) and also on hardware - How images are then deployed and tested in hardware (LAVA triggers OpenQA via VNC to begin testing on boards)