With redpesk, we provide customers the ability to cross-build an embedded, CentOS Stream-based Linux distribution in the cloud. This requires a significant infrastructure: Koji/RPM builders, Angular-based WebUI, Gitlab forge, network and RPM package dependency management, Qemu test lab management, all need to come together and be connected, in a mix of Qemu virtual machines and LXC containers. Fortunately, Ansible and Proxmox comes to the rescue to manage this complexity.

In this talk, we'll present our architecture of a self-contained CI/CD environment in the cloud, to cross-build RPM packages and Linux images. We will then dive into the specifics of using Ansible to drive Proxmox and deploy a mix of Packer-built Qemu virtual machines and LXC containers. Those provide a full Koji build system (hub and builders), an Angular frontend, Go backend, a Gitlab forge as well as network isolation/firewalling and a Qemu virtual target lab. We'll continue with lessons learned from doing these deployments for multiple customers. We will finish describing solutions we are currently working on, like Ansible AWX, to address the challenges of doing it at scale and increase automation.