Many emerging scientific workflows that target high-end HPC systems require a complex interplay with resource and job management software (RJMS). However, portable, efficient and easy-to-use scheduling of these workflows is still an unsolved problem. In this talk, we present Flux, a next-generation RJMS designed specifically to address the key scheduling challenges of modern workflows in a scalable, easy-to-use, and portable manner. At the heart of Flux lies its ability to be seamlessly nested within batch allocations created by itself as well as other system schedulers (e.g., SLURM, MOAB, LSF, etc), serving the target workflows as their “personal RJMS instances”. In particular, Flux’s consistent and rich set of well-defined APIs portably and efficiently support those workflows that can feature non-traditional patterns such as complex co-scheduling, massive ensembles of small jobs and coordination among jobs in an ensemble.

We will also cover how the Flux-Framework project is structured around open-source development, including our use of the Collective Code Construction Contract (C4), RFCs, LGPL, and various online open-source platforms. We discuss how these choices of open-source processes have influenced the repo structure, the code, our collaborations, and even the sub-teams within the project.