Singularity is the most widely used container solution in High Performance Computing (HPC). Enterprise users interested in AI, Deep Learning, compute-driven analytics, and IoT, are increasingly demanding HPC-like compute infrastructures. Singularity has many features that make it the preferred container solution for this new type of “Enterprise Performance Computing” (EPC) workload.

2018 has proven to be a remarkable year for Singularity. In this talk we will take a look back at what has been accomplished this year, including what’s new in the Version 3.0 release. For this release, the core Singularity codebase was rewritten from a Bash/Python/C combination into a Go-C architecture. Motivated at the outset by the need to both modernize and unify the Singularity codebase, Go emerged as the obvious choice to replace scripting languages Bash and Python. While advantageous in and of itself, from the perspective of software lifecycle management for example, the choice of Go offered the potential for a significant strategic upside. Briefly, it is increasingly the case that Go is the go-to language for many of the major projects in the broader container ecosystem. The second big change is the format used for container images, as Singularity 3.0 introduced the Singularity Image Format (SIF) - a new file format specifically tailored for container images. SIF allows for storage of the different components that make up a container. For example, SIF-based containers may include OS partition images (read-only), user writable sections, recipes used to create the container, cryptographic signatures for data integrity and authenticity, and whatever else the community can think of - as SIF is an extensible format. A SIF file resembles a generic Linux file system in structure and content. A global header identifies the file as being of type SIF, and the necessary metadata to detail the remaining content of the image file. Following these descriptors is the corresponding data - namely the OS partition image, recipe, environment variables and signature blocks. As Singularity is drawn into full compliance with standards emerging from the Open Container Initiative (OCI), data encryption is a requirement slated for immediate action. Also related to OCI compliance, and involving SIF specifically, will be the need to harmonize the existing signing and verification capability inherent in SIF with those requirements currently emerging from within the OCI. Because extensibility was designed into SIF from the outset, OCI compliance can be addressed through technically efficient and effective means. Finally, and related to compliance with respect to the OCI Image Specification, will be efforts to ensure that SIF images can be appropriately ‘ingested’ – e.g., for subsequent use by an OCI-compliant runtime. In other words, SIF will become a packaging format for OCI-based containers. Still related to SIF, compliance will include the ability to mount an arbitrary number of SIF images as an OCI bundle. Needed for compliance with the OCI Runtime Specification, at a high level, this translates to mounting an arbitrary number of files – as SIF encapsulates each Singularity container into a single file. Keywords Go, HPC, Linux Containers, Singularity, Docker, OCI