At least since the final Multics report in 1977, researchers identified the ability to isolate kernel subsystems as a critical mechanism for increasing the reliability and security of the kernel. Unfortunately, despite many attempts to introduce isolation to the kernel, modern systems remain monolithic. Historically, software and hardware mechanisms introduce a prohibitively high overhead for the isolation of subsystems with the tightest performance budgets. Today, however, the balance of isolation and performance is starting to change with the development of Rust, arguably, the first practical programming language that achieves safety without garbage collection.

RedLeaf is a new operating system developed from scratch in Rust with the goal to explore the impact of language safety on operating system organization, and specifically on the ability to utilize fine-grained isolation and its benefits in the kernel. In contrast to commodity systems, RedLeaf does not rely on hardware address spaces for isolation and instead uses only type and memory safety of the Rust language. Departure from costly hardware isolation mechanisms allows us to explore the design space of systems that embrace lightweight fine-grained isolation of kernel subsystems. We develop a new abstraction of a language-based isolation domain that implements a unit of information hiding and isolation of faults. Domains can be dynamically loaded and cleanly terminated. Building on RedLeaf isolation mechanisms, we demonstrate the possibility to implement end-to-end zero-copy, fault isolation, and transparent recovery of device drivers.

To evaluate the practicality of RedLeaf abstractions, we implemented a subset of the POSIX interface as a collection of RedLeaf domains. RedLeaf's isolation mechanisms allow us to support the transparent recovery of device drivers. To demonstrate that Rust and fine-grained isolation introduce a practically-acceptable overhead, we develop efficient versions of 10Gbps network and PCIe-attached solid state-disk NVMe device drivers that match the performance of carefully-optimized kernel-bypass device drivers used in modern network and storage processing frameworks.