Many language runtimes like the JVM rely on JIT compilation to improve application performance, but JIT compilers actively compete with applications for CPU and memory resources. Because of this, a JIT compiler's activities may hinder application throughput and create performance hiccups that can affect quality of service while also complicating resource provisioning. On top of that, it takes a while even for the best JIT compilers to fully compile the performance critical methods of an application to deliver the steady state performance we all expect from the JVM. An approach we are exploring is to decouple the JIT compiler from the rest of the JVM so that JIT compilation can operate as a separate, remote service. This approach opens up the possibility of an elastic, cloud based service offering JIT compilation for many applications, languages, and even different architectures simultaneously.

In this talk we will present our experience building an open source JIT-as-a-service prototype based on the Eclipse OpenJ9 JVM. We will discuss some of the engineering challenges we encountered, demonstrate the advantages we've seen thus far, and present performance data for enterprise grade Java EE benchmarks.