Quickwit is an opensource distributed search engine. The most challenging component of a search engine is its indexing pipeline.

It takes a stream of JSON documents, processes them in batches, and emits index artefacts we call splits.

We needed our implementation to be robust, testable, observable, and efficient... But even more importantly, our code had to be easy to work with.

I will explain why we chose to develop our own actor framework and discuss the unique features of our implementation.