Large-scale data stream processing has come a long way to where it is today. It combines all the essential requirements of modern data analytics: subsecond latency, high throughput and impressively, strong consistency. Apache Flink is a system that serves as a proof-of-concept of these characteristics and it is mainly well-known for its lightweight fault tolerance. Data engineers and analysts can now let the system handle Terabytes of computational state without worrying about failures that can potentially occur.

In this talk, I am going to explain all the fundamental challenges behind exactly-once processing guarantees in large-scale streaming in a simple and intuitive way. I will further guide you through the tricks and pitfalls that we faced in our effort to make state management easy to use, transparent and yet extraordinarily powerful. Finally, I will demonstrate how you can declare state and the in-flight protocol that is running underneath your processing pipeline to guarantee that your computation will always run consistently and uninterrupted, until infinity.