Kafka Streams and Protobuf
stream processing at trivago
- Track: HPC, Big Data and Data Science devroom
- Room: H.2213
- Day: Saturday
- Start: 17:30
- End: 17:55
A lot of components in the Kafka and hadoop ecosystem assume you are working with avro messages. In this talk I will highlight some of the advantages and challenges of stream processing with protobuf encoded messages, and why it might be worth using them.
A powerful feature of protobuf messages is its handling of unknown fields, and the ability to partially parse messages with different schema versions or no schema at all. This helps in stream processing when you don't actually need to know all message fields in order to process and forward them.
Speakers
 |
Clemens Valiente |