Re: Serializers and Schemas

Hi people, This is what I was talking about regarding a generic de/serializer for POJO classes [1]. The Serde class in [2] can be used in both Kafka [3] and Flink [4], and it works out of the box for any POJO class. Do you see anything wrong in this approach? Any way to improve it? Cheers, Matt

Re: Serializers and Schemas

Hi Matt, 1. There’s some in-progress work on wrapper util classes for Kafka de/serializers here [1] that allows Kafka de/serializers to be used with the Flink Kafka Consumers/Producers with minimal user overhead. The PR also has some proposed adds to the documentations for the wrappers. 2. I fe

Re: Serializers and Schemas

Why not use a self-describing format (json), stream as String and read through a json reader and avoid top-level reflection? Github.com/milindparikh/streamingsi https://github.com/milindparikh/streamingsi/tree/master/epic-poc/sprint-2-simulated-data-no-cdc-advanced-eventing/2-dataprocessing ?

Re: Serializers and Schemas

I've read your example, but I've found the same problem. You're serializing your POJO as a string, where all fields are separated by "\t". This may work for you, but not in general. https://github.com/luigiselmi/pilot-sc4-fcd-producer/blob/master/src/main/java/eu/bde/sc4pilot/fcd/FcdTaxiEvent.java

Re: Serializers and Schemas

Hi Matt, I had the same problem, trying to read some records in event time using a POJO, doing some transformation and save the result into Kafka for further processing. I am not yet done but maybe the code I wrote starting from the Flink Forward 2016 training docs