A common choice is Apache Avro. You can to define a schema for you Pojos and generate serializers and deserializers.
2016-11-18 5:11 GMT+01:00 Matt <dromitl...@gmail.com>: > Just to be clear, what I'm looking for is a way to serialize a POJO class > for Kafka but also for Flink, I'm not sure the interface of both frameworks > are compatible but it seems they aren't. > > For Kafka (producer) I need a Serializer and a Deserializer class, and for > Flink (consumer) a SerializationSchema and DeserializationSchema class. > > Any example of how to put this together would be greatly appreciated. > > On Thu, Nov 17, 2016 at 9:12 PM, Dromit <dromitl...@gmail.com> wrote: > >> Tzu-Li Tai, thanks for your response. >> >> I've seen the example you mentioned before, TaxiRideSchema.java, but it's >> way too simplified. >> >> In a real POJO class you may have multiple fields such as integers, >> strings, doubles, etc. So serializing them as a string like in the example >> wouldn't work (you can't put together two arbitrary strings and later split >> the byte array to get each of them, same for two integers, and nearly any >> other types). >> >> I feel there should be a more general way of doing this regardless of the >> fields on the class you're de/serializing. >> >> What do you do in these cases? It should be a pretty common scenario! >> >> Regards, >> Matt >> >> On Wed, Nov 16, 2016 at 2:01 PM, Philipp Bussche < >> philipp.buss...@gmail.com> wrote: >> >>> Hi Dromit >>> >>> I started using Flink with Kafka but am currently looking into Kinesis to >>> replace Kafka. >>> The reason behind this is that eventually my application will run in >>> somebody's cloud and if I go for AWS then I don't have to take care of >>> operating Kafka and Zookeeper myself. I understand this can be a >>> challenging >>> task. >>> Up to know where the Kafka bit is only running in a local test >>> environment I >>> am happy running it as I just start 2 Docker containers and it does the >>> job. >>> But this also means I have no clue how Kafka really works and what I >>> need to >>> be careful with. >>> Besides knowledge which is required as it seems for Kafka costs is >>> another >>> aspect here. >>> If one wants to operate a Kafka cluster plus Zookeeper on let's say the >>> Amazon cloud this might actually be more expensive than "just" using >>> Kinesis >>> as a service. >>> There are apparently draw backs in terms of functionality and performance >>> but for my use case that does not seem to matter. >>> >>> Philipp >>> >>> >>> >>> -- >>> View this message in context: http://apache-flink-user-maili >>> ng-list-archive.2336050.n4.nabble.com/Why-use-Kafka-after- >>> all-tp10112p10155.html >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive at Nabble.com. >>> >> >> >
