On Mon, Jun 3, 2019 at 5:14 PM Yohei Onishi <[email protected]> wrote:
> Hi Nicolas, > > Are you running you job on Dataflow? According to GCP support, Dataflow > currently does not support Schema Registry. But we can still use Schema > Registry to serialize / deserialize your message using > custom KafkaAvroSerializer. > > In my case I implemented my custom KafkaAvroDeserializer to deserialize > Avro format message. > You can implement custom KafkaAvroSerializer in a similar way. > > https://stackoverflow.com/questions/54755668/how-to-deserialising-kafka-avro-messages-using-apache-beam/55917157#55917157 > > I hope this helps you. > > Yohei Onishi > > > On Wed, May 29, 2019 at 5:44 PM Nicolas Delsaux <[email protected]> > wrote: > >> Hello all >> >> I have a beam job that I use to read messages from RabbitMq t write them >> in kafka. >> >> As of now, messages are read/written as JSON. >> >> Obviously, it's not that optimal storage, so i would like to transform >> the messages to avro prior to write them in Kafka. I have the URL of a >> schema registry I can use to store/get my schema. >> > Can you simply have a ParDo that sits in between read and write steps and perform the conversion (and also connect to schema registry) ? > >> But I see nowhere in Beam documentation how to transform my JSON into >> Avro data (except by deserializing my JSON to a java class that i will >> later transform into avro). Is that deserialization to class the only >> way ? or is it possible to generate an avro generic record from my json >> "directly" ? >> >> Once my avro data is generated, how can I write it to my Kafka topic ? >> >> >> Thanks ! >> >>
