Hi Alexey, The short answer is, I hope the code in my sample project makes its way back into Beam, but it may take some time. Hopefully until then people who are in a similar position have a demo of a workaround they can work with.
The longer answer is: The history of this is a bit lengthy, let me make a summary for those who are not aware of it. 1. Beam uses old versions (5.3) of Confluent. Those versions do not support protobuf or JSON schemas. That was added in 5.5. Confluent is now at version 6. 2. The newer Confluent libs depend on Avro 1.9 (last time I checked). Which conflicts with Beams dependency on Avro 1.8. 3. Avro 1.9+ has breaking changes from 1.8. 4. There were at least two attempts at updating Beam to use Avro 1.9. One of those was mine. It included an attempt to move all of Avro outside of Beam core to it's own module, like Beam's protobuf support. 5. Unfortunately, that isn't a viable path forward for the community as it breaks Beam's backwards compatibility. But it did provide insights on the spread of Avro inside of Beam. 6. The latest agreement is that there needs some tests that need to be added to Beam to check compatibility with Avro 1.9+. I wanted to take this task up, but didn't get to it yet. Furthermore, I'm not sure how those tests are going to help with the upgrade of the Confluent libs as I'm not sure we can have two versions of Avro inside of Beam. I'll have to check to see what's possible. Once the Confluent libs are updated, adding support for protofbufs and the Confluent schema registry looks trivial. We just need to decide if we use the ConfluentSchemaRegistryDeserializerProvider for the task or if we have a separate class for similar functionality using protobuf. On Tue., Jan. 12, 2021, 09:58 Alexey Romanenko, <aromanenko....@gmail.com> wrote: > Hi Cristian, > > Great! > > Would you be interested to add this functionality into > original ConfluentSchemaRegistryDeserializerProvider (along with Json > support maybe) [1] ? > Though, it seems that it will require an update of Confluent deps to at > least version 5.5 [2] and it can be blocked by Beam Avro deps update. > > [1] https://issues.apache.org/jira/browse/BEAM-9330 > [2] https://www.confluent.io/blog/introducing-confluent-platform-5-5/ > > On 9 Jan 2021, at 05:16, Cristian Constantinescu <zei...@gmail.com> wrote: > > Hi everyone, > > Beam currently has a dependency on older versions of the Confluent libs. > It makes it difficult to use Protobufs with the Confluent Schema Registry > as ConfluentSchemaRegistryDeserializerProvider only supports Avro. > > I put up together a very simple project to demo how it can be done without > touching any files inside of Beam. You can find it here: > https://github.com/zeidoo/beam-confluent-schema-registry-protobuf > > Any comments are welcomed, especially if there are better ways of doing > things. > > Cheers! > > >
