[DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format #2

Hi All, I'm going to attempt to revive the discussion for FLIP-34440 since the original discussion thread has stalled. Fresh start! :) Let's start with discussing the approach as per the Flink contribution guides! First of all, I think it's imp

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

> was wondering why? In the confluent avro and existing debezium >> formats, >> > > there is no schema id in the config, but there is the ability to >> specify >> > a >> > > complete schema. In the protobuf format there is no schema id. >>

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

ed something like schemaIdForSink or > > schemaIdForSerilization; to prevent confusion with the deserialization > > case. We could have the schema as you suggest so we are compatible with > the > > confluent avro format. > > > > > > WDYT? > > Kind regards, Da

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

; > > I would suggest aiming to copy what the confluent DeSer libraries do > > rather than having a schema id hard coded in the config. > > > > WDYT? > > Kind regards, David. > > > > From: Kevin Lam > > Date: Tuesday, 26 March 2024 at 20:06 > > To: de

RE: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

are compatible with the confluent avro format. WDYT? Kind regards, David. From: Anupam Aggarwal Date: Saturday, 13 April 2024 at 16:08 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format Hi David, Thank you for the suggestion

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

y what the confluent DeSer libraries do > rather than having a schema id hard coded in the config. > > WDYT? > Kind regards, David. > > From: Kevin Lam > Date: Tuesday, 26 March 2024 at 20:06 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] FLINK-

RE: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

> Some thoughts on this. > > > > > > > > I suggested an Apicurio registry format in the dev list, and > > was > > > > > > advised > > > > > > > > to raise a FLIP for this, I suggest the same would apply here > > (or >

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

gt; > > > > > > this > > > > > > > > back to open source. > > > > > > > > If you already have an implementation, let's decide which one > > to > > > > use > > > > > :) > > > > > > > > > > > >

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

> > > I suggested an Apicurio registry format in the dev list, and > > was > > > > > > advised > > > > > > > > to raise a FLIP for this, I suggest the same would apply here > > (or > > > > the > > > > > >

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

one). I am prototyping > > an > > > > Avro > > > > > > > Apicurio format, prior to raising the Flip, and notice that > the > > > > > > readSchema > > > > > > > in the SchemaCoder only takes a byte array ,but I need to pass &g

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

s). > > > > > > > > > > > > I assume: > > > > > > > > > > > > * for the confluent Protobuf format you would extend the > > Protobuf > > > > > > format to drive some Schema Registry logic for Protobuf (simi

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

it) where the magic byte _ schema id can be obtained and > > the > > > > > schema looked up using the Confluent Schema registry. > > > > > * It would be good if any protobuf format enhancements for > Schema > > > > > registries pass down the Kafka

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

istry > > > could > > > > work with this. > > > > * It would make sense to have the Confluent schema lookup in > common > > > > code, which is part of the SchemaCoder readSchema logic. > > > > * I assume the ProtobufSchemaCoder re

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

t would make sense to have the Confluent schema lookup in common > > > code, which is part of the SchemaCoder readSchema logic. > > > * I assume the ProtobufSchemaCoder readSchema would return a > Protobuf > > > Schema object. > > > > > > > &g

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

her these Kafka only formats should be moved to the > > Kafka connector repo, or whether they might in the future be used outside > > Kafka – e.g. Avro/Protobuf files in a database. > >Kind regards, David. > > > > > > From: Kevin Lam > > Date: Wednesday, 21 Fe

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

te: Wednesday, 21 February 2024 at 18:51 > To: dev@flink.apache.org > Subject: [EXTERNAL] [DISCUSS] FLINK-34440 Support Debezium Protobuf > Confluent Format > I would love to get some feedback from the community on this JIRA issue: > https://issues.apache.org/jira/projects/FLINK/issu

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

@flink.apache.org Subject: [EXTERNAL] [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format I would love to get some feedback from the community on this JIRA issue: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34440 I am looking into creating a PR and would appreciate some

[DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

I would love to get some feedback from the community on this JIRA issue: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34440 I am looking into creating a PR and would appreciate some review on the approach. In terms of design I think we can mirror the `debezium-avro-confluent` and `a