Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-07-25 Thread Danny Cranmer
t; > > > > As the design is significantly different, there will be a new period of > > discussion before going to vote. > > > > > > > > Fyi I will be on vacation after this week until the 13th of August. > > > > > > > > Kind regard

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-07-18 Thread Martijn Visser
, there will be a new period of > discussion before going to vote. > > > > Fyi I will be on vacation after this week until the 13th of August. > > > > Kind regards, David. > > > > > > > > *From: *David Radley > *Date: *Tuesday, 9 July 2024 a

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-07-16 Thread David Radley
: Tuesday, 9 July 2024 at 11:17 To: dev@flink.apache.org Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi Kevin, I have agreed a design with Chesnay and Danny. I am implementing a prototype, to prove it works, then will update the Flip text with the new design. Initial testing

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-07-09 Thread David Radley
identifier, I hope to put up a new design in the Flip by the end of next week, for wider review, Kind regards, David. From: Kevin Lam Date: Monday, 8 July 2024 at 21:16 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format Hi David, Any updates on the Kafka

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-07-08 Thread Kevin Lam
>Kind regards, David. > > > > > > > From: David Radley > Date: Wednesday, 29 May 2024 at 11:39 > To: dev@flink.apache.org > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi Danny, > Thank you for your feedback on this. > > I agree that

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-06-14 Thread David Radley
at 19:32 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format Hello, > I am curious what you mean by abused. I just meant we will end up adding more and more fields to this map over time, and it may be hard to undo. > For Apicurio it can be sent at the

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-29 Thread David Radley
@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format Hello, > I am curious what you mean by abused. I just meant we will end up adding more and more fields to this map over time, and it may be hard to undo. > For Apicurio it can be sent at the start of the payloa

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-24 Thread Danny Cranmer
t; [2] https://github.com/apache/flink-connector-kafka/pull/99 > [3] > https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry > > > > > > > > > > > > > F

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-24 Thread David Radley
/www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-json_registry From: Danny Cranmer Date: Friday, 24 May 2024 at 12:22 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avr

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-24 Thread Danny Cranmer
Hello, Apologies, I am on vacation and have limited access to email. I can see the logic here and why you ended up where you did. I can also see there are other useful metadata fields that we might want to pass through, which might result in this Map being abused (Kafka Topic, Kinesis Shard,

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-22 Thread David Radley
Hi Danny, Did you have a chance you have a look at my responses to your feedback? I am hoping to keep the momentum going on this one, kind regards, David. From: David Radley Date: Tuesday, 14 May 2024 at 17:21 To: dev@flink.apache.org Subject: [EXTERNAL] [DISCUSS] FLIP-XXX Apicurio-avro

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread David Radley
Hi Martijn, I have started a vote thread – please could you update the Flip with the link to the vote thread, Kind regards, David. From: David Radley Date: Thursday, 2 May 2024 at 10:39 To: dev@flink.apache.org Subject: [EXTERNAL] RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread David Radley
Fabulous, thanks Martijn  From: Martijn Visser Date: Thursday, 2 May 2024 at 10:08 To: dev@flink.apache.org Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format Done :) On Thu, May 2, 2024 at 11:01 AM David Radley wrote: > Hi Martijn, > Thank you very much for l

Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread Martijn Visser
024 at 16:31 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > I've copied and pasted it into > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format > ; > please take a

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread David Radley
and verbose. Please could you replace the Flip with content from the above link, Kind regards, David, From: Martijn Visser Date: Wednesday, 1 May 2024 at 16:31 To: dev@flink.apache.org Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi David

Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-01 Thread Martijn Visser
Kind regards, David. > > > From: David Radley > Date: Monday, 22 April 2024 at 09:48 > To: dev@flink.apache.org > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi Martijn, > A gentle nudge, is this ok for you or one of the PMC or committers to >

FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-01 Thread David Radley
Hi Martijn, Any news? Kind regards, David. From: David Radley Date: Monday, 22 April 2024 at 09:48 To: dev@flink.apache.org Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi Martijn, A gentle nudge, is this ok for you or one of the PMC or committers to create

FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-22 Thread David Radley
Hi Martijn, A gentle nudge, is this ok for you or one of the PMC or committers to create a Flip now? Kind regards, David. From: David Radley Date: Monday, 15 April 2024 at 12:29 To: dev@flink.apache.org Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi Martijn

Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-15 Thread David Radley
. If this it too much or too little, I can amend, Kind regards, David. From: Martijn Visser Date: Friday, 12 April 2024 at 18:11 To: dev@flink.apache.org Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi David, I tried, but the format wasn't as the FLIP template expects

Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-12 Thread Martijn Visser
te: Monday, 8 April 2024 at 09:40 > To: dev@flink.apache.org > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi, > I have posted a Google Doc [0] to the mailing list for a discussion thread > for a Flip proposal to introduce a Apicurio-avro format. The discuss

Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-12 Thread David Radley
Hi, A gentle nudge. Please could a committer/PMC member raise the Flip for this, Kind regards, David. From: David Radley Date: Monday, 8 April 2024 at 09:40 To: dev@flink.apache.org Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi, I have posted a Google Doc [0

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-08 Thread David Radley
: Friday, 22 March 2024 at 13:05 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format Hi David, Thanks a lot for clarification. Sounds good to me. Regards, Jeyhun On Fri, Mar 22, 2024 at 10:54 AM David Radley wrote: > Hi Jeyhun, > Thanks for your feedback.

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-08 Thread David Radley
Hi, The discussions seem to have ended. I have made some minor changes to the document [1], I would like to leave the discussion open to the end of this week. If there are no more discussions, I will ask for the Flip to be copied into the standard FLIP location and start the voting next week.

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-04-02 Thread Balint Bene
e compatible as per the > > > https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109 > > Then this should work. > > > > Does this address

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-03-22 Thread Jeyhun Karimov
a.java#L109 > Then this should work. > > Does this address your question? > Kind regards, David. > > > From: Jeyhun Karimov > Date: Thursday, 21 March 2024 at 21:06 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David,

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-03-22 Thread David Radley
/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109 Then this should work. Does this address your question? Kind regards, David. From: Jeyhun Karimov Date: Thursday, 21 March 2024 at 21:06 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-03-21 Thread Jeyhun Karimov
Hi David, Thanks for the FLIP. +1 for it. I have a minor comment. Can you please elaborate more on mechanisms in place to ensure data consistency and integrity, particularly in the event of schema conflicts? Since each message includes a schema ID for inbound and outbound messages, can you

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-03-20 Thread David Radley
Thank you very much for your feedback Mark. I have made the changes in the latest google document. On reflection I agree with you that the globalIdPlacement format configuration should apply to the deserialization as well, so it is declarative. I am also going to have a new configuration option

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-03-20 Thread Mark Nuttall
+1 to this A few small comments: Currently, if users have Avro schemas in an Apicurio Registry (an open source Apache 2 licensed schema registry), then the natural way to work with those Avro flows is to use the schemas in the Apicurio Repository. 'those Avro flows' ... this is the first