RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Martijn, I have started a vote thread – please could you update the Flip with the link to the vote thread, Kind regards, David. From: David Radley Date: Thursday, 2 May 2024 at 10:39 To: dev@flink.apache.org Subject: [EXTERNAL] RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format Fabulous, thanks Martijn From: Martijn Visser Date: Thursday, 2 May 2024 at 10:08 To: dev@flink.apache.org Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format Done :) On Thu, May 2, 2024 at 11:01 AM David Radley wrote: > Hi Martijn, > Thank you very much for looking at this. In response to your feedback; I > produced a reduced version which is on this link. > > > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing > > The original version you have copied is a bit out-dated and verbose. > Please could you replace the Flip with content from the above link, > Kind regards, David, > > From: Martijn Visser > Date: Wednesday, 1 May 2024 at 16:31 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > I've copied and pasted it into > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format > ; > please take a look if it's as expected. > > Best regards, > > Martijn > > On Wed, May 1, 2024 at 3:47 PM David Radley > wrote: > > > Hi Martijn, > > Any news? > >Kind regards, David. > > > > > > From: David Radley > > Date: Monday, 22 April 2024 at 09:48 > > To: dev@flink.apache.org > > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi Martijn, > > A gentle nudge, is this ok for you or one of the PMC or committers to > > create a Flip now? > >Kind regards, David. > > > > From: David Radley > > Date: Monday, 15 April 2024 at 12:29 > > To: dev@flink.apache.org > > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi Martijn, > > Thanks for looking at this. I have used the template in a new Google Doc > > > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing > . > > I have significantly reduced the content in the Flip, in line with what I > > see as the template and its usage. If this it too much or too little, I > can > > amend, > > > > Kind regards, David. > > > > From: Martijn Visser > > Date: Friday, 12 April 2024 at 18:11 > > To: dev@flink.apache.org > > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi David, > > > > I tried, but the format wasn't as the FLIP template expects, so I ended > up > > needing to change the entire formatting and that was just too much work > to > > be honest. If you could make sure that especially the headers match with > > the FLIP template, and that all of the contents from the FLIP template is > > there, that would make things much easier. > > > > Thanks, > > > > Martijn > > > > On Fri, Apr 12, 2024 at 6:08 PM David Radley > > wrote: > > > > > Hi, > > > A gentle nudge. Please could a committer/PMC member raise the Flip for > > > this, > > > Kind regards, David. > > > > > > > > > From: David Radley > > > Date: Monday, 8 April 2024 at 09:40 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > > Hi, > > > I have posted a Google Doc [0] to the mailing list for a discussion > > thread > > > for a Flip proposal to introduce a Apicurio-avro format. The > discussions > > > have been resolved, please could a committer/PMC member copy the > contents > > > from the Google Doc, and create a FLIP number for this,. as per the > > process > > > [1], > > > Kind regards, David. > > > [0] > > > > > > > > > https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP > > > > > > From: Jeyhun Karimov > > > Date: Friday, 22 March 2024 at 13:05 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > > Hi David, > > > > > > Thanks a lot for clarification. > > > Sounds goo
RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Fabulous, thanks Martijn From: Martijn Visser Date: Thursday, 2 May 2024 at 10:08 To: dev@flink.apache.org Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format Done :) On Thu, May 2, 2024 at 11:01 AM David Radley wrote: > Hi Martijn, > Thank you very much for looking at this. In response to your feedback; I > produced a reduced version which is on this link. > > > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing > > The original version you have copied is a bit out-dated and verbose. > Please could you replace the Flip with content from the above link, > Kind regards, David, > > From: Martijn Visser > Date: Wednesday, 1 May 2024 at 16:31 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > I've copied and pasted it into > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format > ; > please take a look if it's as expected. > > Best regards, > > Martijn > > On Wed, May 1, 2024 at 3:47 PM David Radley > wrote: > > > Hi Martijn, > > Any news? > >Kind regards, David. > > > > > > From: David Radley > > Date: Monday, 22 April 2024 at 09:48 > > To: dev@flink.apache.org > > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi Martijn, > > A gentle nudge, is this ok for you or one of the PMC or committers to > > create a Flip now? > >Kind regards, David. > > > > From: David Radley > > Date: Monday, 15 April 2024 at 12:29 > > To: dev@flink.apache.org > > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi Martijn, > > Thanks for looking at this. I have used the template in a new Google Doc > > > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing > . > > I have significantly reduced the content in the Flip, in line with what I > > see as the template and its usage. If this it too much or too little, I > can > > amend, > > > > Kind regards, David. > > > > From: Martijn Visser > > Date: Friday, 12 April 2024 at 18:11 > > To: dev@flink.apache.org > > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi David, > > > > I tried, but the format wasn't as the FLIP template expects, so I ended > up > > needing to change the entire formatting and that was just too much work > to > > be honest. If you could make sure that especially the headers match with > > the FLIP template, and that all of the contents from the FLIP template is > > there, that would make things much easier. > > > > Thanks, > > > > Martijn > > > > On Fri, Apr 12, 2024 at 6:08 PM David Radley > > wrote: > > > > > Hi, > > > A gentle nudge. Please could a committer/PMC member raise the Flip for > > > this, > > > Kind regards, David. > > > > > > > > > From: David Radley > > > Date: Monday, 8 April 2024 at 09:40 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > > Hi, > > > I have posted a Google Doc [0] to the mailing list for a discussion > > thread > > > for a Flip proposal to introduce a Apicurio-avro format. The > discussions > > > have been resolved, please could a committer/PMC member copy the > contents > > > from the Google Doc, and create a FLIP number for this,. as per the > > process > > > [1], > > > Kind regards, David. > > > [0] > > > > > > > > > https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP > > > > > > From: Jeyhun Karimov > > > Date: Friday, 22 March 2024 at 13:05 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > > Hi David, > > > > > > Thanks a lot for clarification. > > > Sounds good to me. > > > > > > Regards, > > > Jeyhun > > > > > > On Fri, Mar 22, 2024 at 10:54 AM David Radley > > > > wrote: > > > > > > > Hi Jeyhun, > > > > Thanks for your feedback. > > > > > >
Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Done :) On Thu, May 2, 2024 at 11:01 AM David Radley wrote: > Hi Martijn, > Thank you very much for looking at this. In response to your feedback; I > produced a reduced version which is on this link. > > > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing > > The original version you have copied is a bit out-dated and verbose. > Please could you replace the Flip with content from the above link, > Kind regards, David, > > From: Martijn Visser > Date: Wednesday, 1 May 2024 at 16:31 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > I've copied and pasted it into > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format > ; > please take a look if it's as expected. > > Best regards, > > Martijn > > On Wed, May 1, 2024 at 3:47 PM David Radley > wrote: > > > Hi Martijn, > > Any news? > >Kind regards, David. > > > > > > From: David Radley > > Date: Monday, 22 April 2024 at 09:48 > > To: dev@flink.apache.org > > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi Martijn, > > A gentle nudge, is this ok for you or one of the PMC or committers to > > create a Flip now? > >Kind regards, David. > > > > From: David Radley > > Date: Monday, 15 April 2024 at 12:29 > > To: dev@flink.apache.org > > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi Martijn, > > Thanks for looking at this. I have used the template in a new Google Doc > > > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing > . > > I have significantly reduced the content in the Flip, in line with what I > > see as the template and its usage. If this it too much or too little, I > can > > amend, > > > > Kind regards, David. > > > > From: Martijn Visser > > Date: Friday, 12 April 2024 at 18:11 > > To: dev@flink.apache.org > > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi David, > > > > I tried, but the format wasn't as the FLIP template expects, so I ended > up > > needing to change the entire formatting and that was just too much work > to > > be honest. If you could make sure that especially the headers match with > > the FLIP template, and that all of the contents from the FLIP template is > > there, that would make things much easier. > > > > Thanks, > > > > Martijn > > > > On Fri, Apr 12, 2024 at 6:08 PM David Radley > > wrote: > > > > > Hi, > > > A gentle nudge. Please could a committer/PMC member raise the Flip for > > > this, > > > Kind regards, David. > > > > > > > > > From: David Radley > > > Date: Monday, 8 April 2024 at 09:40 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > > Hi, > > > I have posted a Google Doc [0] to the mailing list for a discussion > > thread > > > for a Flip proposal to introduce a Apicurio-avro format. The > discussions > > > have been resolved, please could a committer/PMC member copy the > contents > > > from the Google Doc, and create a FLIP number for this,. as per the > > process > > > [1], > > > Kind regards, David. > > > [0] > > > > > > > > > https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP > > > > > > From: Jeyhun Karimov > > > Date: Friday, 22 March 2024 at 13:05 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > > Hi David, > > > > > > Thanks a lot for clarification. > > > Sounds good to me. > > > > > > Regards, > > > Jeyhun > > > > > > On Fri, Mar 22, 2024 at 10:54 AM David Radley > > > > wrote: > > > > > > > Hi Jeyhun, > > > > Thanks for your feedback. > > > > > > > > So for outbound messages, the message includes the global ID. We > > register > > > > the schema and match on the artifact id. So if the schema then &g
RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Martijn, Thank you very much for looking at this. In response to your feedback; I produced a reduced version which is on this link. https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing The original version you have copied is a bit out-dated and verbose. Please could you replace the Flip with content from the above link, Kind regards, David, From: Martijn Visser Date: Wednesday, 1 May 2024 at 16:31 To: dev@flink.apache.org Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi David, I've copied and pasted it into https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format; please take a look if it's as expected. Best regards, Martijn On Wed, May 1, 2024 at 3:47 PM David Radley wrote: > Hi Martijn, > Any news? >Kind regards, David. > > > From: David Radley > Date: Monday, 22 April 2024 at 09:48 > To: dev@flink.apache.org > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi Martijn, > A gentle nudge, is this ok for you or one of the PMC or committers to > create a Flip now? >Kind regards, David. > > From: David Radley > Date: Monday, 15 April 2024 at 12:29 > To: dev@flink.apache.org > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi Martijn, > Thanks for looking at this. I have used the template in a new Google Doc > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing. > I have significantly reduced the content in the Flip, in line with what I > see as the template and its usage. If this it too much or too little, I can > amend, > > Kind regards, David. > > From: Martijn Visser > Date: Friday, 12 April 2024 at 18:11 > To: dev@flink.apache.org > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > I tried, but the format wasn't as the FLIP template expects, so I ended up > needing to change the entire formatting and that was just too much work to > be honest. If you could make sure that especially the headers match with > the FLIP template, and that all of the contents from the FLIP template is > there, that would make things much easier. > > Thanks, > > Martijn > > On Fri, Apr 12, 2024 at 6:08 PM David Radley > wrote: > > > Hi, > > A gentle nudge. Please could a committer/PMC member raise the Flip for > > this, > > Kind regards, David. > > > > > > From: David Radley > > Date: Monday, 8 April 2024 at 09:40 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi, > > I have posted a Google Doc [0] to the mailing list for a discussion > thread > > for a Flip proposal to introduce a Apicurio-avro format. The discussions > > have been resolved, please could a committer/PMC member copy the contents > > from the Google Doc, and create a FLIP number for this,. as per the > process > > [1], > > Kind regards, David. > > [0] > > > > > https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP > > > > From: Jeyhun Karimov > > Date: Friday, 22 March 2024 at 13:05 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi David, > > > > Thanks a lot for clarification. > > Sounds good to me. > > > > Regards, > > Jeyhun > > > > On Fri, Mar 22, 2024 at 10:54 AM David Radley > > wrote: > > > > > Hi Jeyhun, > > > Thanks for your feedback. > > > > > > So for outbound messages, the message includes the global ID. We > register > > > the schema and match on the artifact id. So if the schema then evolved, > > > adding a new version, the global ID would still be unique and the same > > > version would be targeted. If you wanted to change the Flink table > > > definition in line with a higher version, then you could do this – the > > > artifact id would need to match for it to use the same schema and a > > higher > > > artifact version would need to be provided. I notice that Apicurio has > > > rules around compatibility that you can configure, I suppose if we > > attempt > > > to create an artifact that breaks these rules , then the register > schema > > > will fail and the associated operation should fail (e.g. an insert). I &g
Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David, I've copied and pasted it into https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format; please take a look if it's as expected. Best regards, Martijn On Wed, May 1, 2024 at 3:47 PM David Radley wrote: > Hi Martijn, > Any news? >Kind regards, David. > > > From: David Radley > Date: Monday, 22 April 2024 at 09:48 > To: dev@flink.apache.org > Subject: FW: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi Martijn, > A gentle nudge, is this ok for you or one of the PMC or committers to > create a Flip now? >Kind regards, David. > > From: David Radley > Date: Monday, 15 April 2024 at 12:29 > To: dev@flink.apache.org > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi Martijn, > Thanks for looking at this. I have used the template in a new Google Doc > https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing. > I have significantly reduced the content in the Flip, in line with what I > see as the template and its usage. If this it too much or too little, I can > amend, > > Kind regards, David. > > From: Martijn Visser > Date: Friday, 12 April 2024 at 18:11 > To: dev@flink.apache.org > Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > I tried, but the format wasn't as the FLIP template expects, so I ended up > needing to change the entire formatting and that was just too much work to > be honest. If you could make sure that especially the headers match with > the FLIP template, and that all of the contents from the FLIP template is > there, that would make things much easier. > > Thanks, > > Martijn > > On Fri, Apr 12, 2024 at 6:08 PM David Radley > wrote: > > > Hi, > > A gentle nudge. Please could a committer/PMC member raise the Flip for > > this, > > Kind regards, David. > > > > > > From: David Radley > > Date: Monday, 8 April 2024 at 09:40 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi, > > I have posted a Google Doc [0] to the mailing list for a discussion > thread > > for a Flip proposal to introduce a Apicurio-avro format. The discussions > > have been resolved, please could a committer/PMC member copy the contents > > from the Google Doc, and create a FLIP number for this,. as per the > process > > [1], > > Kind regards, David. > > [0] > > > > > https://docs.google.com/document/d/14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w/edit?usp=sharing > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-CreateyourOwnFLIP > > > > From: Jeyhun Karimov > > Date: Friday, 22 March 2024 at 13:05 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi David, > > > > Thanks a lot for clarification. > > Sounds good to me. > > > > Regards, > > Jeyhun > > > > On Fri, Mar 22, 2024 at 10:54 AM David Radley > > wrote: > > > > > Hi Jeyhun, > > > Thanks for your feedback. > > > > > > So for outbound messages, the message includes the global ID. We > register > > > the schema and match on the artifact id. So if the schema then evolved, > > > adding a new version, the global ID would still be unique and the same > > > version would be targeted. If you wanted to change the Flink table > > > definition in line with a higher version, then you could do this – the > > > artifact id would need to match for it to use the same schema and a > > higher > > > artifact version would need to be provided. I notice that Apicurio has > > > rules around compatibility that you can configure, I suppose if we > > attempt > > > to create an artifact that breaks these rules , then the register > schema > > > will fail and the associated operation should fail (e.g. an insert). I > > have > > > not tried this. > > > > > > > > > For inbound messages, using the global id in the header – this targets > > one > > > version of the schema. I can create different messages on the topic > built > > > with different schema versions, and I can create different tables in > > Flink, > > > as long as the reader and writer schemas are compatible as per the > > > > > > https://github.com/apache/flink/blob/779459168c46b7b4c600ef52f99a5435f81b9048/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L109 > > > Then this should work. > > > > > > Does this address your question? > > > Kind regards, David. > > > > > > > > > From: Jeyhun Karimov > > > Date: Thursday, 21 March 2024 at 21:06 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > > Hi David, > > > > > > Thanks for the FLIP. +1 for it. > > > I have a minor comment. > > > > > > Can you please elaborate more on mechanisms in place to ensure data > > > consistency and integrity, particularly in the event of schema > conflicts? > >
