Re: Avro schema evolution support in AVRO CPP

2024-01-12 Thread John McClean 
fwiw, I'm using it and it works fine, at least for my use cases.

J

On Fri, Jan 12, 2024 at 1:55 AM Martin Grigorov 
wrote:

> Hi Vivek,
>
> I am not sure there is anyone to give you an exact answer. The C++ SDK has
> not been actively developed in the last few years.
> The best is to try it for your use cases and see if it works or not. The
> next step is to contribute Pull Requests for the missing functionalities!
>
> Martin
>
> On Thu, Jan 11, 2024 at 8:59 AM Vivek Kumar <
> vivek.ku...@eclipsetrading.com.invalid> wrote:
>
>> +dev
>>
>>
>> Regards,
>> Vivek Kumar
>>
>> [http://www.eclipsetrading.com/logo.png]
>>
>> Senior Software Developer
>> 23/F One Hennessy
>> 1 Hennessy Road
>> Wan Chai
>> Hong Kong
>> www.eclipsetrading.com
>> +852 2108 7352
>>
>> Follow us today on our online platforms
>> [Facebook][Linked-In]<
>> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
>> https://www.instagram.com/eclipsetrading>
>> 
>> From: Vivek Kumar 
>> Sent: Thursday, January 11, 2024 11:07 AM
>> To: user@avro.apache.org 
>> Subject: Avro schema evolution support in AVRO CPP
>>
>> Hi Avro team,
>>
>> I am writing this email to check the support of Avro schema evolution in
>> CPP - i.e. provide both the producer and consumer schema when decoding the
>> data.
>>
>> I can see that there's a resolvingDecoder function in AVRO CPP that takes
>> two schemas. See
>>
>> https://avro.apache.org/docs/1.10.2/api/cpp/html/index.html#ReadingDifferentSchema
>>
>> But there's a FIXME comment in this function. See
>> https://issues.apache.org/jira/browse/AVRO-3720 and
>> https://github.com/apache/avro/blob/main/lang/c%2B%2B/api/Decoder.hh#L218.
>> Does this mean resolvingDecoder does not work properly? Could you please
>> explain what scenarios are not covered by resolvingDecoder and how can we
>> use it to support "Avro Schema Evolution" in c++?
>>
>> Thanks
>>
>>
>> Regards,
>> Vivek Kumar
>>
>> [http://www.eclipsetrading.com/logo.png]
>>
>> Senior Software Developer
>> 23/F One Hennessy
>> 1 Hennessy Road
>> Wan Chai
>> Hong Kong
>> www.eclipsetrading.com
>> +852 2108 7352
>>
>> Follow us today on our online platforms
>> [Facebook][Linked-In]<
>> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
>> https://www.instagram.com/eclipsetrading>
>>
>

Re: Avro schema evolution support in AVRO CPP

2024-01-12 Thread Andrew Marlow 
"In practice, it is very rare for schema evolution to change the order of
the fields." - I'll say. Since we are talking about a protocol that is
deliberately not self-describing we cannot just pluck out what we want -
how would such code get to it? This is why the standard advice in these
situations is to never reorder, remove or rename fields and to always add
new stuff to the end.

On Fri, 12 Jan 2024 at 13:19, Thiruvalluvan MG 
wrote:

>  Out-of-order fields are not handled transparently in C++ if you are
> manually using the resolving decoder. (It's the same situation in Java as
> well).
> But, in C++ and in Java, if you generate code for the given Avro schema,
> the generated code takes care of the field ordering issue. Similarly, in
> both bindings, Generic data structures work properly with the field-order.
> If you are using the resolving decoder in your code directly, care must be
> exercises, If the reader schema and writer schema are both records and they
> have fields in different order (it is okay to insert or remove fields), the
> protocol is to first get the field-order array (which is essentially a
> permutation of the reader field ids 0 to n-1) from the resolving decoder
> and then read the fields of the reader schema in the order specified in the
> field-order array. This is done in order to avoid buffering by the decoder.
> Buffering can take a large number of allocations if the out-of-order fields
> is an array or map.
> In practice, it is very rare for schema evolution to change the order of
> the fields.
> Thanks
> ThiruOn Friday, 12 January, 2024 at 03:24:11 pm IST, Martin Grigorov <
> mgrigo...@apache.org> wrote:
>
>  Hi Vivek,
>
> I am not sure there is anyone to give you an exact answer. The C++ SDK has
> not been actively developed in the last few years.
> The best is to try it for your use cases and see if it works or not. The
> next step is to contribute Pull Requests for the missing functionalities!
>
> Martin
>
> On Thu, Jan 11, 2024 at 8:59 AM Vivek Kumar <
> vivek.ku...@eclipsetrading.com.invalid> wrote:
>
> > +dev
> >
> >
> > Regards,
> > Vivek Kumar
> >
> > [http://www.eclipsetrading.com/logo.png]
> >
> > Senior Software Developer
> > 23/F One Hennessy
> > 1 Hennessy Road
> > Wan Chai
> > Hong Kong
> > www.eclipsetrading.com
> > +852 2108 7352
> >
> > Follow us today on our online platforms
> > [Facebook][Linked-In]<
> > https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> > https://www.instagram.com/eclipsetrading>
> > 
> > From: Vivek Kumar 
> > Sent: Thursday, January 11, 2024 11:07 AM
> > To: user@avro.apache.org 
> > Subject: Avro schema evolution support in AVRO CPP
> >
> > Hi Avro team,
> >
> > I am writing this email to check the support of Avro schema evolution in
> > CPP - i.e. provide both the producer and consumer schema when decoding
> the
> > data.
> >
> > I can see that there's a resolvingDecoder function in AVRO CPP that takes
> > two schemas. See
> >
> >
> https://avro.apache.org/docs/1.10.2/api/cpp/html/index.html#ReadingDifferentSchema
> >
> > But there's a FIXME comment in this function. See
> > https://issues.apache.org/jira/browse/AVRO-3720 and
> >
> https://github.com/apache/avro/blob/main/lang/c%2B%2B/api/Decoder.hh#L218.
> > Does this mean resolvingDecoder does not work properly? Could you please
> > explain what scenarios are not covered by resolvingDecoder and how can we
> > use it to support "Avro Schema Evolution" in c++?
> >
> > Thanks
> >
> >
> > Regards,
> > Vivek Kumar
> >
> > [http://www.eclipsetrading.com/logo.png]
> >
> > Senior Software Developer
> > 23/F One Hennessy
> > 1 Hennessy Road
> > Wan Chai
> > Hong Kong
> > www.eclipsetrading.com
> > +852 2108 7352
> >
> > Follow us today on our online platforms
> > [Facebook][Linked-In]<
> > https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> > https://www.instagram.com/eclipsetrading>
> >
>



-- 
Regards,

Andrew Marlow
http://www.andrewpetermarlow.co.uk

Re: Avro schema evolution support in AVRO CPP

2024-01-12 Thread Thiruvalluvan MG via user 
 Out-of-order fields are not handled transparently in C++ if you are manually 
using the resolving decoder. (It's the same situation in Java as well).
But, in C++ and in Java, if you generate code for the given Avro schema, the 
generated code takes care of the field ordering issue. Similarly, in both 
bindings, Generic data structures work properly with the field-order.
If you are using the resolving decoder in your code directly, care must be 
exercises, If the reader schema and writer schema are both records and they 
have fields in different order (it is okay to insert or remove fields), the 
protocol is to first get the field-order array (which is essentially a 
permutation of the reader field ids 0 to n-1) from the resolving decoder and 
then read the fields of the reader schema in the order specified in the 
field-order array. This is done in order to avoid buffering by the decoder. 
Buffering can take a large number of allocations if the out-of-order fields is 
an array or map.
In practice, it is very rare for schema evolution to change the order of the 
fields.
Thanks
ThiruOn Friday, 12 January, 2024 at 03:24:11 pm IST, Martin Grigorov 
 wrote:  
 
 Hi Vivek,

I am not sure there is anyone to give you an exact answer. The C++ SDK has
not been actively developed in the last few years.
The best is to try it for your use cases and see if it works or not. The
next step is to contribute Pull Requests for the missing functionalities!

Martin

On Thu, Jan 11, 2024 at 8:59 AM Vivek Kumar <
vivek.ku...@eclipsetrading.com.invalid> wrote:

> +dev
>
>
> Regards,
> Vivek Kumar
>
> [http://www.eclipsetrading.com/logo.png]
>
> Senior Software Developer
> 23/F One Hennessy
> 1 Hennessy Road
> Wan Chai
> Hong Kong
> www.eclipsetrading.com
> +852 2108 7352
>
> Follow us today on our online platforms
> [Facebook][Linked-In]<
> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> https://www.instagram.com/eclipsetrading>
> 
> From: Vivek Kumar 
> Sent: Thursday, January 11, 2024 11:07 AM
> To: user@avro.apache.org 
> Subject: Avro schema evolution support in AVRO CPP
>
> Hi Avro team,
>
> I am writing this email to check the support of Avro schema evolution in
> CPP - i.e. provide both the producer and consumer schema when decoding the
> data.
>
> I can see that there's a resolvingDecoder function in AVRO CPP that takes
> two schemas. See
>
> https://avro.apache.org/docs/1.10.2/api/cpp/html/index.html#ReadingDifferentSchema
>
> But there's a FIXME comment in this function. See
> https://issues.apache.org/jira/browse/AVRO-3720 and
> https://github.com/apache/avro/blob/main/lang/c%2B%2B/api/Decoder.hh#L218.
> Does this mean resolvingDecoder does not work properly? Could you please
> explain what scenarios are not covered by resolvingDecoder and how can we
> use it to support "Avro Schema Evolution" in c++?
>
> Thanks
>
>
> Regards,
> Vivek Kumar
>
> [http://www.eclipsetrading.com/logo.png]
>
> Senior Software Developer
> 23/F One Hennessy
> 1 Hennessy Road
> Wan Chai
> Hong Kong
> www.eclipsetrading.com
> +852 2108 7352
>
> Follow us today on our online platforms
> [Facebook][Linked-In]<
> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> https://www.instagram.com/eclipsetrading>
>

Re: Avro schema evolution support in AVRO CPP

2024-01-12 Thread Martin Grigorov 
Hi Vivek,

I am not sure there is anyone to give you an exact answer. The C++ SDK has
not been actively developed in the last few years.
The best is to try it for your use cases and see if it works or not. The
next step is to contribute Pull Requests for the missing functionalities!

Martin

On Thu, Jan 11, 2024 at 8:59 AM Vivek Kumar <
vivek.ku...@eclipsetrading.com.invalid> wrote:

> +dev
>
>
> Regards,
> Vivek Kumar
>
> [http://www.eclipsetrading.com/logo.png]
>
> Senior Software Developer
> 23/F One Hennessy
> 1 Hennessy Road
> Wan Chai
> Hong Kong
> www.eclipsetrading.com
> +852 2108 7352
>
> Follow us today on our online platforms
> [Facebook][Linked-In]<
> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> https://www.instagram.com/eclipsetrading>
> 
> From: Vivek Kumar 
> Sent: Thursday, January 11, 2024 11:07 AM
> To: user@avro.apache.org 
> Subject: Avro schema evolution support in AVRO CPP
>
> Hi Avro team,
>
> I am writing this email to check the support of Avro schema evolution in
> CPP - i.e. provide both the producer and consumer schema when decoding the
> data.
>
> I can see that there's a resolvingDecoder function in AVRO CPP that takes
> two schemas. See
>
> https://avro.apache.org/docs/1.10.2/api/cpp/html/index.html#ReadingDifferentSchema
>
> But there's a FIXME comment in this function. See
> https://issues.apache.org/jira/browse/AVRO-3720 and
> https://github.com/apache/avro/blob/main/lang/c%2B%2B/api/Decoder.hh#L218.
> Does this mean resolvingDecoder does not work properly? Could you please
> explain what scenarios are not covered by resolvingDecoder and how can we
> use it to support "Avro Schema Evolution" in c++?
>
> Thanks
>
>
> Regards,
> Vivek Kumar
>
> [http://www.eclipsetrading.com/logo.png]
>
> Senior Software Developer
> 23/F One Hennessy
> 1 Hennessy Road
> Wan Chai
> Hong Kong
> www.eclipsetrading.com
> +852 2108 7352
>
> Follow us today on our online platforms
> [Facebook][Linked-In]<
> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> https://www.instagram.com/eclipsetrading>
>