Re: Formal spec for Avro Schema

[Martin Grigorov]

Hi Elliot,

I am not sure which document you are referring to - the new proposal by
Clemens or the official specification.
Please start a new email thread or file a Jira ticket if you think
something needs to be improved in the specification!

On Wed, May 15, 2024 at 10:56 AM Elliot West  wrote:

> I note that the enum type appears to be missing the specification of the
> default attribute.
>
> On Wed, 15 May 2024 at 08:26, Martin Grigorov 
> wrote:
>
>> Hi Clemens,
>>
>> What is the difference between your document and the specification [1] ?
>> I haven't read it completely but it looks very similar to the
>> specification to me.
>>
>> 1. https://avro.apache.org/docs/1.11.1/specification/
>> 2.
>> https://github.com/apache/avro/tree/main/doc/content/en/docs/%2B%2Bversion%2B%2B/Specification
>> - sources of the specification
>>
>> On Wed, May 15, 2024 at 9:28 AM Clemens Vasters
>>  wrote:
>>
>>> I wrote a formal spec for the Avro Schema format.
>>>
>>>
>>>
>>> https://gist.github.com/clemensv/498c481965c425b218ee156b38b49333
>>>
>>>
>>>
>>> Where would that go in the repo?
>>>
>>>
>>>
>>>
>>>
>>>
>>> <http://www.microsoft.com/en-us/news/ImageDetail.aspx?id=4DABA54CBB4D25A9E9905BC59E4A6D44E33694EA>
>>>
>>> *Clemens Vasters*
>>>
>>> Messaging Platform Architect
>>>
>>> Microsoft Azure
>>>
>>> È+49 151 44063557
>>>
>>> *  cleme...@microsoft.com
>>> European Microsoft Innovation Center GmbH | Gewürzmühlstrasse 11 |
>>> 80539 Munich| Germany
>>> <https://www.google.com/maps/search/Gew%C3%BCrzm%C3%BChlstrasse+11%C2%A0%7C+80539+Munich%7C+Germany?entry=gmail=g>
>>> Geschäftsführer/General Managers: Keith Dolliver, Benjamin O. Orndorff
>>> Amtsgericht Aachen, HRB 12066
>>>
>>>
>>>
>>>
>>>
>>

Re: Formal spec for Avro Schema

[Martin Grigorov]

Hi Clemens,

On Wed, May 15, 2024 at 11:18 AM Clemens Vasters
 wrote:

> Hi Martin,
>
> we find Avro Schema to be a great fit for describing application data
> structures in general and even independent of wire-serialization scenarios.


> Therefore, I would like to have a spec that focuses specifically on the
> schema format, is grounded in the IETF RFC specs, and which follows the
> conventions set by IETF, so that folks who need a sane schema format to
> describe data structures independent of implementation can use that.
>

Do you say that the specification document is implementation dependent ?
If this is the case then maybe we should work on improving it instead of
duplicating it.


>
> The benefit for the Avro serialization framework of having such a formal
> spec that is untangled from the wire-serialization specs is that all
> schemas defined by that schema model are compatible with the framework.
>

What do you mean by "framework" here ?


>
> The differences are organization, scope, and language style (including
> keywords etc.). The expressed ruleset is the same.
>

I don't think it is a good idea to add a second document that is very
similar to the specification but uses a different language style.
To me this looks like a duplication.
IMO it would be better to suggest (many) (smaller) improvements for the
existing document.



>
> Best Regards
> Clemens
>
> -Original Message-
> From: Martin Grigorov 
> Sent: Wednesday, May 15, 2024 9:26 AM
> To: d...@avro.apache.org
> Cc: user@avro.apache.org
> Subject: Re: Formal spec for Avro Schema
>
> [Sie erhalten nicht häufig E-Mails von mgrigo...@apache.org. Weitere
> Informationen, warum dies wichtig ist, finden Sie unter
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi Clemens,
>
> What is the difference between your document and the specification [1] ?
> I haven't read it completely but it looks very similar to the
> specification to me.
>
> 1. https://avro.apache.org/docs/1.11.1/specification/
> 2.
>
> https://github.com/apache/avro/tree/main/doc/content/en/docs/%2B%2Bversion%2B%2B/Specification
> - sources of the specification
>
> On Wed, May 15, 2024 at 9:28 AM Clemens Vasters 
> 
> wrote:
>
> > I wrote a formal spec for the Avro Schema format.
> >
> >
> >
> > https://gist/
> > .github.com%2Fclemensv%2F498c481965c425b218ee156b38b49333=05%7C02
> > %7Cclemensv%40microsoft.com%7C5cd57d6ebe504e02e6dd08dc74b06a33%7C72f98
> > 8bf86f141af91ab2d7cd011db47%7C1%7C0%7C638513548275308005%7CUnknown%7CT
> > WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> > 6Mn0%3D%7C0%7C%7C%7C=n24LJspeNxYRKjlD0tgJzxQh3CzuILK%2FRe30gbarB
> > ec%3D=0
> >
> >
> >
> > Where would that go in the repo?
> >
> >
> >
> >
> >
> >
> > <http://www/.
> > microsoft.com%2Fen-us%2Fnews%2FImageDetail.aspx%3Fid%3D4DABA54CBB4D25A
> > 9E9905BC59E4A6D44E33694EA=05%7C02%7Cclemensv%40microsoft.com%7C5c
> > d57d6ebe504e02e6dd08dc74b06a33%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C638513548275312403%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> > CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=x6ZAZ
> > YEAjqkSVznt3N%2FKGjZzE%2BJietvHZuaiqVQYuDY%3D=0>
> >
> > *Clemens Vasters*
> >
> > Messaging Platform Architect
> >
> > Microsoft Azure
> >
> > È+49 151 44063557
> >
> > *  cleme...@microsoft.com
> > European Microsoft Innovation Center GmbH | Gewürzmühlstrasse 11 |
> > 80539
> > Munich| Germany
> > Geschäftsführer/General Managers: Keith Dolliver, Benjamin O. Orndorff
> > Amtsgericht Aachen, HRB 12066
> >
> >
> >
> >
> >
>

Re: Formal spec for Avro Schema

[Martin Grigorov]

Hi Clemens,

What is the difference between your document and the specification [1] ?
I haven't read it completely but it looks very similar to the specification
to me.

1. https://avro.apache.org/docs/1.11.1/specification/
2.
https://github.com/apache/avro/tree/main/doc/content/en/docs/%2B%2Bversion%2B%2B/Specification
- sources of the specification

On Wed, May 15, 2024 at 9:28 AM Clemens Vasters
 wrote:

> I wrote a formal spec for the Avro Schema format.
>
>
>
> https://gist.github.com/clemensv/498c481965c425b218ee156b38b49333
>
>
>
> Where would that go in the repo?
>
>
>
>
>
>
> 
>
> *Clemens Vasters*
>
> Messaging Platform Architect
>
> Microsoft Azure
>
> È+49 151 44063557
>
> *  cleme...@microsoft.com
> European Microsoft Innovation Center GmbH | Gewürzmühlstrasse 11 | 80539
> Munich| Germany
> Geschäftsführer/General Managers: Keith Dolliver, Benjamin O. Orndorff
> Amtsgericht Aachen, HRB 12066
>
>
>
>
>

Re: avro_schema_from_json return code 22

[Martin Grigorov]

Hi,

Error 22 in Linux is EINVAL - Invalid argument

Martin

On Fri, Feb 9, 2024 at 5:21 PM Alfredo Cardigliano 
wrote:

> Hi
> I am integrating Avro in an application, using avro_schema_from_json to
> parse the schema,
> while this function is working fine when compiling a C sample application,
> this fails returning
> error code 22 when compiling in a C++ application. I am using the libavro
> shipped with latest
> Ubuntu 22.
>
> Any hint about this return value (22)? Thank you.
>
> Alfredo

Re: Avro query on Testing if Beta feature of Generating faster code is enabled

[Martin Grigorov]

On Tue, Feb 6, 2024 at 3:03 PM Chirag Nahata 
wrote:

> hey Martin,
>
> Thanks, This worked for me and I can confirm that I have enabled the flag
> * "-Dorg.apache.avro.specific.use_custom_coders=true"* .
>

> Can you suggest to me how I can observe the performance improvements
> before/after?
>

You can run your favorite profiler with your application.




>
> sincerely
> Chirag Nahata
>
> On Tue, Feb 6, 2024 at 5:03 PM Martin Grigorov 
> wrote:
>
>>
>>
>> On Tue, Feb 6, 2024 at 1:15 PM Siddharth Baranidharan
>>  wrote:
>>
>>> Hey Oscar,
>>> Thank you for your response.
>>> My application is an enterprise server(java based)-agent(python based)
>>> model which uses avro-protocol for communication between server & agents.
>>> To the best of my knowledge I have added the flag “-
>>> *Dorg.apache.avro.specific.use_custom_coders=true*” as a java option to
>>> the
>>> server side.
>>>
>>
>> You can put a breakpoint at
>> https://github.com/apache/avro/blob/d143d6262f7d6c688e695d1656f8609605835604/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificData.java#L230
>> and see what is the value of "useCustomCoderFlag"
>>
>>
>>
>>>
>>> My question:
>>> How can I test whether this feature is successfully enabled & to answer
>>> your 2nd question, yes I want to test its effects & I am willing to trust
>>> the tests that you have for the flag. Can you detail the steps I need to
>>> follow for the same?
>>>
>>> Thank you,
>>> Siddharth B
>>>
>>> On Tue, Feb 6, 2024 at 1:18 PM Oscar Westra van Holthe - Kind <
>>> os...@westravanholthe.nl> wrote:
>>>
>>> > Hi,
>>> >
>>> > What do you mean by testing if the flag is successfully turned on? Do
>>> you
>>> > need to test its effects? Are you willing to trust the tests we have
>>> on the
>>> > flag?
>>> >
>>> > As for testing the performance difference, we do have a performance
>>> test
>>> > module. Perhaps you can use the same technique?
>>> >
>>> > Kind regards,
>>> > Oscar
>>> >
>>> > --
>>> > Oscar Westra van Holthe - Kind 
>>> >
>>> > Op di 6 feb. 2024 07:15 schreef chirag :
>>> >
>>> >> Hi Team,
>>> >>
>>> >> On the Avro Docs it is mentioned that:- to turn new approach to
>>> generating
>>> >> code that speeds up decoding and encoding set feature flag/system flag
>>> >> org.apache.avro.specific.use_custom_coders to true at runtime.(here
>>> >> <
>>> >>
>>> https://avro.apache.org/docs/1.11.1/getting-started-java/#beta-feature-generating-faster-code
>>> >> >
>>> >> ).
>>> >>
>>> >> Enquiring if:
>>> >>
>>> >>1. There is a way to see if this flag is successfully turned on
>>> during
>>> >>runtime?
>>> >>2. There is a way to measure the performance improvement in doing
>>> so?
>>> >>
>>> >> I have added this system flag to my distributed enterprise
>>> application but
>>> >> I am not sure if it is enabled and if there is a performance
>>> improvement
>>> >> on
>>> >> doing so.
>>> >>
>>> >> Sincerely
>>> >> Chirag Nahata
>>> >>
>>> >
>>>
>>

Re: Avro query on Testing if Beta feature of Generating faster code is enabled

[Martin Grigorov]

On Tue, Feb 6, 2024 at 1:15 PM Siddharth Baranidharan
 wrote:

> Hey Oscar,
> Thank you for your response.
> My application is an enterprise server(java based)-agent(python based)
> model which uses avro-protocol for communication between server & agents.
> To the best of my knowledge I have added the flag “-
> *Dorg.apache.avro.specific.use_custom_coders=true*” as a java option to the
> server side.
>

You can put a breakpoint at
https://github.com/apache/avro/blob/d143d6262f7d6c688e695d1656f8609605835604/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificData.java#L230
and see what is the value of "useCustomCoderFlag"



>
> My question:
> How can I test whether this feature is successfully enabled & to answer
> your 2nd question, yes I want to test its effects & I am willing to trust
> the tests that you have for the flag. Can you detail the steps I need to
> follow for the same?
>
> Thank you,
> Siddharth B
>
> On Tue, Feb 6, 2024 at 1:18 PM Oscar Westra van Holthe - Kind <
> os...@westravanholthe.nl> wrote:
>
> > Hi,
> >
> > What do you mean by testing if the flag is successfully turned on? Do you
> > need to test its effects? Are you willing to trust the tests we have on
> the
> > flag?
> >
> > As for testing the performance difference, we do have a performance test
> > module. Perhaps you can use the same technique?
> >
> > Kind regards,
> > Oscar
> >
> > --
> > Oscar Westra van Holthe - Kind 
> >
> > Op di 6 feb. 2024 07:15 schreef chirag :
> >
> >> Hi Team,
> >>
> >> On the Avro Docs it is mentioned that:- to turn new approach to
> generating
> >> code that speeds up decoding and encoding set feature flag/system flag
> >> org.apache.avro.specific.use_custom_coders to true at runtime.(here
> >> <
> >>
> https://avro.apache.org/docs/1.11.1/getting-started-java/#beta-feature-generating-faster-code
> >> >
> >> ).
> >>
> >> Enquiring if:
> >>
> >>1. There is a way to see if this flag is successfully turned on
> during
> >>runtime?
> >>2. There is a way to measure the performance improvement in doing so?
> >>
> >> I have added this system flag to my distributed enterprise application
> but
> >> I am not sure if it is enabled and if there is a performance improvement
> >> on
> >> doing so.
> >>
> >> Sincerely
> >> Chirag Nahata
> >>
> >
>

Re: [Java] Avro does not respect logicalType with map type

[Martin Grigorov]

Hi,



On Fri, Feb 2, 2024 at 11:37 AM Dupa Trianko  wrote:

> I have lib class Dimension3D which contains 3 fields and I want to convert
> it as one avro filed.
>
> My definition of logical type connected classes:
> public class Dim3AvroLogicalType {
>
>   public static final String LOGICAL_TYPE_NAME = "dim3";
>
>   public static class Dim3LogicalType extends LogicalType {
>
> Dim3LogicalType() {
>   super(LOGICAL_TYPE_NAME);
> }
>
> @Override
> public void validate(Schema schema) {
>   super.validate(schema);
>   if (schema.getType() != Schema.Type.MAP) {
> throw new IllegalArgumentException(
> String.format("Logical type '%s' must be backed by Map",
> LOGICAL_TYPE_NAME));
>   }
> }
>   }
>
>   public static class Dim3Factory implements
> LogicalTypes.LogicalTypeFactory {
>
> @Override
> public LogicalType fromSchema(Schema schema) {
>   return new Dim3LogicalType();
> }
>
> @Override
> public String getTypeName() {
>   return LOGICAL_TYPE_NAME;
> }
>   }
>
>   public static class Dim3Conversion extends Conversion {
>
> @Override
> public Class getConvertedType() {
>   return Dimension3D.class;
> }
>
> @Override
> public String getLogicalTypeName() {
>   return LOGICAL_TYPE_NAME;
> }
>
> @Override
> public Dimension3D fromMap(Map value, Schema schema, LogicalType
> type) {
>   return Dimension3D.of(
>   Length.of(new BigDecimal(value.get("length").toString())),
>   Length.of(new BigDecimal(value.get("width").toString())),
>   Length.of(new BigDecimal(value.get("height").toString()))
>   );
> }
>
> @Override
> public Map toMap(Dimension3D dimension3D, Schema schema,
> LogicalType type) {
>   return Map.of(
>   "length", dimension3D.getLength().toDecimal().toString(),
>   "width", dimension3D.getWidth().toDecimal().toString(),
>   "height", dimension3D.getHeight().toDecimal().toString()
>   );
> }
>   }
> }
>
> interesting part of avro definition:
>   {
> "name": "dimension",
> "type": {
>   "type": "map",
>   "logicalType": "dim3",
>   "values": "string"
> }
>   }
>
> after executing maven task I got:
>
> public java.util.Map getDimension().
> Moreover there is no proper converter added to MODEL$ while other
> converters are added correctly.
>
> I'm using:
>   
> org.apache.avro
> avro-maven-plugin
> 1.11.3
>

Did you register your custom factory and conversion ?
See
https://github.com/apache/avro/blob/d143d6262f7d6c688e695d1656f8609605835604/lang/java/integration-test/codegen-test/pom.xml#L60-L69



>
> proper customLogicalTypeFactory & customConversion are added into
> configuration.
>
> How to properly use logical type with map backing type?
>

Re: Avro schema evolution support in AVRO CPP

[Martin Grigorov]

Hi Vivek,

I am not sure there is anyone to give you an exact answer. The C++ SDK has
not been actively developed in the last few years.
The best is to try it for your use cases and see if it works or not. The
next step is to contribute Pull Requests for the missing functionalities!

Martin

On Thu, Jan 11, 2024 at 8:59 AM Vivek Kumar <
vivek.ku...@eclipsetrading.com.invalid> wrote:

> +dev
>
>
> Regards,
> Vivek Kumar
>
> [http://www.eclipsetrading.com/logo.png]
>
> Senior Software Developer
> 23/F One Hennessy
> 1 Hennessy Road
> Wan Chai
> Hong Kong
> www.eclipsetrading.com
> +852 2108 7352
>
> Follow us today on our online platforms
> [Facebook][Linked-In]<
> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> https://www.instagram.com/eclipsetrading>
> 
> From: Vivek Kumar 
> Sent: Thursday, January 11, 2024 11:07 AM
> To: user@avro.apache.org 
> Subject: Avro schema evolution support in AVRO CPP
>
> Hi Avro team,
>
> I am writing this email to check the support of Avro schema evolution in
> CPP - i.e. provide both the producer and consumer schema when decoding the
> data.
>
> I can see that there's a resolvingDecoder function in AVRO CPP that takes
> two schemas. See
>
> https://avro.apache.org/docs/1.10.2/api/cpp/html/index.html#ReadingDifferentSchema
>
> But there's a FIXME comment in this function. See
> https://issues.apache.org/jira/browse/AVRO-3720 and
> https://github.com/apache/avro/blob/main/lang/c%2B%2B/api/Decoder.hh#L218.
> Does this mean resolvingDecoder does not work properly? Could you please
> explain what scenarios are not covered by resolvingDecoder and how can we
> use it to support "Avro Schema Evolution" in c++?
>
> Thanks
>
>
> Regards,
> Vivek Kumar
>
> [http://www.eclipsetrading.com/logo.png]
>
> Senior Software Developer
> 23/F One Hennessy
> 1 Hennessy Road
> Wan Chai
> Hong Kong
> www.eclipsetrading.com
> +852 2108 7352
>
> Follow us today on our online platforms
> [Facebook][Linked-In]<
> https://www.linkedin.com/company/eclipse-trading>[Instagram]<
> https://www.instagram.com/eclipsetrading>
>

Re: Subscribe to Avro users mailing list

[Martin Grigorov]

Hi,

I didn't notice an email to be moderated, so I guess you are already
subscribed.
Feel free to start new topics/threads and ask any questions!

On Fri, Mar 10, 2023 at 11:31 AM Abhishek Dasgupta <
abhishekdasgupta...@gmail.com> wrote:

> Hi Team,
> I would like to ask questions.
>

Re: Ints & Bools Issue

[Martin Grigorov]

Hello David,

Please file a ticket at JIRA for this problem. And attach or link to a a
mini application demonstrating the problem.
I assume the bug is in the C++ code, since the Java SDK is more widely used.

On Mon, Jan 23, 2023 at 5:28 AM David Funnell 
wrote:

> Hello,
>
> My coworker and I are having an issue with what seems like the decoder not
> respecting byte lengths of ints and bools.
>
> I am working in c++ and we use kafka to consume and produce.  I am the
> producer and am encoding to binary and sending a data set to a server.
>
> My coworker is working in java, and is consuming and decoding, and we are
> using the same schema.  We have had an issue where any values sent as type
> int or boolean, appear to cause any following fields to start reading from
> the wrong section in memory when decoded.
>
> We have a simple set up and have followed documentation closely, the only
> error we get is an occasional exception that is very vague, but seems to be
> something to do with the int binary decoder.  We are unsure what the issue
> is, or if there's a configuration setting with avro we have missed.  We
> have tried statically casting values before the encoding process just in
> case, but this hasn't made a difference
>
> We have overcome this by using floats for everything int or bool however
> some help in not having to do this would be appreciated.
>
> Thanks
>
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system manager.
> This message contains confidential information and is intended only for the
> individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. If you are not the intended recipient
> you are notified that disclosing, copying, distributing or taking any
> action in reliance on the contents of this information is strictly
> prohibited.

Re: Rust Serde Deserialize

[Martin Grigorov]

https://github.com/lerouxrgd/rsgen-avro/pull/36

On Tue, Aug 30, 2022 at 9:07 AM Martin Grigorov 
wrote:

> Hi Rajiv,
>
> Now, when you said it I remembered about this prerequisite for serializing
> bytes.
> I will open a PR to rsgen-avro to add #[serde(with = "serde_bytes")] when
> "bytes" schema is used!
>
> Regards,
> Martin
>
> On Mon, Aug 29, 2022 at 5:10 PM Rajiv M Ranganath <
> rajiv.rangan...@gmail.com> wrote:
>
>> Hi Martin,
>>
>> It looks like there are no changes needed on the Avro side. I was not
>> using Serde correctly.
>>
>> On the Rust side, we need to use `serde_bytes` [1] crate, and define our
>> Rust struct as follows.
>>
>> ```
>> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
>> serde::Serialize)]
>> #[serde(default)]
>> pub struct Abcd {
>> #[serde(with = "serde_bytes")]
>> pub b: Option>,
>> }
>> ```
>>
>> That seems to make things work.
>>
>> Thanks again for the reply.
>>
>> Best,
>> Rajiv
>>
>> [1] https://docs.rs/serde_bytes/latest/serde_bytes/
>>
>> On Mon, Aug 29, 2022 at 5:18 PM Rajiv M Ranganath
>>  wrote:
>> >
>> > On Mon, Aug 29, 2022 at 3:18 PM Martin Grigorov 
>> wrote:
>> >
>> > [...]
>> >
>> > > Do you want to contribute the code as a failing unit test in a Pull
>> > > Request ?
>> > > With a fix would be awesome!
>> >
>> > Okay. Please give me a couple of days. I'll investigate and open a PR.
>> >
>> > Best,
>> > Rajiv
>>
>

Re: Rust Serde Deserialize

[Martin Grigorov]

Hi Rajiv,

Now, when you said it I remembered about this prerequisite for serializing
bytes.
I will open a PR to rsgen-avro to add #[serde(with = "serde_bytes")] when
"bytes" schema is used!

Regards,
Martin

On Mon, Aug 29, 2022 at 5:10 PM Rajiv M Ranganath 
wrote:

> Hi Martin,
>
> It looks like there are no changes needed on the Avro side. I was not
> using Serde correctly.
>
> On the Rust side, we need to use `serde_bytes` [1] crate, and define our
> Rust struct as follows.
>
> ```
> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
> serde::Serialize)]
> #[serde(default)]
> pub struct Abcd {
> #[serde(with = "serde_bytes")]
> pub b: Option>,
> }
> ```
>
> That seems to make things work.
>
> Thanks again for the reply.
>
> Best,
> Rajiv
>
> [1] https://docs.rs/serde_bytes/latest/serde_bytes/
>
> On Mon, Aug 29, 2022 at 5:18 PM Rajiv M Ranganath
>  wrote:
> >
> > On Mon, Aug 29, 2022 at 3:18 PM Martin Grigorov 
> wrote:
> >
> > [...]
> >
> > > Do you want to contribute the code as a failing unit test in a Pull
> > > Request ?
> > > With a fix would be awesome!
> >
> > Okay. Please give me a couple of days. I'll investigate and open a PR.
> >
> > Best,
> > Rajiv
>

Re: Rust Serde Deserialize

[Martin Grigorov]

Hi Rajiv,

There are 0 known issues for the Rust SDK! :-)

Do you want to contribute the code as a failing unit test in a Pull Request
?
With a fix would be awesome!

Thanks!

Martin

On Mon, Aug 29, 2022 at 12:11 PM Rajiv M Ranganath <
rajiv.rangan...@gmail.com> wrote:

> Hi Martin,
>
> On Mon, Aug 29, 2022 at 1:43 PM Martin Grigorov 
> wrote:
>
> [...]
>
> > I'd recommend you this nice tool for generating Rust structs from Avro
> > schema: https://github.com/lerouxrgd/rsgen-avro
>
> Thanks for the reply and the pointer to `rsgen-avro`. :-)
>
> It looks like there seems to be an issue in the way Serde is serializing
> the schema.
>
> Here is the example:
>
> ```main.rs
> use apache_avro::types::{Record, Value};
> use apache_avro::Schema;
>
> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
> serde::Serialize)]
> #[serde(default)]
> pub struct Abcd {
> pub b: Option>,
> }
>
> #[inline(always)]
> fn default_abcd_b() -> Option> {
> None
> }
>
> impl Default for Abcd {
> fn default() -> Abcd {
> Abcd {
> b: default_abcd_b(),
> }
> }
> }
>
> fn main() {
> let writers_schema = Schema::parse_str(
> r#"
> {
>   "type": "record",
>   "name": "Abcd",
>   "fields": [
> {"name": "b", "type": ["null", "bytes"], "default": null}
>   ]
> }
> "#,
> )
> .unwrap();
>
> let mut abcd_manual = Record::new(_schema).unwrap();
> abcd_manual.put(
> "b",
> Value::Union(1,
> Box::new(Value::Bytes("hello_world".as_bytes().to_vec(,
> );
>
> println!("{:?}", abcd_manual);
>
> let abcd_manual_bytes =
> apache_avro::to_avro_datum(_schema, abcd_manual).unwrap();
> println!("{:?}", abcd_manual_bytes);
>
> let abcd_serde_value = apache_avro::to_value(Abcd {
> b: Some("hello_world".as_bytes().to_vec()),
> }).unwrap();
>
> println!("{:?}", abcd_serde_value);
>
> let abcd_serde_bytes = apache_avro::to_avro_datum(_schema,
> abcd_serde_value);
>
> println!("{:?}", abcd_serde_bytes);
> }
> ```
>
> Rather than creating an Avro Value of the form,
>
> ```
> Record { fields: [("b", Union(1, Bytes([104, 101, 108, 108, 111, 95,
> 119, 111, 114, 108, 100])))], schema_lookup: {"b": 0} }
> ```
>
> Serde seems to be generating an Avro Value,
>
> ```
> Record([("b", Union(1, Array([Int(104), Int(101), Int(108), Int(108),
> Int(111), Int(95), Int(119), Int(111), Int(114), Int(108),
> Int(100)])))])
> ```
>
> which is causing the subsequent conversion to bytes to fail.
>
> I was wondering if this is a known issue?
>
> Best,
> Rajiv
>

Re: Rust Serde Deserialize

[Martin Grigorov]

Hi Rajiv,

I'd recommend you this nice tool for generating Rust structs from Avro
schema: https://github.com/lerouxrgd/rsgen-avro

$ cat q.avsc
   1   │ {
   2   │   "type": "record",
   3   │   "name": "Abcd",
   4   │   "fields": [
   5   │ {"name": "b", "type": ["null", "bytes"], "default": null}
   6   │   ]
   7   │ }

generates:

cat q.rs
   1   │
   2   │ #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize,
serde::Serialize)]
   3   │ #[serde(default)]
   4   │ pub struct Abcd {
   5   │ pub b: Option>,
   6   │ }
   7   │
   8   │ #[inline(always)]
   9   │ fn default_abcd_b() -> Option> { None }
  10   │
  11   │ impl Default for Abcd {
  12   │ fn default() -> Abcd {
  13   │ Abcd {
  14   │ b: default_abcd_b(),
  15   │ }
  16   │ }
  17   │ }


On Mon, Aug 29, 2022 at 10:05 AM Rajiv M Ranganath <
rajiv.rangan...@gmail.com> wrote:

> Hi,
>
> I am new to Avro. When I have an Arvo schema of the form,
>
> ```
> record Abcd {
> union { null, bytes } efgh = null;
> }
> ```
>
> What would be the corresponding Rust struct?
>
> I tried
>
> ```
> #[derive(Debug, Deserialize, PartialEq)]
> struct Abcd {
> efgh: Option>,
> }
> ```
>
> But for some reason, `apache_avro::from_value` is giving an
> `Err(DeserializeValue("not an array"))` error.
>
> Best,
> Rajiv
>

Re: Unable to resolve NettyTransceiver/ NettyServer (Netty based classes) in avro-ipc 1.11.0

[Martin Grigorov]

On Wed, May 18, 2022 at 4:28 PM Abhishek Dasgupta <
abhishekdasgupta...@gmail.com> wrote:

> Hi Martin,
>  Thanks for the prompt reply but I am still unable to resolve the classes
> above. Any other suggestions?
>

https://github.com/apache/avro/blob/master/lang/java/ipc-netty/src/main/java/org/apache/avro/ipc/netty/NettyTransceiver.java
https://github.com/apache/avro/blob/master/lang/java/ipc-netty/src/main/java/org/apache/avro/ipc/netty/NettyServer.java

Both classes are in avro-ipc-netty. Maybe you didn't add the dependency
properly in your project ?!



>
> On Wed, May 18, 2022 at 6:11 PM Martin Grigorov 
> wrote:
>
>> Hi,
>>
>> You need to use
>> https://search.maven.org/artifact/org.apache.avro/avro-ipc-netty/1.11.0/bundle
>> instead.
>>
>>
>> On Wed, May 18, 2022 at 2:41 PM Abhishek Dasgupta <
>> abhishekdasgupta...@gmail.com> wrote:
>>
>>>
>>>
>>>
>>> Hi,
>>>   I am solving this CVE-2021-43045 in my project. I am upgrading Avro
>>> and Avro-ipc from 1.7.6-cdh5.12.0 to 1.11.0. There are some
>>> NettyTransceiver, NettyServer calls in the project. After upgrade, such
>>> imports become un-resolvable which results in compilation error. From Avro
>>> 1.9.0 onwards these classes moved to Avro.ipc.*netty* package. I add
>>> this is my code but ’netty’ package is not resolved. In External
>>> dependencies section avro-ipc (1.11.0) does not contain netty folder.
>>> Kindly suggest what to do here as it is critical fix in my project. Here
>>> are my dependencies:
>>>
>>>  org.apache.avro
>>> avro 1.11.0 
>>>
>>>  org.apache.avro
>>> avro-ipc 1.11.0 
>>>
>>>
>>> Thanks,
>>> Abhishek
>>>
>>>
>>>

Re: Unable to resolve NettyTransceiver/ NettyServer (Netty based classes) in avro-ipc 1.11.0

[Martin Grigorov]

Hi,

You need to use
https://search.maven.org/artifact/org.apache.avro/avro-ipc-netty/1.11.0/bundle
instead.


On Wed, May 18, 2022 at 2:41 PM Abhishek Dasgupta <
abhishekdasgupta...@gmail.com> wrote:

>
>
>
> Hi,
>   I am solving this CVE-2021-43045 in my project. I am upgrading Avro and
> Avro-ipc from 1.7.6-cdh5.12.0 to 1.11.0. There are some NettyTransceiver,
> NettyServer calls in the project. After upgrade, such imports become
> un-resolvable which results in compilation error. From Avro 1.9.0 onwards
> these classes moved to Avro.ipc.*netty* package. I add this is my code
> but ’netty’ package is not resolved. In External dependencies section
> avro-ipc (1.11.0) does not contain netty folder. Kindly suggest what to do
> here as it is critical fix in my project. Here are my dependencies:
>
>  org.apache.avro
> avro 1.11.0 
>
>  org.apache.avro
> avro-ipc 1.11.0 
>
>
> Thanks,
> Abhishek
>
>
>

Re: Gigantic list of Avro spec issues

[Martin Grigorov]

On Fri, Feb 18, 2022 at 5:00 AM Askar Safin  wrote:

> > By the way, we have what we believe to be the most correct and complete
> Rust
> > implementation of Avro here:
> > https://github.com/MaterializeInc/materialize/tree/main/src/avro
> Hi. I spent a lot of time searching for perfect Avro implementation for
> Rust. And indeed Materialize implementation turned to be the best as I
> think.
> Still, I rejected it because "avro_derive" doesn't work for Rust enums
> (with data). I. e. I wanted Rust enums to be converted to Avro
> unions. I rejected other Rust implementations for various reasons, too. So
> I wrote my own. (I can show code, but note that my implementation is
> incomplete.) Now (because of various Avro problems) I think I will throw
> away this code and simply will create my own binary format instead.
>
> I indeed suggest Avro project to base its Rust implementation on
> Materialize's code, not on "avro-rs" from https://crates.io/crates/avro-rs
>  .
> "avro-rs" doesn't even support recursive types. And adding recursive types
> support will require major rewrite. There is pull request on
> their github for recursive types, but it's still unmerged
>

I hate being the party spoiler but:
- https://github.com/flavray/avro-rs/pull/99#issuecomment-1017195592
- https://github.com/apache/avro/pull/1456/files

TL;DR - apache-avro (this is the new crate name) does support recursive
types!



>
>
> ==
> Askar Safin
> http://safinaskar.com
> https://sr.ht/~safinaskar
> https://github.com/safinaskar
>

Re: UUID Logical type not working in Java

[Martin Grigorov]

Hi,

Which version of Avro do you use ?
Which language ?
If Java, then please try 1.12.0-SNAPSHOT from
https://repository.apache.org/content/groups/snapshots/
If it still does not work then please show us your schema or even better
create a reproducer and share it with us, e.g. at Github.

On Fri, Jan 14, 2022 at 10:22 PM Swamy Thota  wrote:

> Hello All,
>
> I'm trying to use uuid logical type in one of my AVRO schema with string
> as avro type. The generated class has the field type as string but I'm
> expecting it to be UUID, is this a known issue? It works for other logical
> types local-timestamp-millis as an example. Appreciate your help in advance.
>
> Thanks,
> Swamy Thota
>

Re: Papers discussing Apache Avro

[Martin Grigorov]

Hi Juan,

Thank you for sharing your work with us!

It comes right in time for me!
I am working on the interop tests for the new Rust module and it seems
there is some problem to read the .avro files generated by Java. I may need
to dive in the binary diffs.

Regards,
Martin

On Thu, Jan 13, 2022 at 11:14 PM Juan Cruz Viotti  wrote:

> Hey there!
>
> As part of my MSc dissertation at University of Oxford, I wrote and
> published two papers covering the characteristics of various binary
> serialization formats, including Apache Avro and performing a
> space-efficiency benchmark, respectively.
>
> Sharing them here in case anybody finds them interesting! The first
> paper explains how Apache Avro works including an annotated hexadecimal
> example and the second compares Apache Avro to various other popular
> serialization formats.
>
> - A Survey of JSON-compatible Binary Serialization Specifications:
>   https://arxiv.org/abs/2201.02089
> - A Benchmark of JSON-compatible Binary Serialization Specifications:
>   https://arxiv.org/abs/2201.03051
>
> The benchmark study has proved Apache Avro to be one of the most
> space-efficient formats considered.
>
> All the best!
>
> --
> Juan Cruz Viotti
> Technical Lead @ Postman.com
> https://www.jviotti.com
>

Re: Serialization with optional fields using C++ library

[Martin Grigorov]

Hi Anton,

I don't see any unit tests / examples for this at
https://github.com/apache/avro/tree/master/lang/c%2B%2B/ test|examples, so
I guess it is not implemented yet.
You could add an entry for basicSchemas (
https://github.com/apache/avro/blob/1aa963c44d1b9da3dfcf74acb3eeed56439332a0/lang/c%2B%2B/test/SchemaTests.cc#L30)
and if it fails then create an issue / PR.

On Wed, Dec 22, 2021 at 10:13 AM Anton  wrote:

> Hello, I have problem with serialization of data having optional fields.
> When I pass null in corresponding field it works but when it is non-null
> then serialization fails.
>
> Schema:
>
> {
>
> "type": "record",
>
> "name": "schema",
>
> "namespace": "space",
>
> "fields": [
>
>{
>
>"name": "username",
>
>"type": "string"
>
>},
>
>{
>
>"name": "data",
>
>"type": [
>
>"null",
>
>"string"
>
>],
>
>"defaults": null
>
>},
>
>{
>
>"name": "timestamp",
>
>"type": "long"
>
>}
>
> ]
>
> }
>
>
>
> Data that works:
>
> {"username":"miguno","data":null,"timestamp": 1366150681 }
>
>
>
> Data that fails:
>
> {"username":"miguno","data":"test","timestamp": 1366150681 }
>
>
>
> Should it work or I have some error in my schema? I didn’t find any active
> issues in jira so I guess the concept of optional fields should work just
> fine, also in C++.
>
>
>
> The code is:
>
>
>
> std::unique_ptr in = avro::memoryInputStream((const
> uint8_t*)[0], json.size()); // json is incoming data
>
> avro::ValidSchema validSchema;
>
> std::istringstream ins(schema); // schema is avro-schema
>
> try {
>
> avro::compileJsonSchema(ins, validSchema);
>
> }
>
> catch (const std::exception& e1) {
>
> std::string errstr = e1.what();
>
> }
>
> avro::DecoderPtr jd = avro::jsonDecoder(validSchema);
>
> avro::GenericDatum datum(validSchema);
>
> jd->init(*in);
>
> avro::decode(*jd, datum); //serialization with non-null data fails
> somewhere inside this step
>

Re: New website

[Martin Grigorov]

On Mon, Dec 13, 2021 at 9:38 AM Martin Grigorov 
wrote:

> Hi Ryan,
>
> On Sun, Dec 12, 2021 at 1:36 PM Ryan Skraba  wrote:
>
>> Hello!
>>
>> I realized that I haven't commented on this mailing list thread -- I
>> made some comments on https://issues.apache.org/jira/browse/AVRO-2175
>>
>> This looks amazing and we should merge it very soon :D  It's not
>> perfect, but it's really a great improvement and definitely not worst
>> than the existing website!
>>
>> I've been taking a look at what we need to use the existing
>> infrastructure, and there's interesting links at:
>>
>> - https://infra.apache.org/release-download-pages.html
>> -
>> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features
>> - https://infra.apache.org/website-guidelines.html
>> - https://infra.apache.org/project-site.html
>>
>> I like that Beam has the website in the main repo, but notably the
>> INFRA recommendation is that we use a separate repo for the website
>> (named `avro-site` though).  Any thoughts?  It's always something we
>> can try and change later!  Would it make it easier for the javadoc and
>> other languages if they were in the same repo, or does it make little
>> difference?
>>
>
> Since some docs are generated by other tools (C/C++/C#) and they need to
> be copied to the rest of the website I agree that it would be easier if the
> website is in the main repo!
> Also any changes to the spec/IDL will be in one PR (same repo) instead of
> two (main+website).
>
>
>>
>> The old site actually contains all of the documentation for EVERY
>> release, which can be found here:
>>
>> - https://svn.apache.org/repos/asf/avro/site/publish/docs/
>> - https://avro.apache.org/docs/
>>
>> Would it be tricky to adjust your work to mirror the existing
>> structure for existing docs?  I'm not even too fussy about not
>> breaking the links in /docs/current/ but all of the existing pages
>> such as /docs/1.7.7/ should be maintained if possible!
>>
>
> I could make the "Documention" nav tab into a dropdown with menu items,
> one for each version.
> 1.11.0 and 1.10.2, and any future release, will be proper Markdown
> documents. The menu items for all previous versions will be links to the
> respective folder at https://avro.apache.org/docs/.
>

Done!
https://avro-website.netlify.app/


> If someone wants to migrate some more old docs to Markdown then please
> fork the repo and send a PR!
>
>
>>
>> There's so many good suggestions here for future work and improving
>> our message and communication, I created
>> https://issues.apache.org/jira/browse/AVRO-3264 to point to this
>> discussion after we get this up.
>>
>> Thanks again for your great work!
>>
>> Ryan
>>
>> On Thu, Nov 4, 2021 at 4:26 PM Lee Hambley  wrote:
>> >
>> > I speak only for myself, but I am working in an environment where I am
>> regularly checking docs all the way back to 1.8.x because we have legacy
>> systems we cannot upgrade, and I am often referencing rules about schema
>> canonical form. I value a lot the sidebar bottom version switching
>> navigation from sites such as here
>> https://fastavro.readthedocs.io/en/latest/writer.html#using-the-record-hint-to-specify-which-branch-of-a-union-to-take
>> ... but I know it can be extraordinarily difficult to make it work
>> correctly with these static site generators.
>> >
>> > Lee Hambley
>> > http://lee.hambley.name/
>> > +49 (0) 170 298 5667
>> >
>> >
>> > On Thu, 4 Nov 2021 at 16:23, Martin Grigorov 
>> wrote:
>> >>
>> >>
>> >>
>> >> On Thu, Nov 4, 2021 at 5:04 PM Ismaël Mejía  wrote:
>> >>>
>> >>> Wow this is pretty neat ! Nice job Martin! A modern website can
>> >>> encourage more contributions.
>> >>> I am more interested on content than aesthetics first. Is everything
>> >>> already migrated? Anything missing? Any issue to report?
>> >>
>> >>
>> >> Everything is migrated for the documentation of the *current* version.
>> >> The old site contains documentation for both current and current-1. Is
>> this something you would like to preserve ?
>> >>
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Nov 2, 2021 at 7:01 PM Martin Grigorov 
>> wrote:
>> >>> >
>> >>> > Hi,
>> >>> >
>> >>> &g

Re: New website

[Martin Grigorov]

Hi Ryan,

On Sun, Dec 12, 2021 at 1:36 PM Ryan Skraba  wrote:

> Hello!
>
> I realized that I haven't commented on this mailing list thread -- I
> made some comments on https://issues.apache.org/jira/browse/AVRO-2175
>
> This looks amazing and we should merge it very soon :D  It's not
> perfect, but it's really a great improvement and definitely not worst
> than the existing website!
>
> I've been taking a look at what we need to use the existing
> infrastructure, and there's interesting links at:
>
> - https://infra.apache.org/release-download-pages.html
> -
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features
> - https://infra.apache.org/website-guidelines.html
> - https://infra.apache.org/project-site.html
>
> I like that Beam has the website in the main repo, but notably the
> INFRA recommendation is that we use a separate repo for the website
> (named `avro-site` though).  Any thoughts?  It's always something we
> can try and change later!  Would it make it easier for the javadoc and
> other languages if they were in the same repo, or does it make little
> difference?
>

Since some docs are generated by other tools (C/C++/C#) and they need to be
copied to the rest of the website I agree that it would be easier if the
website is in the main repo!
Also any changes to the spec/IDL will be in one PR (same repo) instead of
two (main+website).


>
> The old site actually contains all of the documentation for EVERY
> release, which can be found here:
>
> - https://svn.apache.org/repos/asf/avro/site/publish/docs/
> - https://avro.apache.org/docs/
>
> Would it be tricky to adjust your work to mirror the existing
> structure for existing docs?  I'm not even too fussy about not
> breaking the links in /docs/current/ but all of the existing pages
> such as /docs/1.7.7/ should be maintained if possible!
>

I could make the "Documention" nav tab into a dropdown with menu items, one
for each version.
1.11.0 and 1.10.2, and any future release, will be proper Markdown
documents. The menu items for all previous versions will be links to the
respective folder at https://avro.apache.org/docs/.
If someone wants to migrate some more old docs to Markdown then please fork
the repo and send a PR!


>
> There's so many good suggestions here for future work and improving
> our message and communication, I created
> https://issues.apache.org/jira/browse/AVRO-3264 to point to this
> discussion after we get this up.
>
> Thanks again for your great work!
>
> Ryan
>
> On Thu, Nov 4, 2021 at 4:26 PM Lee Hambley  wrote:
> >
> > I speak only for myself, but I am working in an environment where I am
> regularly checking docs all the way back to 1.8.x because we have legacy
> systems we cannot upgrade, and I am often referencing rules about schema
> canonical form. I value a lot the sidebar bottom version switching
> navigation from sites such as here
> https://fastavro.readthedocs.io/en/latest/writer.html#using-the-record-hint-to-specify-which-branch-of-a-union-to-take
> ... but I know it can be extraordinarily difficult to make it work
> correctly with these static site generators.
> >
> > Lee Hambley
> > http://lee.hambley.name/
> > +49 (0) 170 298 5667
> >
> >
> > On Thu, 4 Nov 2021 at 16:23, Martin Grigorov 
> wrote:
> >>
> >>
> >>
> >> On Thu, Nov 4, 2021 at 5:04 PM Ismaël Mejía  wrote:
> >>>
> >>> Wow this is pretty neat ! Nice job Martin! A modern website can
> >>> encourage more contributions.
> >>> I am more interested on content than aesthetics first. Is everything
> >>> already migrated? Anything missing? Any issue to report?
> >>
> >>
> >> Everything is migrated for the documentation of the *current* version.
> >> The old site contains documentation for both current and current-1. Is
> this something you would like to preserve ?
> >>
> >>>
> >>>
> >>>
> >>> On Tue, Nov 2, 2021 at 7:01 PM Martin Grigorov 
> wrote:
> >>> >
> >>> > Hi,
> >>> >
> >>> > Anyone willing to send a PR with the suggested improvement?
> >>> > Or at least open an issue with the well formulated text and I will
> add it!
> >>> >
> >>> > Regards,
> >>> > Martin
> >>> >
> >>> > On Tue, Nov 2, 2021, 18:08 Oscar Westra van Holthe - Kind <
> os...@westravanholthe.nl> wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> This is a huge improvement. Responsive, excellent navigation, syntax
&

Re: Converting SpecificRecord to GenericRecord of different Schema Versions

[Martin Grigorov]

Hi,

You will need to write a new org.apache.avro.io.Encoder.
If you succeed making it then please share it with the commuity via Pull
Request!
If you don't - please create an issue and we will try to help!

On Sun, Dec 5, 2021 at 6:31 PM KV 59  wrote:

> Hi All,
>
> Here is my situation, I have a SpecificRecord for a Schema S and I want to
> convert it into a GenericRecord of a compatible Schema T (It is an older
> version of S). I have seen many such examples but all strategies point to
> serializing the SpecificRecord to either bytes or JSON and deserializing
> back to the GenericRecord. This seems to be inefficient especially if the
> records are huge and in a high volume streaming scenario like mine.
>
> I cannot simply cast the SpecificRecord to GenericRecord because of some
> type incompatibilities like Enums and Instants.
>
> I have been looking at the SpecificRecordDatumWriter/Reader sources and
> try to build a Mapper which just sets the value in the GenericRecord but I
> cannot write such a mapper without the help of any of the protected and
> private methods in them.
>
> There is a same problem of converting a POJO to GenericRecord as well
>
>
> Appreciate your inputs and recommendations
>
> Thanks
> Kishore
>
>
>
>
>

Re: New website

[Martin Grigorov]

On Thu, Nov 4, 2021 at 5:04 PM Ismaël Mejía  wrote:

> Wow this is pretty neat ! Nice job Martin! A modern website can
> encourage more contributions.
> I am more interested on content than aesthetics first. Is everything
> already migrated? Anything missing? Any issue to report?
>

Everything is migrated for the documentation of the *current* version.
The old site contains documentation for both current and current-1. Is this
something you would like to preserve ?


>
>
> On Tue, Nov 2, 2021 at 7:01 PM Martin Grigorov 
> wrote:
> >
> > Hi,
> >
> > Anyone willing to send a PR with the suggested improvement?
> > Or at least open an issue with the well formulated text and I will add
> it!
> >
> > Regards,
> > Martin
> >
> > On Tue, Nov 2, 2021, 18:08 Oscar Westra van Holthe - Kind <
> os...@westravanholthe.nl> wrote:
> >>
> >> Hi,
> >>
> >> This is a huge improvement. Responsive, excellent navigation, syntax
> >> highlighting, ...
> >>
> >> The only downside I see was already mentioned by Lee: the landing page
> is
> >> too empty (also in a mobile browser).
> >> I think we could really benefit from mentioning the unique selling
> point of
> >> Avro here: "Your Data. Any Time, Anywhere." And then mention the
> language
> >> availability & excellent schema evolution.
> >>
> >> Kind regards,
> >> Oscar
> >>
> >> On Thu, 28 Oct 2021 at 10:43, Martin Grigorov 
> wrote:
> >>
> >> > Hi all,
> >> >
> >> > Please check the new candidate for Apache Avro website:
> >> > https://avro-website.netlify.app/
> >> >
> >> > It is based on Hugo and uses Docsy theme.
> >> > Its source code and instructions how to build could be found at
> >> > https://github.com/martin-g/avro-website.
> >> > The JIRA ticket is: https://issues.apache.org/jira/browse/AVRO-2175
> >> >
> >> > I am not web designer, so some things may look not finished.
> >> > I've just copied the HTML content from the old site (
> >> > https://avro.apache.org/) and converted it to Markdown for Hugo.
> >> >
> >> > Any feedback is welcome! With Pull Requests would be awesome!
> >> >
> >> > Regards,
> >> > Martin
> >> >
>

Re: New website

[Martin Grigorov]

Hi,

Anyone willing to send a PR with the suggested improvement?
Or at least open an issue with the well formulated text and I will add it!

Regards,
Martin

On Tue, Nov 2, 2021, 18:08 Oscar Westra van Holthe - Kind <
os...@westravanholthe.nl> wrote:

> Hi,
>
> This is a huge improvement. Responsive, excellent navigation, syntax
> highlighting, ...
>
> The only downside I see was already mentioned by Lee: the landing page is
> too empty (also in a mobile browser).
> I think we could really benefit from mentioning the unique selling point of
> Avro here: "Your Data. Any Time, Anywhere." And then mention the language
> availability & excellent schema evolution.
>
> Kind regards,
> Oscar
>
> On Thu, 28 Oct 2021 at 10:43, Martin Grigorov 
> wrote:
>
> > Hi all,
> >
> > Please check the new candidate for Apache Avro website:
> > https://avro-website.netlify.app/
> >
> > It is based on Hugo and uses Docsy theme.
> > Its source code and instructions how to build could be found at
> > https://github.com/martin-g/avro-website.
> > The JIRA ticket is: https://issues.apache.org/jira/browse/AVRO-2175
> >
> > I am not web designer, so some things may look not finished.
> > I've just copied the HTML content from the old site (
> > https://avro.apache.org/) and converted it to Markdown for Hugo.
> >
> > Any feedback is welcome! With Pull Requests would be awesome!
> >
> > Regards,
> > Martin
> >
>

Re: New website

[Martin Grigorov]

Hi Niels,

On Thu, Oct 28, 2021 at 2:29 PM Niels Basjes  wrote:

> Hi,
>
> To me this already looks a lot better than the default website, especially
> because now it also supports mobile devices.
> The exact look and feel for sites like this is always a discussion thing
> as a step 1: I don't have any input on this right now.
>
> What I am thinking about are things like:
> Where do we want to host this one?
> - On the existing Apache infrastructure?
>

Using exisitng Apache infrastructure should work! Other projects also use
Hugo, e.g. Apache Airflow.


> - Using Github pages? This would make it possible to automatically
> regenerate the site on a push to the master/main branch.
> - Somewhere else like the netlify this demo is hosted on?
>

I had a small issue with Netlify - it does not use `--recursive-modules`
for cloning and thus does not have the Docsy theme.
I didn't spend time in debugging it.
But if Avro team prefers Netlify then we could add the theme resources
directly, i.e. not use Git modules.


>
> Also:
> Do we want this to be a separate repository?
> Or do we want this to be part of the main code repository?
>

I don't see problems with any of these.
Maybe it will be one idea easier if it is in the same repo. This way
"build.sh dist" could copy the C/C++/C# docs in the same Git repo.

Regards,
Martin


>
> Niels
>
>
>
> On Thu, Oct 28, 2021 at 10:44 AM Martin Grigorov 
> wrote:
>
>> Hi all,
>>
>> Please check the new candidate for Apache Avro website:
>> https://avro-website.netlify.app/
>>
>> It is based on Hugo and uses Docsy theme.
>> Its source code and instructions how to build could be found at
>> https://github.com/martin-g/avro-website.
>> The JIRA ticket is: https://issues.apache.org/jira/browse/AVRO-2175
>>
>> I am not web designer, so some things may look not finished.
>> I've just copied the HTML content from the old site (
>> https://avro.apache.org/) and converted it to Markdown for Hugo.
>>
>> Any feedback is welcome! With Pull Requests would be awesome!
>>
>> Regards,
>> Martin
>>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

New website

[Martin Grigorov]

Hi all,

Please check the new candidate for Apache Avro website:
https://avro-website.netlify.app/

It is based on Hugo and uses Docsy theme.
Its source code and instructions how to build could be found at
https://github.com/martin-g/avro-website.
The JIRA ticket is: https://issues.apache.org/jira/browse/AVRO-2175

I am not web designer, so some things may look not finished.
I've just copied the HTML content from the old site (
https://avro.apache.org/) and converted it to Markdown for Hugo.

Any feedback is welcome! With Pull Requests would be awesome!

Regards,
Martin

Re: How to use avro::jsonEncoder with unicode symbols?

[Martin Grigorov]

Hi Anton,

Which Avro module do you use ? Java, Python, ... ?
Please show us your code!

On Tue, Oct 19, 2021 at 6:03 PM Anton  wrote:

> Hello,
>
>
>
> I’m trying to deserialize avro data to json and now I can’t properly
> receive non-ASCII symbols from encoder. In OutputStream of encoder I’m
> seeing Unicode codes like \\u042a in place of non ASCII symbols.
>
> How to properly translate this data to Unicode strings?
>