[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[ https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671368#comment-16671368 ] ASF GitHub Bot commented on AVRO-1891: -- scatrin commented on issue #329: Improved conversions handling + pluggable conversions support [AVRO-1891, AVRO-2065] URL: https://github.com/apache/avro/pull/329#issuecomment-434988845 Thanks @kgalieva, I'll take a look during the weekend. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Generated Java code fails with union containing logical type > > > Key: AVRO-1891 > URL: https://issues.apache.org/jira/browse/AVRO-1891 > Project: Avro > Issue Type: Bug > Components: java, logical types >Affects Versions: 1.8.1 >Reporter: Ross Black >Priority: Blocker > Fix For: 1.8.3 > > Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, > AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch > > > Example schema: > {code} > { > "type": "record", > "name": "RecordV1", > "namespace": "org.brasslock.event", > "fields": [ > { "name": "first", "type": ["null", {"type": "long", > "logicalType":"timestamp-millis"}]} > ] > } > {code} > The avro compiler generates a field using the relevant joda class: > {code} > public org.joda.time.DateTime first > {code} > Running the following code to perform encoding: > {code} > final RecordV1 record = new > RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z")); > final DatumWriter datumWriter = new > SpecificDatumWriter<>(record.getSchema()); > final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192); > final BinaryEncoder encoder = > EncoderFactory.get().directBinaryEncoder(stream, null); > datumWriter.write(record, encoder); > encoder.flush(); > final byte[] bytes = stream.toByteArray(); > {code} > fails with the exception stacktrace: > {code} > org.apache.avro.AvroRuntimeException: Unknown datum type > org.joda.time.DateTime: 2016-07-29T10:15:30.000Z > at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741) > at > org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293) > at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706) > at > org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192) > at > org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110) > at > org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87) > at > org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143) > at > org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60) > at > org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.j
[jira] [Created] (AVRO-2252) I'd like to improve Avro .NET (C#) library (many points)
Anton Ryzhov created AVRO-2252: -- Summary: I'd like to improve Avro .NET (C#) library (many points) Key: AVRO-2252 URL: https://issues.apache.org/jira/browse/AVRO-2252 Project: Avro Issue Type: Wish Components: csharp Reporter: Anton Ryzhov Hello all, The company where I'm working as a .NET developer is actively using Avro format. I'd like to improve Avro .NET (C#) library: 1) clean-up the code: - remove trailing spaces, unused namespace usings, etc. - remove unused dependency of log4net library - replace dependency of json library from direct reference to Nuget package 2) format the code to unify code style everywhere in the library - possibly using the Microsoft recommended code style for C# https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/inside-a-program/coding-conventions 3) use the latest C# 7.0 language features to make the code more compact and readable 4) make .NET 4.5 and .NET standard 2.0 versions of the library, keeping the existing compatibility with the .NET 3.5 - add asynchronous API to the .NET 4.5 and .NET standard 2.0 versions (async methods along with the synchronous ones). What do you think? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2252) I'd like to improve Avro .NET (C#) library (many points)
[ https://issues.apache.org/jira/browse/AVRO-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Ryzhov updated AVRO-2252: --- Description: Hello all, The company where I'm working as a .NET developer is actively using Avro format. I'd like to improve Avro .NET (C#) library: 1) clean-up the code: - remove trailing spaces, unused namespace usings, etc. - remove unused dependency of log4net library - replace dependency of json library from direct reference to Nuget package 2) format the code to unify code style everywhere in the library - possibly using the Microsoft recommended code style for C# [https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/inside-a-program/coding-conventions] 3) use the latest C# 7.0 language features to make the code more compact and readable 4) make .NET 4.5 and .NET standard 2.0 versions of the library, keeping the existing compatibility with the .NET 3.5 - add asynchronous API to the .NET 4.5 and .NET standard 2.0 versions (async methods along with the synchronous ones). What do you think? was: Hello all, The company where I'm working as a .NET developer is actively using Avro format. I'd like to improve Avro .NET (C#) library: 1) clean-up the code: - remove trailing spaces, unused namespace usings, etc. - remove unused dependency of log4net library - replace dependency of json library from direct reference to Nuget package 2) format the code to unify code style everywhere in the library - possibly using the Microsoft recommended code style for C# https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/inside-a-program/coding-conventions 3) use the latest C# 7.0 language features to make the code more compact and readable 4) make .NET 4.5 and .NET standard 2.0 versions of the library, keeping the existing compatibility with the .NET 3.5 - add asynchronous API to the .NET 4.5 and .NET standard 2.0 versions (async methods along with the synchronous ones). What do you think? > I'd like to improve Avro .NET (C#) library (many points) > > > Key: AVRO-2252 > URL: https://issues.apache.org/jira/browse/AVRO-2252 > Project: Avro > Issue Type: Wish > Components: csharp >Reporter: Anton Ryzhov >Priority: Major > > Hello all, > The company where I'm working as a .NET developer is actively using Avro > format. > I'd like to improve Avro .NET (C#) library: > 1) clean-up the code: > - remove trailing spaces, unused namespace usings, etc. > - remove unused dependency of log4net library > - replace dependency of json library from direct reference to Nuget package > 2) format the code to unify code style everywhere in the library > - possibly using the Microsoft recommended code style for C# > [https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/inside-a-program/coding-conventions] > 3) use the latest C# 7.0 language features to make the code more compact and > readable > 4) make .NET 4.5 and .NET standard 2.0 versions of the library, keeping the > existing compatibility with the .NET 3.5 > - add asynchronous API to the .NET 4.5 and .NET standard 2.0 versions (async > methods along with the synchronous ones). > What do you think? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1795) Python2: Cannot parse nested schemas
[ https://issues.apache.org/jira/browse/AVRO-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671833#comment-16671833 ] Chong Wang commented on AVRO-1795: -- Found an example in http://gisgeek.blogspot.com/2012/12/using-apache-avro-with-python.html seems did what is required. > Python2: Cannot parse nested schemas > > > Key: AVRO-1795 > URL: https://issues.apache.org/jira/browse/AVRO-1795 > Project: Avro > Issue Type: Bug > Components: python >Affects Versions: 1.8.0 >Reporter: Jakob Homan >Assignee: Jakob Homan >Priority: Major > > In the Java client, one can parse nested schemas by loading the nested schema > before the nesting schema. > For example, a header can be defined in one file: > {code:javascript}{ "namespace": "python.avro", > "type": "record", > "name": "header", > "fields": [ > { "name": "header_field", "type": "string" } >] > }{code} > and then included in another schema: > {code:javascript}{ "namespace": "python.avro", > "type": "record", > "name": "event", > "fields": [ > { "name": "header", "type": "python.avro.header" }, > { "name": "event_field", "type": "string" } > ] > }{code} > As long as one instantiates the Parser and loads the header first, the > schemas will be reconciled and merged correctly. > However, the Python client does not support this. The {{parse}} method of > the {{schema.py}} file always instantiates a new Names object to hold the > schemas: > {code}def parse(json_string): > """Constructs the Schema from the JSON text.""" > # TODO(hammer): preserve stack trace from JSON parse > # parse the JSON > try: > json_data = json.loads(json_string) > except: > raise SchemaParseException('Error parsing JSON: %s' % json_string) > # Initialize the names object > names = Names() > # construct the Avro Schema object > return make_avsc_object(json_data, names){code} > Some possible fixes for this are: > 1) Create a separate Parser class to mimic the Schema.Parser Java approach, > while deprecating the current parse method. > 2) Include Names as a global variable to the parse method, allowing multiple > parse calls to populate the same namespace. This breaks current behavior > (and at least one unit test depends on it), so would be backwards compatible. > 3) Create a new parse method that returns not only the schema, but also the > Names instance and accepts that instance. This keeps the code nice and > functional while exposing the Names class, which previously had been not > particularly public. > I like the first approach. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2208) Unneeded OSGi import generated for Guava dependency
[ https://issues.apache.org/jira/browse/AVRO-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672153#comment-16672153 ] ASF GitHub Bot commented on AVRO-2208: -- dkulp closed pull request #325: AVRO-2208 Unneeded OSGi import generated for Guava dependency URL: https://github.com/apache/avro/pull/325 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/lang/java/avro/pom.xml b/lang/java/avro/pom.xml index bf12ebb15..eff144a57 100644 --- a/lang/java/avro/pom.xml +++ b/lang/java/avro/pom.xml @@ -37,6 +37,7 @@ !org.apache.avro*, + !com.google.common*, com.thoughtworks.paranamer, org.codehaus.jackson*, org.xerial.snappy;resolution:=optional, This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unneeded OSGi import generated for Guava dependency > --- > > Key: AVRO-2208 > URL: https://issues.apache.org/jira/browse/AVRO-2208 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arnoud Glimmerveen >Priority: Major > Fix For: 1.9.0 > > > The classes from Guava used by Avro are shaded into the final jar, however > the OSGi imports are calculated before shading, resulting in a > com.google.common* dependency declared in Import-Package manifest header. As > a consequence in an OSGi environment one has to install a version of Guava > even though that version will not be used by Avro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2208) Unneeded OSGi import generated for Guava dependency
[ https://issues.apache.org/jira/browse/AVRO-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp reassigned AVRO-2208: - Assignee: Daniel Kulp > Unneeded OSGi import generated for Guava dependency > --- > > Key: AVRO-2208 > URL: https://issues.apache.org/jira/browse/AVRO-2208 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arnoud Glimmerveen >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The classes from Guava used by Avro are shaded into the final jar, however > the OSGi imports are calculated before shading, resulting in a > com.google.common* dependency declared in Import-Package manifest header. As > a consequence in an OSGi environment one has to install a version of Guava > even though that version will not be used by Avro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2208) Unneeded OSGi import generated for Guava dependency
[ https://issues.apache.org/jira/browse/AVRO-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2208: -- Fix Version/s: 1.9.0 > Unneeded OSGi import generated for Guava dependency > --- > > Key: AVRO-2208 > URL: https://issues.apache.org/jira/browse/AVRO-2208 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arnoud Glimmerveen >Priority: Major > Fix For: 1.9.0 > > > The classes from Guava used by Avro are shaded into the final jar, however > the OSGi imports are calculated before shading, resulting in a > com.google.common* dependency declared in Import-Package manifest header. As > a consequence in an OSGi environment one has to install a version of Guava > even though that version will not be used by Avro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2208) Unneeded OSGi import generated for Guava dependency
[ https://issues.apache.org/jira/browse/AVRO-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2208. --- Resolution: Fixed > Unneeded OSGi import generated for Guava dependency > --- > > Key: AVRO-2208 > URL: https://issues.apache.org/jira/browse/AVRO-2208 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arnoud Glimmerveen >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The classes from Guava used by Avro are shaded into the final jar, however > the OSGi imports are calculated before shading, resulting in a > com.google.common* dependency declared in Import-Package manifest header. As > a consequence in an OSGi environment one has to install a version of Guava > even though that version will not be used by Avro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2207) avro-maven-plugin don't specify encoding for files generated from avdls and avprs
[ https://issues.apache.org/jira/browse/AVRO-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672255#comment-16672255 ] ASF subversion and git services commented on AVRO-2207: --- Commit 26700949812e39f6ddedc63c53f8cb5562fd7f15 in avro's branch refs/heads/master from [~arkadiusx] [ https://gitbox.apache.org/repos/asf?p=avro.git;h=2670094 ] AVRO-2207: avro-maven-plugin: Using project.build.sourceEncoding as a encoding for files generated from avdl/avpr > avro-maven-plugin don't specify encoding for files generated from avdls and > avprs > - > > Key: AVRO-2207 > URL: https://issues.apache.org/jira/browse/AVRO-2207 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arek Burdach >Priority: Major > > When someone generates source file from avdl/avpr, compiler uses system > default encoding for output files. It can be different from this expected by > project's compiler. It is quite usual thing that developers specify source > encoding on project level using {{project.build.sourceEncoding}} property and > expects that it will solve the problem. > The problem doesn't exists when sources are generated from avsc. That > implementation sets output encoding to this taken from > {{project.build.sourceEncoding}}. > Proposed solutions is to use the same behaviour in other two cases (avdl and > avpr). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2207) avro-maven-plugin don't specify encoding for files generated from avdls and avprs
[ https://issues.apache.org/jira/browse/AVRO-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672254#comment-16672254 ] ASF GitHub Bot commented on AVRO-2207: -- dkulp closed pull request #324: AVRO-2207: avro-maven-plugin: Using project.build.sourceEncoding for files generated from avdl/avpr URL: https://github.com/apache/avro/pull/324 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/IDLProtocolMojo.java b/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/IDLProtocolMojo.java index 3c9542e3a..dfee5c73c 100644 --- a/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/IDLProtocolMojo.java +++ b/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/IDLProtocolMojo.java @@ -94,6 +94,7 @@ protected void doCompile(String filename, File sourceDirectory, File outputDirec compiler.setFieldVisibility(getFieldVisibility()); compiler.setCreateSetters(createSetters); compiler.setEnableDecimalLogicalType(enableDecimalLogicalType); + compiler.setOutputCharacterEncoding(project.getProperties().getProperty("project.build.sourceEncoding")); compiler.compileToDestination(null, outputDirectory); } catch (ParseException e) { throw new IOException(e); diff --git a/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/ProtocolMojo.java b/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/ProtocolMojo.java index a4dbe65af..b370d664c 100644 --- a/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/ProtocolMojo.java +++ b/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/ProtocolMojo.java @@ -62,6 +62,7 @@ protected void doCompile(String filename, File sourceDirectory, File outputDirec compiler.setFieldVisibility(getFieldVisibility()); compiler.setCreateSetters(createSetters); compiler.setEnableDecimalLogicalType(enableDecimalLogicalType); + compiler.setOutputCharacterEncoding(project.getProperties().getProperty("project.build.sourceEncoding")); compiler.compileToDestination(src, outputDirectory); } This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > avro-maven-plugin don't specify encoding for files generated from avdls and > avprs > - > > Key: AVRO-2207 > URL: https://issues.apache.org/jira/browse/AVRO-2207 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arek Burdach >Priority: Major > > When someone generates source file from avdl/avpr, compiler uses system > default encoding for output files. It can be different from this expected by > project's compiler. It is quite usual thing that developers specify source > encoding on project level using {{project.build.sourceEncoding}} property and > expects that it will solve the problem. > The problem doesn't exists when sources are generated from avsc. That > implementation sets output encoding to this taken from > {{project.build.sourceEncoding}}. > Proposed solutions is to use the same behaviour in other two cases (avdl and > avpr). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2207) avro-maven-plugin don't specify encoding for files generated from avdls and avprs
[ https://issues.apache.org/jira/browse/AVRO-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp reassigned AVRO-2207: - Assignee: Daniel Kulp > avro-maven-plugin don't specify encoding for files generated from avdls and > avprs > - > > Key: AVRO-2207 > URL: https://issues.apache.org/jira/browse/AVRO-2207 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arek Burdach >Assignee: Daniel Kulp >Priority: Major > > When someone generates source file from avdl/avpr, compiler uses system > default encoding for output files. It can be different from this expected by > project's compiler. It is quite usual thing that developers specify source > encoding on project level using {{project.build.sourceEncoding}} property and > expects that it will solve the problem. > The problem doesn't exists when sources are generated from avsc. That > implementation sets output encoding to this taken from > {{project.build.sourceEncoding}}. > Proposed solutions is to use the same behaviour in other two cases (avdl and > avpr). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2207) avro-maven-plugin don't specify encoding for files generated from avdls and avprs
[ https://issues.apache.org/jira/browse/AVRO-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2207. --- Resolution: Fixed Fix Version/s: 1.9.0 > avro-maven-plugin don't specify encoding for files generated from avdls and > avprs > - > > Key: AVRO-2207 > URL: https://issues.apache.org/jira/browse/AVRO-2207 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arek Burdach >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > When someone generates source file from avdl/avpr, compiler uses system > default encoding for output files. It can be different from this expected by > project's compiler. It is quite usual thing that developers specify source > encoding on project level using {{project.build.sourceEncoding}} property and > expects that it will solve the problem. > The problem doesn't exists when sources are generated from avsc. That > implementation sets output encoding to this taken from > {{project.build.sourceEncoding}}. > Proposed solutions is to use the same behaviour in other two cases (avdl and > avpr). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)