[jira] [Resolved] (AVRO-2335) Remove Joda Time Library
[
https://issues.apache.org/jira/browse/AVRO-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2335.
-
Resolution: Fixed
> Remove Joda Time Library
>
>
> Key: AVRO-2335
> URL: https://issues.apache.org/jira/browse/AVRO-2335
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: David Mollitor
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.10.0
>
>
> Since [AVRO-2079] was completed, Avro supports Java's standard {{java.time}}
> libraries. Please remove the Joda Time implementations and the Joda Time
> dependency from Avro. Avro 2.0 perhaps?
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. *Users are now asked to migrate to java.time (JSR-310).*
> https://www.joda.org/joda-time/
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (AVRO-2539) ThriftData produces not compatible avro schemas
[
https://issues.apache.org/jira/browse/AVRO-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2539:
Description:
Currently, ThrifdData class produces not compatible avro schema in terms of
AvroCompatibility rules.
For example, consider this thrift structs:
{code:java}
struct V1 {
1: required string f1,
2: optional string f2
}
struct V2 {
1: required string f1,
2: optional string f2,
3: optional string f3
}{code}
Produced schemas will be:
{noformat}
{"type":"record","name":"V1","namespace":"serialization.thrift.test","fields":[{"name":"f1","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"f2","type":["null",{"type":"string","avro.java.string":"String"}],"default":null}]}
{"type":"record","name":"V2","namespace":"serialization.thrift.test","fields":[{"name":"f1","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"f2","type":["null",{"type":"string","avro.java.string":"String"}]},
{"name":"f3","type":["null",{"type":"string","avro.java.string":"String"}]}]}
{noformat}
The problem is that if i try to check this schemas using BACKWARD compatibility
checker i will get false, because fields f2 and f3 has no default values even
if they are optional.
-Also, if i use default value in my thrift definition the resulting avro schema
will not contain it.-
There is possibility to fix default null values for optional fields using
NULL_DEFAULT_VALUE, but it will ignore the real default values.
-To get the real default values specified in *.thrift we can use instance of
thrift message to get default value, but this will require some refactoring of
such methods as getSchema and nullable in ThriftData.class-
was:
Currently, ThrifdData class produces not compatible avro schema in terms of
AvroCompatibility rules.
For example, consider this thrift structs:
{code:java}
struct V1 {
1: required string f1,
2: optional string f2
}
struct V1 {
1: required string f1,
2: optional string f2,
3: optional string f3
}{code}
Produced schemas will be:
{noformat}
{"type":"record","name":"V1","namespace":"serialization.thrift.test","fields":[{"name":"f1","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"f2","type":["null",{"type":"string","avro.java.string":"String"}],"default":null}]}
{"type":"record","name":"V2","namespace":"serialization.thrift.test","fields":[{"name":"f1","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"f2","type":["null",{"type":"string","avro.java.string":"String"}]},
{"name":"f3","type":["null",{"type":"string","avro.java.string":"String"}]}]}
{noformat}
The problem is that if i try to check this schemas using BACKWARD compatibility
checker i will get false, because fields f2 and f3 has no default values even
if they are optional.
-Also, if i use default value in my thrift definition the resulting avro schema
will not contain it.-
There is possibility to fix default null values for optional fields using
NULL_DEFAULT_VALUE, but it will ignore the real default values.
-To get the real default values specified in *.thrift we can use instance of
thrift message to get default value, but this will require some refactoring of
such methods as getSchema and nullable in ThriftData.class-
> ThriftData produces not compatible avro schemas
> ---
>
> Key: AVRO-2539
> URL: https://issues.apache.org/jira/browse/AVRO-2539
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0
>Reporter: Nikita Ryanov
>Priority: Major
>
> Currently, ThrifdData class produces not compatible avro schema in terms of
> AvroCompatibility rules.
> For example, consider this thrift structs:
> {code:java}
> struct V1 {
> 1: required string f1,
> 2: optional string f2
> }
> struct V2 {
> 1: required string f1,
> 2: optional string f2,
> 3: optional string f3
> }{code}
> Produced schemas will be:
> {noformat}
> {"type":"record","name":"V1","namespace":"serialization.thrift.test","fields":[{"name":"f1","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"f2","type":["null",{"type":"string","avro.java.string":"String"}],"default":null}]}
> {"type":"record","name":"V2","namespace":"serialization.thrift.test","fields":[{"name":"f1","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"f2","type":["null",{"type":"string","avro.java.string":"String"}]},
> {"name":"f3","type":["null",{"type":"string","avro.java.string":"String"}]}]}
> {noformat}
> The problem is that if i try to check this schemas using BACKWARD
> compatibility checker i will get false, because fields f2 and f3 has no
> default values even if they are optional.
> -Also, if i use default value in my thrift definition the resulting avro
> schema will not con
[jira] [Updated] (AVRO-2552) Exclude generated Thrift classes from Spotless check
[ https://issues.apache.org/jira/browse/AVRO-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2552: Environment: (was: Avro Thrift module includes several generated test classes, which fail when spotless is applied. These files should be ignored from the spotless check.) > Exclude generated Thrift classes from Spotless check > > > Key: AVRO-2552 > URL: https://issues.apache.org/jira/browse/AVRO-2552 > Project: Apache Avro > Issue Type: Bug > Components: java >Reporter: Nandor Kollar >Assignee: Nandor Kollar >Priority: Minor > > Avro Thrift module includes several generated test classes, which fail when > spotless is applied. These files should be ignored from the spotless check. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (AVRO-2552) Exclude generated Thrift classes from Spotless check
[ https://issues.apache.org/jira/browse/AVRO-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2552: Status: Patch Available (was: Open) > Exclude generated Thrift classes from Spotless check > > > Key: AVRO-2552 > URL: https://issues.apache.org/jira/browse/AVRO-2552 > Project: Apache Avro > Issue Type: Bug > Components: java > Environment: Avro Thrift module includes several generated test > classes, which fail when spotless is applied. These files should be ignored > from the spotless check. >Reporter: Nandor Kollar >Assignee: Nandor Kollar >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (AVRO-2552) Exclude generated Thrift classes from Spotless check
[ https://issues.apache.org/jira/browse/AVRO-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2552: Description: Avro Thrift module includes several generated test classes, which fail when spotless is applied. These files should be ignored from the spotless check. > Exclude generated Thrift classes from Spotless check > > > Key: AVRO-2552 > URL: https://issues.apache.org/jira/browse/AVRO-2552 > Project: Apache Avro > Issue Type: Bug > Components: java > Environment: Avro Thrift module includes several generated test > classes, which fail when spotless is applied. These files should be ignored > from the spotless check. >Reporter: Nandor Kollar >Assignee: Nandor Kollar >Priority: Minor > > Avro Thrift module includes several generated test classes, which fail when > spotless is applied. These files should be ignored from the spotless check. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (AVRO-2552) Exclude generated Thrift classes from Spotless check
Nandor Kollar created AVRO-2552: --- Summary: Exclude generated Thrift classes from Spotless check Key: AVRO-2552 URL: https://issues.apache.org/jira/browse/AVRO-2552 Project: Apache Avro Issue Type: Bug Components: java Environment: Avro Thrift module includes several generated test classes, which fail when spotless is applied. These files should be ignored from the spotless check. Reporter: Nandor Kollar Assignee: Nandor Kollar -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AVRO-2471) Java maven plugin code generation doesn't add conversion for timestamp-micros
[
https://issues.apache.org/jira/browse/AVRO-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918639#comment-16918639
]
Nandor Kollar commented on AVRO-2471:
-
I think this brings up a broader question: how to handle logical types when
there's more than one logical type mapping registered for the same Java type
(in this case for {{Instant}})? For generic and specific data the actual
conversion to be used can be differentiated based on the logical type in the
schema, but how do we resolve the ambiguity for reflective data, where the
schema is inferred from the Java types, and there's no additional logical type
information available? Should it default to the first match in the registered
conversions, or should it throw an error, and require explicit {{AvroSchema}}
annotation?
> Java maven plugin code generation doesn't add conversion for timestamp-micros
> -
>
> Key: AVRO-2471
> URL: https://issues.apache.org/jira/browse/AVRO-2471
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Marek Tracz
>Priority: Major
>
> Field in schema: (there is no single field with timestamp-millis logical type)
> {code:java}
> {
> "name": "RECORDING_TIME",
> "type": [
> "null",
> {
> "type": "long",
> "logicalType": "timestamp-micros"
> }
> ],
> "default": null
> }
> {code}
> Maven plugin configuration:
> {code:xml}
>
> org.apache.avro
> avro-maven-plugin
> 1.9.0
>
>
>
> schema
>
>
> String
>
> true
>
> ${project.basedir}/src/main/resources/
>
>
>
>
> {code}
> Part of the generated class:
> {code:java}
> private static SpecificData MODEL$ = new SpecificData();
> static {
> MODEL$.addLogicalTypeConversion(new
> org.apache.avro.data.TimeConversions.DateConversion());
> MODEL$.addLogicalTypeConversion(new
> org.apache.avro.data.TimeConversions.TimestampMillisConversion()); // <---
> this should be TimestampMicrosConversion
> MODEL$.addLogicalTypeConversion(new
> org.apache.avro.Conversions.DecimalConversion());
> }
> {code}
> For example this code:
> {code:java}
> Data data = Data.newBuilder()
> .setRECORDINGTIME(Instant.now())
> .build();
> {code}
> Fails during comparison:
> {noformat}
> org.apache.kafka.common.errors.SerializationException: Error serializing Avro
> message
> Caused by: org.apache.avro.AvroRuntimeException: Unknown datum type
> java.time.Instant: 2019-07-12T14:24:47.322Z
> at
> org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:887)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:420)
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:850)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:249)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:142)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:98)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:195)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:130)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:98)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:195)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:130)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
> at
> io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:92)
> at
> io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53)
> at
> org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:65)
> at
> org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:55)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:841)
>
[jira] [Resolved] (AVRO-2530) Bump spotless-maven-plugin from 1.21.1 to 1.24.1
[ https://issues.apache.org/jira/browse/AVRO-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2530. - Resolution: Fixed > Bump spotless-maven-plugin from 1.21.1 to 1.24.1 > > > Key: AVRO-2530 > URL: https://issues.apache.org/jira/browse/AVRO-2530 > Project: Apache Avro > Issue Type: Task >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AVRO-2485) ClassCastException on Nullable Map Value Schemas with Backward Compatible Changes (Field Addition)
[
https://issues.apache.org/jira/browse/AVRO-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893148#comment-16893148
]
Nandor Kollar commented on AVRO-2485:
-
[~wisegas] I don't know any location for the release schedule, actually
releases are made on a more or less ad-hoc manner, based on community
consensus, and it also depends when someone has time to create the release
artifacts and kick off the process. I'd recommend you to either/both subscribe
to dev and users group to get updates about the release process, or watch the
relevant release Jira: AVRO-2476.
> ClassCastException on Nullable Map Value Schemas with Backward Compatible
> Changes (Field Addition)
> --
>
> Key: AVRO-2485
> URL: https://issues.apache.org/jira/browse/AVRO-2485
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
> Environment: Operating System: Mac OS 10.13.6
> JDK version: 1.8_172
> JUnit version: 4.12
>Reporter: Sage Pierce
>Priority: Critical
>
> In one of our Java applications, upon upgrading from
> {{org.apache.avro:avro:1.8.2}} to {{org.apache.avro:avro:1.9.0}}, we are
> seeing deserialization failures with the following exception:
> {code:java}
> java.lang.ClassCastException: org.apache.avro.Resolver$ReaderUnion cannot be
> cast to org.apache.avro.Resolver$Container
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:99)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:110)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:141)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:65)
> at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
> at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:50)
> at
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:297)
> at
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:128)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> Further inspection of the stack trace and problematic data shows the issue
> lies with nullable (unioned) Map Schemas, whose values are complex types, and
> those types having backward compatible changes between the Writer and Reader
> Schemas. The following test exposes the issue:
> {code:java|title=AvroUpgradeTest.java}
> import java.io.ByteArrayOutputStream;
> import java.util.Collections;
> import org.apache.avro.Schema;
> import org.apache.avro.io.DecoderFactory;
> import org.apache.avro.io.EncoderFactory;
> import org.apache.avro.reflect.ReflectData;
> import org.apache.avro.reflect.ReflectDatumWriter;
> import org.junit.Test;
> import static org.junit.Assert.assertEquals;
> public class AvroUpgradeTest {
> @Test
> public void
> nullableMapWithBackwardCompatibleValueChangesCanBeDeserialized() throws
> Exception {
> TestDataWithNullableMap data = new TestDataWithNullableMap();
> data.setMap(Collections.singletonMap("KEY", TestData.create()));
> Schema writerSchema =
> ReflectData.get().getSchema(TestDataWithNullableMap.class);
> Schema readerMapSchema =
> Schema.createMap(Schema.createRecord(TestData.class.getName(), null, null,
> false,
> Collections.singletonList(new Schema.Field("data1",
> Schema.create(Schema.Type.STRING), null, Object.class.cast(null);
> Schema readerSchema =
> Schema.createRecord(TestDataWithNullableMap.class.getName(), null, null,
> false,
> Collections.singletonList(new Schema.Field("map",
> ReflectData.makeNullable(readerMapSchema), null, Object.class.cast(null;
> ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
> new ReflectDatumWriter<>(writerSchema, ReflectData.get()).write(data,
> EncoderFactory.get().directBinaryEncoder(outputStream, null));
> byte[] serialized = outputStream.toByteArray();
> TestDataWithNullableMap deserialized = new
> ReflectDecoderAvroDeserializer.ReflectDecoderDatumReader(writerSchema,
> readerSchema, ReflectData.get())
> .read(null, DecoderFactory.get().binaryDecoder(serialized, 0,
> serialized.length, null));
> assertEquals(data.getMap().get("KEY").getData1(),
> deserialized.getMap().get("KEY").getData1());
> }
> }
> {code}
> Note: I'm having to do a some manual Schema generation to force the
> appearance of the Writer's Schema making a (backward compatible) field
> addition.
> Relevant POJOs are as follows:
> {code:java|title=TestDataWithNullableMap.java}
> import java.util.Map;
> impor
[jira] [Commented] (AVRO-2485) ClassCastException on Nullable Map Value Schemas with Backward Compatible Changes (Field Addition)
[
https://issues.apache.org/jira/browse/AVRO-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892879#comment-16892879
]
Nandor Kollar commented on AVRO-2485:
-
[~wisegas] I think this is a know issue (see AVRO-2400 and AVRO-2415), which is
fixed on master, and will be released in 1.9.1.
> ClassCastException on Nullable Map Value Schemas with Backward Compatible
> Changes (Field Addition)
> --
>
> Key: AVRO-2485
> URL: https://issues.apache.org/jira/browse/AVRO-2485
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
> Environment: Operating System: Mac OS 10.13.6
> JDK version: 1.8_172
> JUnit version: 4.12
>Reporter: Sage Pierce
>Priority: Critical
>
> In one of our Java applications, upon upgrading from
> {{org.apache.avro:avro:1.8.2}} to {{org.apache.avro:avro:1.9.0}}, we are
> seeing deserialization failures with the following exception:
> {code:java}
> java.lang.ClassCastException: org.apache.avro.Resolver$ReaderUnion cannot be
> cast to org.apache.avro.Resolver$Container
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:99)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:110)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:141)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:65)
> at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
> at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:50)
> at
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:297)
> at
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:128)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> Further inspection of the stack trace and problematic data shows the issue
> lies with nullable (unioned) Map Schemas, whose values are complex types, and
> those types having backward compatible changes between the Writer and Reader
> Schemas. The following test exposes the issue:
> {code:java|title=AvroUpgradeTest.java}
> import java.io.ByteArrayOutputStream;
> import java.util.Collections;
> import org.apache.avro.Schema;
> import org.apache.avro.io.DecoderFactory;
> import org.apache.avro.io.EncoderFactory;
> import org.apache.avro.reflect.ReflectData;
> import org.apache.avro.reflect.ReflectDatumWriter;
> import org.junit.Test;
> import static org.junit.Assert.assertEquals;
> public class AvroUpgradeTest {
> @Test
> public void
> nullableMapWithBackwardCompatibleValueChangesCanBeDeserialized() throws
> Exception {
> TestDataWithNullableMap data = new TestDataWithNullableMap();
> data.setMap(Collections.singletonMap("KEY", TestData.create()));
> Schema writerSchema =
> ReflectData.get().getSchema(TestDataWithNullableMap.class);
> Schema readerMapSchema =
> Schema.createMap(Schema.createRecord(TestData.class.getName(), null, null,
> false,
> Collections.singletonList(new Schema.Field("data1",
> Schema.create(Schema.Type.STRING), null, Object.class.cast(null);
> Schema readerSchema =
> Schema.createRecord(TestDataWithNullableMap.class.getName(), null, null,
> false,
> Collections.singletonList(new Schema.Field("map",
> ReflectData.makeNullable(readerMapSchema), null, Object.class.cast(null;
> ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
> new ReflectDatumWriter<>(writerSchema, ReflectData.get()).write(data,
> EncoderFactory.get().directBinaryEncoder(outputStream, null));
> byte[] serialized = outputStream.toByteArray();
> TestDataWithNullableMap deserialized = new
> ReflectDecoderAvroDeserializer.ReflectDecoderDatumReader(writerSchema,
> readerSchema, ReflectData.get())
> .read(null, DecoderFactory.get().binaryDecoder(serialized, 0,
> serialized.length, null));
> assertEquals(data.getMap().get("KEY").getData1(),
> deserialized.getMap().get("KEY").getData1());
> }
> }
> {code}
> Note: I'm having to do a some manual Schema generation to force the
> appearance of the Writer's Schema making a (backward compatible) field
> addition.
> Relevant POJOs are as follows:
> {code:java|title=TestDataWithNullableMap.java}
> import java.util.Map;
> import org.apache.avro.reflect.Nullable;
> public class TestDataWithNullableMap {
> @Nullable
> private Map map;
> public Map getMap() {
> return map;
> }
> public void setMap(Map map) {
> this.map = map;
> }
> }
> {code}
> —
> {code:java|title=Test
[jira] [Updated] (AVRO-2483) Fix broken links to wiki pages in the documentation
[ https://issues.apache.org/jira/browse/AVRO-2483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2483: Resolution: Fixed Status: Resolved (was: Patch Available) > Fix broken links to wiki pages in the documentation > --- > > Key: AVRO-2483 > URL: https://issues.apache.org/jira/browse/AVRO-2483 > Project: Apache Avro > Issue Type: Bug > Components: doc >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Minor > Fix For: 1.10.0 > > > The Avro site has some links to the wiki, but they refers to the obsolete URL. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AVRO-2483) Fix broken links to wiki pages in the documentation
[ https://issues.apache.org/jira/browse/AVRO-2483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2483: Fix Version/s: 1.10.0 > Fix broken links to wiki pages in the documentation > --- > > Key: AVRO-2483 > URL: https://issues.apache.org/jira/browse/AVRO-2483 > Project: Apache Avro > Issue Type: Bug > Components: doc >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Minor > Fix For: 1.10.0 > > > The Avro site has some links to the wiki, but they refers to the obsolete URL. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AVRO-2461) Add compression level support to avro-tools' fromjson and recodec
[
https://issues.apache.org/jira/browse/AVRO-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2461:
Resolution: Fixed
Fix Version/s: 1.10.0
Status: Resolved (was: Patch Available)
> Add compression level support to avro-tools' fromjson and recodec
> -
>
> Key: AVRO-2461
> URL: https://issues.apache.org/jira/browse/AVRO-2461
> Project: Apache Avro
> Issue Type: Improvement
> Components: java, tools
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 1.10.0
>
>
> These commands have the {{--level}} option, but it's applied only to deflate
> and xz.
> {code}
> $ java -jar avro-tools-1.9.0.jar fromjson
> Expected 1 arg: input_file
> Option Description
> -- ---
> --codec Compression codec (default: null)
> --levelCompression level (only applies to deflate and xz)
> (default: -1)
> {code}
> Zstandard also has compression level feature, so it'd be useful if users can
> specify it.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (AVRO-2466) Fix a malformed schema in the share/test/schemas directory
[
https://issues.apache.org/jira/browse/AVRO-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2466:
Resolution: Fixed
Fix Version/s: 1.10.0
Status: Resolved (was: Patch Available)
> Fix a malformed schema in the share/test/schemas directory
> --
>
> Key: AVRO-2466
> URL: https://issues.apache.org/jira/browse/AVRO-2466
> Project: Apache Avro
> Issue Type: Bug
> Components: misc
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
> Fix For: 1.10.0
>
>
> The schema defined in share/test/schemas/reserved.avsc is invalid because of
> its trailing comma:
> {code}
> $ python -c 'from avro.schema import parse;
> parse(open("../../share/test/schemas/reserved.avsc").read())'
> Traceback (most recent call last):
> File "", line 1, in
> File "/home/sekikn/repo/avro/lang/py/src/avro/schema.py", line 976, in parse
> json_data = json.loads(json_string)
> File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python2.7/json/decoder.py", line 367, in decode
> raise ValueError(errmsg("Extra data", s, end, len(s)))
> avro.schema.SchemaParseException: Error parsing JSON: {"name":
> "org.apache.avro.test.Reserved", "type": "enum",
> "symbols": ["default","class","int"]},
> , error = Extra data: line 2 column 39 - line 3 column 1 (char 96 - 98)
> $ cat ../../share/test/schemas/reserved.avsc
> {"name": "org.apache.avro.test.Reserved", "type": "enum",
> "symbols": ["default","class","int"]},
> {code}
> This file doesn't seem to be used as far as I investigated, but I'd rather
> fix this than remove since it might be useful for some test.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (AVRO-2328) Support distinguishing between LocalDateTime and Instant semantics in timestamps
[ https://issues.apache.org/jira/browse/AVRO-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2328: Resolution: Fixed Fix Version/s: 1.10.0 Status: Resolved (was: Patch Available) > Support distinguishing between LocalDateTime and Instant semantics in > timestamps > > > Key: AVRO-2328 > URL: https://issues.apache.org/jira/browse/AVRO-2328 > Project: Apache Avro > Issue Type: Task >Reporter: Zoltan Ivanfi >Assignee: Nandor Kollar >Priority: Major > Fix For: 1.10.0 > > > Different SQL engines of the Hadoop stack support different timestamp > semantics. The range of supported semantics is about to be extended even > further. While some of the new timestamp types can be added to SQL without > explicit support from the file formats, others require new physical types. > File format support would be beneficial even for timestamp semantics where it > is not strictly required, because it would enable correct interpretation > without an SQL schema or any other kind of manual configuration. > This JIRA is about supporting the LocalDateTime and Instant semantics. See > [this > document|https://docs.google.com/document/d/1E-7miCh4qK6Mg54b-Dh5VOyhGX8V4xdMXKIHJL36a9U/edit#] > for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2460) Add zstd codec support to the Python3 bindings
[ https://issues.apache.org/jira/browse/AVRO-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2460: Resolution: Fixed Status: Resolved (was: Patch Available) > Add zstd codec support to the Python3 bindings > -- > > Key: AVRO-2460 > URL: https://issues.apache.org/jira/browse/AVRO-2460 > Project: Apache Avro > Issue Type: Improvement > Components: python >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2460) Add zstd codec support to the Python3 bindings
[ https://issues.apache.org/jira/browse/AVRO-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2460: Fix Version/s: 1.10.0 > Add zstd codec support to the Python3 bindings > -- > > Key: AVRO-2460 > URL: https://issues.apache.org/jira/browse/AVRO-2460 > Project: Apache Avro > Issue Type: Improvement > Components: python >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2450) log message about failure to load SnappyCodec
[ https://issues.apache.org/jira/browse/AVRO-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2450: Resolution: Fixed Status: Resolved (was: Patch Available) > log message about failure to load SnappyCodec > - > > Key: AVRO-2450 > URL: https://issues.apache.org/jira/browse/AVRO-2450 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.9.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.10.0, 1.9.1 > > > We catch Throwable and then silently return null without giving any means to > find out why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2450) log message about failure to load SnappyCodec
[ https://issues.apache.org/jira/browse/AVRO-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2450: Fix Version/s: 1.9.1 1.10.0 > log message about failure to load SnappyCodec > - > > Key: AVRO-2450 > URL: https://issues.apache.org/jira/browse/AVRO-2450 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.9.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.10.0, 1.9.1 > > > We catch Throwable and then silently return null without giving any means to > find out why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2448) Flaky test for open file descriptors
[
https://issues.apache.org/jira/browse/AVRO-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2448:
Fix Version/s: 1.9.1
> Flaky test for open file descriptors
>
>
> Key: AVRO-2448
> URL: https://issues.apache.org/jira/browse/AVRO-2448
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Trivial
> Fix For: 1.10.0, 1.9.1
>
>
> In random situations (like here
> [https://travis-ci.org/apache/avro/jobs/548635040] ) the test
> TestDataFileReader : testForLeakingFileDescriptors fails with a message like
> this
>
> {quote}[ERROR]
> testForLeakingFileDescriptors(org.apache.avro.TestDataFileReader) Time
> elapsed: 0.026 s <<< FAILURE!
> java.lang.AssertionError: File descriptor leaked from new DataFileReader()
> expected:<38> but was:<35>{quote}
> The assessment is that during the test run the garbage collector cleaned up
> some of the open file descriptors and thus the expected number of open file
> descriptors is lower than expected by the test.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2448) Flaky test for open file descriptors
[
https://issues.apache.org/jira/browse/AVRO-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2448.
-
Resolution: Fixed
Fix Version/s: 1.10.0
> Flaky test for open file descriptors
>
>
> Key: AVRO-2448
> URL: https://issues.apache.org/jira/browse/AVRO-2448
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Trivial
> Fix For: 1.10.0
>
>
> In random situations (like here
> [https://travis-ci.org/apache/avro/jobs/548635040] ) the test
> TestDataFileReader : testForLeakingFileDescriptors fails with a message like
> this
>
> {quote}[ERROR]
> testForLeakingFileDescriptors(org.apache.avro.TestDataFileReader) Time
> elapsed: 0.026 s <<< FAILURE!
> java.lang.AssertionError: File descriptor leaked from new DataFileReader()
> expected:<38> but was:<35>{quote}
> The assessment is that during the test run the garbage collector cleaned up
> some of the open file descriptors and thus the expected number of open file
> descriptors is lower than expected by the test.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2428) Update Hadoop version in the MR document
[ https://issues.apache.org/jira/browse/AVRO-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2428: Fix Version/s: 1.9.1 > Update Hadoop version in the MR document > > > Key: AVRO-2428 > URL: https://issues.apache.org/jira/browse/AVRO-2428 > Project: Apache Avro > Issue Type: Improvement > Components: doc >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Minor > Fix For: 1.10.0, 1.9.1 > > > Support for Hadoop 1.x is removed by AVRO-2059, but the examples still use > Hadoop 1.1.0 (e.g., http://avro.apache.org/docs/1.9.0/mr.html). Should we > also update them? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2430) Bump zstd 1.3.8-6 to 1.4.0-1
[ https://issues.apache.org/jira/browse/AVRO-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2430. - Resolution: Fixed > Bump zstd 1.3.8-6 to 1.4.0-1 > > > Key: AVRO-2430 > URL: https://issues.apache.org/jira/browse/AVRO-2430 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0, 1.9.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2428) Update Hadoop version in the MR document
[ https://issues.apache.org/jira/browse/AVRO-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2428. - Resolution: Fixed Fix Version/s: 1.10.0 > Update Hadoop version in the MR document > > > Key: AVRO-2428 > URL: https://issues.apache.org/jira/browse/AVRO-2428 > Project: Apache Avro > Issue Type: Improvement > Components: doc >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Minor > Fix For: 1.10.0 > > > Support for Hadoop 1.x is removed by AVRO-2059, but the examples still use > Hadoop 1.1.0 (e.g., http://avro.apache.org/docs/1.9.0/mr.html). Should we > also update them? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2421) Allow multiple @AvroAlias annotations on java classes and their fields
[ https://issues.apache.org/jira/browse/AVRO-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2421. - Resolution: Fixed Fix Version/s: 1.10.0 > Allow multiple @AvroAlias annotations on java classes and their fields > -- > > Key: AVRO-2421 > URL: https://issues.apache.org/jira/browse/AVRO-2421 > Project: Apache Avro > Issue Type: Improvement >Reporter: Johannes Schulte >Assignee: Johannes Schulte >Priority: Minor > Fix For: 1.10.0 > > > The avro schema specification allows defining multiple aliases for fields and > records and this can be useful especially when using java classes to define > the schema (refactoring & namespace/package changes). > With the introduction of @Repeatable in java 1.8 there exists an easy way to > expose this flexibility in the Reflect-based world. This patch enables > defining multiple @AvroAlias annotation on both fields and classes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2421) Allow multiple @AvroAlias annotations on java classes and their fields
[ https://issues.apache.org/jira/browse/AVRO-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2421: --- Assignee: Johannes Schulte > Allow multiple @AvroAlias annotations on java classes and their fields > -- > > Key: AVRO-2421 > URL: https://issues.apache.org/jira/browse/AVRO-2421 > Project: Apache Avro > Issue Type: Improvement >Reporter: Johannes Schulte >Assignee: Johannes Schulte >Priority: Minor > > The avro schema specification allows defining multiple aliases for fields and > records and this can be useful especially when using java classes to define > the schema (refactoring & namespace/package changes). > With the introduction of @Repeatable in java 1.8 there exists an easy way to > expose this flexibility in the Reflect-based world. This patch enables > defining multiple @AvroAlias annotation on both fields and classes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2428) Update Hadoop version in the MR document
[ https://issues.apache.org/jira/browse/AVRO-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866394#comment-16866394 ] Nandor Kollar commented on AVRO-2428: - Good point, the documentation is indeed out of date, +1 on updating this part! > Update Hadoop version in the MR document > > > Key: AVRO-2428 > URL: https://issues.apache.org/jira/browse/AVRO-2428 > Project: Apache Avro > Issue Type: Improvement > Components: doc >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Minor > > Support for Hadoop 1.x is removed by AVRO-2059, but the examples still use > Hadoop 1.1.0 (e.g., http://avro.apache.org/docs/1.9.0/mr.html). Should we > also update them? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2418) BUG in schema resolver, when resolving map<> to union {null, map<>}
[
https://issues.apache.org/jira/browse/AVRO-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2418.
-
Resolution: Duplicate
Issue addressed in AVRO-2415.
> BUG in schema resolver, when resolving map<> to union {null, map<>}
> ---
>
> Key: AVRO-2418
> URL: https://issues.apache.org/jira/browse/AVRO-2418
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Zoltan Farkas
>Priority: Major
>
> Here is unit test to reproduce the issue:
> {code}
> package org.apache.avro.io.parsing;
> import java.io.ByteArrayInputStream;
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.io.InputStream;
> import java.io.OutputStream;
> import java.util.HashMap;
> import java.util.Map;
> import org.junit.Assert;
> import org.apache.avro.Schema;
> import org.apache.avro.SchemaBuilder;
> import org.apache.avro.generic.GenericData;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericDatumWriter;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.generic.GenericRecordBuilder;
> import org.apache.avro.io.DatumReader;
> import org.apache.avro.io.DatumWriter;
> import org.apache.avro.io.Decoder;
> import org.apache.avro.io.DecoderFactory;
> import org.apache.avro.io.Encoder;
> import org.apache.avro.io.EncoderFactory;
> import org.apache.avro.util.Utf8;
> import org.junit.Test;
> /**
> *
> * @author Zoltan Farkas
> */
> public class TestUnionPromotion {
> @Test
> public void testUnionPromotionCollection() throws Exception {
> Schema directFieldSchema =
> SchemaBuilder.record("MyRecord").namespace("ns").fields().name("field1").type().map()
> .values().stringType().noDefault().endRecord();
> Schema schemaWithField =
> SchemaBuilder.record("MyRecord").namespace("ns").fields().name("field1").type().nullable()
> .map().values().stringType().noDefault().endRecord();
> Map data = new HashMap<>();
> data.put("a", "someValue");
> GenericData.Record record = new
> GenericRecordBuilder(directFieldSchema).set("field1", data).build();
> ByteArrayOutputStream bos = new ByteArrayOutputStream();
> writeAvroBin(bos, record);
> Object read = readAvroBin(new ByteArrayInputStream(bos.toByteArray()),
> directFieldSchema, schemaWithField);
> Map name = (Map) ((GenericRecord) read).get("field1");
> Assert.assertEquals("someValue", name.get(new Utf8("a")).toString());
> }
> private static Object readAvroBin(final InputStream input, final Schema
> writerSchema, final Schema readerSchema)
> throws IOException {
> DatumReader reader = new GenericDatumReader(writerSchema, readerSchema);
> DecoderFactory decoderFactory = DecoderFactory.get();
> Decoder decoder = decoderFactory.binaryDecoder(input, null);
> return reader.read(null, decoder);
> }
> private static void writeAvroBin(final OutputStream out, final
> GenericRecord req) throws IOException {
> @SuppressWarnings("unchecked")
> DatumWriter writer = new GenericDatumWriter(req.getSchema());
> Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
> writer.write(req, encoder);
> encoder.flush();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2415) Regression: Schema evolution of union containing array fails in 1.9.0
[
https://issues.apache.org/jira/browse/AVRO-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2415.
-
Resolution: Fixed
> Regression: Schema evolution of union containing array fails in 1.9.0
> -
>
> Key: AVRO-2415
> URL: https://issues.apache.org/jira/browse/AVRO-2415
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Martin Jubelgas
>Assignee: Martin Jubelgas
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> I've come across another example that does not resolve in 1.9.0 but did in
> 1.8.2:
> {code:java}
> public void testArrayInUnion() throws Exception {
> Schema writerSchema = SchemaBuilder.array().items().intType();
> Schema readerSchema = SchemaBuilder.unionOf()
> .nullType()
> .and()
> .array().items().intType().endUnion();
> ResolvingDecoder.resolve(writerSchema, readerSchema );
> }
> {code}
> results in just the same kind of exception (even with pull request #526
> applied)
> {code:java}
> java.lang.ClassCastException: org.apache.avro.Resolver$ReaderUnion cannot be
> cast to org.apache.avro.Resolver$Container
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:95)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:65)
> at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2415) Regression: Schema evolution of union containing array fails in 1.9.0
[
https://issues.apache.org/jira/browse/AVRO-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2415:
Fix Version/s: 1.9.1
1.10.0
> Regression: Schema evolution of union containing array fails in 1.9.0
> -
>
> Key: AVRO-2415
> URL: https://issues.apache.org/jira/browse/AVRO-2415
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Martin Jubelgas
>Assignee: Martin Jubelgas
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> I've come across another example that does not resolve in 1.9.0 but did in
> 1.8.2:
> {code:java}
> public void testArrayInUnion() throws Exception {
> Schema writerSchema = SchemaBuilder.array().items().intType();
> Schema readerSchema = SchemaBuilder.unionOf()
> .nullType()
> .and()
> .array().items().intType().endUnion();
> ResolvingDecoder.resolve(writerSchema, readerSchema );
> }
> {code}
> results in just the same kind of exception (even with pull request #526
> applied)
> {code:java}
> java.lang.ClassCastException: org.apache.avro.Resolver$ReaderUnion cannot be
> cast to org.apache.avro.Resolver$Container
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:95)
> at
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:65)
> at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2328) Support distinguishing between LocalDateTime and Instant semantics in timestamps
[ https://issues.apache.org/jira/browse/AVRO-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2328: Status: Patch Available (was: Open) > Support distinguishing between LocalDateTime and Instant semantics in > timestamps > > > Key: AVRO-2328 > URL: https://issues.apache.org/jira/browse/AVRO-2328 > Project: Apache Avro > Issue Type: Task >Reporter: Zoltan Ivanfi >Assignee: Nandor Kollar >Priority: Major > > Different SQL engines of the Hadoop stack support different timestamp > semantics. The range of supported semantics is about to be extended even > further. While some of the new timestamp types can be added to SQL without > explicit support from the file formats, others require new physical types. > File format support would be beneficial even for timestamp semantics where it > is not strictly required, because it would enable correct interpretation > without an SQL schema or any other kind of manual configuration. > This JIRA is about supporting the LocalDateTime and Instant semantics. See > [this > document|https://docs.google.com/document/d/1E-7miCh4qK6Mg54b-Dh5VOyhGX8V4xdMXKIHJL36a9U/edit#] > for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2398) Bump Jackson to 2.9.9
[ https://issues.apache.org/jira/browse/AVRO-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2398. - Resolution: Fixed Fix Version/s: 1.10.0 > Bump Jackson to 2.9.9 > - > > Key: AVRO-2398 > URL: https://issues.apache.org/jira/browse/AVRO-2398 > Project: Apache Avro > Issue Type: Improvement >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > > This fixes CVE-2019-12086 on Databind: > https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.9 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2394) commit-msg Expects CHANGES.txt
[
https://issues.apache.org/jira/browse/AVRO-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2394.
-
Resolution: Fixed
Fix Version/s: 1.10.0
> commit-msg Expects CHANGES.txt
> --
>
> Key: AVRO-2394
> URL: https://issues.apache.org/jira/browse/AVRO-2394
> Project: Apache Avro
> Issue Type: Bug
> Components: misc
>Reporter: Brian Lachniet
>Assignee: Brian Lachniet
>Priority: Minor
> Fix For: 1.10.0
>
>
> The {{commit-msg}} client-side git hook looks for the issue ID in a
> {{CHANGES.txt}} file, which was removed in AVRO-2173. Remove this check from
> the {{commit-msg}} hook.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2390) Travis jobs are killed with "No output has been received"
[
https://issues.apache.org/jira/browse/AVRO-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2390:
Summary: Travis jobs are killed with "No output has been received" (was:
Tests Travis jobs are killed with "No output has been received")
> Travis jobs are killed with "No output has been received"
> -
>
> Key: AVRO-2390
> URL: https://issues.apache.org/jira/browse/AVRO-2390
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Nandor Kollar
>Priority: Major
>
> Travis jobs fail with
> {code}
> No output has been received in the last 10m0s, this potentially indicates a
> stalled build or something wrong with the build itself.
> Check the details on how to adjust your build configuration on:
> https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
> {code}
> I think travis_wait or using [pv|https://linux.die.net/man/1/pv] should solve
> this problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AVRO-2390) Tests Travis jobs are killed with "No output has been received"
Nandor Kollar created AVRO-2390:
---
Summary: Tests Travis jobs are killed with "No output has been
received"
Key: AVRO-2390
URL: https://issues.apache.org/jira/browse/AVRO-2390
Project: Apache Avro
Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Nandor Kollar
Travis jobs fail with
{code}
No output has been received in the last 10m0s, this potentially indicates a
stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on:
https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
{code}
I think travis_wait or using [pv|https://linux.die.net/man/1/pv] should solve
this problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AVRO-2386) Generated code doesn't compile
Nandor Kollar created AVRO-2386:
---
Summary: Generated code doesn't compile
Key: AVRO-2386
URL: https://issues.apache.org/jira/browse/AVRO-2386
Project: Apache Avro
Issue Type: Bug
Components: java
Affects Versions: 1.9.0
Reporter: Nandor Kollar
{code}
{"namespace": "org.apache.avro.codegentest.some",
"type": "record",
"name": "NestedSomeNamespaceRecord",
"doc" : "Test nested types with different namespace than the outer type",
"fields": [
{
"name": "nestedRecord",
"type": {
"namespace": "org.apache.avro.codegentest.other",
"type": "record",
"name": "NestedOtherNamespaceRecord",
"fields": [
{
"name": "someField",
"type": "int"
}
]
}
}]
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2386) Generated code doesn't compile, because customDecode and
[
https://issues.apache.org/jira/browse/AVRO-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2386:
Summary: Generated code doesn't compile, because customDecode and (was:
Generated code doesn't compile, because )
> Generated code doesn't compile, because customDecode and
> -
>
> Key: AVRO-2386
> URL: https://issues.apache.org/jira/browse/AVRO-2386
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Nandor Kollar
>Priority: Blocker
>
> The generated code for
> {code}
> {"namespace": "org.apache.avro.codegentest.some",
> "type": "record",
> "name": "NestedSomeNamespaceRecord",
> "doc" : "Test nested types with different namespace than the outer type",
> "fields": [
> {
> "name": "nestedRecord",
> "type": {
> "namespace": "org.apache.avro.codegentest.other",
> "type": "record",
> "name": "NestedOtherNamespaceRecord",
> "fields": [
> {
> "name": "someField",
> "type": "int"
> }
> ]
> }
> }]
> }
> {code}
> doesn't compile, because customEncode and customDecode methods are protected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2386) Generated code doesn't compile, because
[
https://issues.apache.org/jira/browse/AVRO-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2386:
Description:
The generated code for
{code}
{"namespace": "org.apache.avro.codegentest.some",
"type": "record",
"name": "NestedSomeNamespaceRecord",
"doc" : "Test nested types with different namespace than the outer type",
"fields": [
{
"name": "nestedRecord",
"type": {
"namespace": "org.apache.avro.codegentest.other",
"type": "record",
"name": "NestedOtherNamespaceRecord",
"fields": [
{
"name": "someField",
"type": "int"
}
]
}
}]
}
{code}
doesn't compile, because customEncode and customDecode methods are protected.
was:
{code}
{"namespace": "org.apache.avro.codegentest.some",
"type": "record",
"name": "NestedSomeNamespaceRecord",
"doc" : "Test nested types with different namespace than the outer type",
"fields": [
{
"name": "nestedRecord",
"type": {
"namespace": "org.apache.avro.codegentest.other",
"type": "record",
"name": "NestedOtherNamespaceRecord",
"fields": [
{
"name": "someField",
"type": "int"
}
]
}
}]
}
{code}
> Generated code doesn't compile, because
>
>
> Key: AVRO-2386
> URL: https://issues.apache.org/jira/browse/AVRO-2386
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Nandor Kollar
>Priority: Blocker
>
> The generated code for
> {code}
> {"namespace": "org.apache.avro.codegentest.some",
> "type": "record",
> "name": "NestedSomeNamespaceRecord",
> "doc" : "Test nested types with different namespace than the outer type",
> "fields": [
> {
> "name": "nestedRecord",
> "type": {
> "namespace": "org.apache.avro.codegentest.other",
> "type": "record",
> "name": "NestedOtherNamespaceRecord",
> "fields": [
> {
> "name": "someField",
> "type": "int"
> }
> ]
> }
> }]
> }
> {code}
> doesn't compile, because customEncode and customDecode methods are protected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2386) Generated code doesn't compile, because customDecode and customEncode are protected
[
https://issues.apache.org/jira/browse/AVRO-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2386:
Summary: Generated code doesn't compile, because customDecode and
customEncode are protected (was: Generated code doesn't compile, because
customDecode and )
> Generated code doesn't compile, because customDecode and customEncode are
> protected
> ---
>
> Key: AVRO-2386
> URL: https://issues.apache.org/jira/browse/AVRO-2386
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Nandor Kollar
>Priority: Blocker
>
> The generated code for
> {code}
> {"namespace": "org.apache.avro.codegentest.some",
> "type": "record",
> "name": "NestedSomeNamespaceRecord",
> "doc" : "Test nested types with different namespace than the outer type",
> "fields": [
> {
> "name": "nestedRecord",
> "type": {
> "namespace": "org.apache.avro.codegentest.other",
> "type": "record",
> "name": "NestedOtherNamespaceRecord",
> "fields": [
> {
> "name": "someField",
> "type": "int"
> }
> ]
> }
> }]
> }
> {code}
> doesn't compile, because customEncode and customDecode methods are protected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2386) Generated code doesn't compile, because
[
https://issues.apache.org/jira/browse/AVRO-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2386:
Summary: Generated code doesn't compile, because (was: Generated code
doesn't compile)
> Generated code doesn't compile, because
>
>
> Key: AVRO-2386
> URL: https://issues.apache.org/jira/browse/AVRO-2386
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Nandor Kollar
>Priority: Blocker
>
> {code}
> {"namespace": "org.apache.avro.codegentest.some",
> "type": "record",
> "name": "NestedSomeNamespaceRecord",
> "doc" : "Test nested types with different namespace than the outer type",
> "fields": [
> {
> "name": "nestedRecord",
> "type": {
> "namespace": "org.apache.avro.codegentest.other",
> "type": "record",
> "name": "NestedOtherNamespaceRecord",
> "fields": [
> {
> "name": "someField",
> "type": "int"
> }
> ]
> }
> }]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2381) logical type for fixed incorrectly parsed as disallowed doc attribute
[ https://issues.apache.org/jira/browse/AVRO-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2381: Priority: Blocker (was: Minor) > logical type for fixed incorrectly parsed as disallowed doc attribute > - > > Key: AVRO-2381 > URL: https://issues.apache.org/jira/browse/AVRO-2381 > Project: Apache Avro > Issue Type: Bug > Components: ruby >Affects Versions: 1.9.0 >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Blocker > Attachments: avro-2381-patch.txt > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2375) Allow repeating AvroMeta use
[ https://issues.apache.org/jira/browse/AVRO-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2375. - Resolution: Fixed > Allow repeating AvroMeta use > > > Key: AVRO-2375 > URL: https://issues.apache.org/jira/browse/AVRO-2375 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ivan Greene >Assignee: Ivan Greene >Priority: Minor > Fix For: 1.9.0 > > > AvroMeta should be a repeatable annotation, to allow the association of more > than one key-value pairs of metadata with a type or field within a schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2375) Allow repeating AvroMeta use
[ https://issues.apache.org/jira/browse/AVRO-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2375: --- Assignee: Ivan Greene > Allow repeating AvroMeta use > > > Key: AVRO-2375 > URL: https://issues.apache.org/jira/browse/AVRO-2375 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ivan Greene >Assignee: Ivan Greene >Priority: Minor > Fix For: 1.9.0 > > > AvroMeta should be a repeatable annotation, to allow the association of more > than one key-value pairs of metadata with a type or field within a schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1547) AvroApp Schema Tool
[ https://issues.apache.org/jira/browse/AVRO-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16817991#comment-16817991 ] Nandor Kollar commented on AVRO-1547: - [~brianolsen87] added you to the contributors and assigned the Jira to you as requested. > AvroApp Schema Tool > --- > > Key: AVRO-1547 > URL: https://issues.apache.org/jira/browse/AVRO-1547 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Lewis John McGibbney >Assignee: Brian Olsen >Priority: Minor > > Over in Gora, I have been thinking for a while that the process of writing > JSON data beans is rather time consuming when beans are LARGE. > I wanted to open this ticket for a while and now only get around to. I > proposed to have the following > A simple HTML webpage that defines a form of sorts, the form will enable > users to create JSON schemas and will be driven by enabling users to enter > Object values based on the current Avro specification document e.g. ti will > be restrictive in scope. > On top of this I propose to then use simple JQuery to send a request to the > JSONBlob API [0], obtain a JSON representation of the data and then pretty > print write this information to a file within the browser. The users can then > save this file focally and do with it what they wish. > I think that this page can easily be hosted alongside the current static Avro > website and that there is no need to write a web application for this yet. > I'll try to work on it sooner rather than later as this would also lower the > barrier for users of Gora (as I am sure it would Users of other technologies > requiring definition of Objects via JSOn schemas). > I've not assigned this against any component as there is none which I feel > appropriate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-1547) AvroApp Schema Tool
[ https://issues.apache.org/jira/browse/AVRO-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-1547: --- Assignee: Brian Olsen > AvroApp Schema Tool > --- > > Key: AVRO-1547 > URL: https://issues.apache.org/jira/browse/AVRO-1547 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Lewis John McGibbney >Assignee: Brian Olsen >Priority: Minor > > Over in Gora, I have been thinking for a while that the process of writing > JSON data beans is rather time consuming when beans are LARGE. > I wanted to open this ticket for a while and now only get around to. I > proposed to have the following > A simple HTML webpage that defines a form of sorts, the form will enable > users to create JSON schemas and will be driven by enabling users to enter > Object values based on the current Avro specification document e.g. ti will > be restrictive in scope. > On top of this I propose to then use simple JQuery to send a request to the > JSONBlob API [0], obtain a JSON representation of the data and then pretty > print write this information to a file within the browser. The users can then > save this file focally and do with it what they wish. > I think that this page can easily be hosted alongside the current static Avro > website and that there is no need to write a web application for this yet. > I'll try to work on it sooner rather than later as this would also lower the > barrier for users of Gora (as I am sure it would Users of other technologies > requiring definition of Objects via JSOn schemas). > I've not assigned this against any component as there is none which I feel > appropriate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2332) Support java_multiple_files protobuf option
[ https://issues.apache.org/jira/browse/AVRO-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2332: Fix Version/s: 1.9.0 > Support java_multiple_files protobuf option > --- > > Key: AVRO-2332 > URL: https://issues.apache.org/jira/browse/AVRO-2332 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ryo Okubo >Assignee: Ryo Okubo >Priority: Major > Fix For: 1.9.0 > > > Support java_multiple_files option, > [https://developers.google.com/protocol-buffers/docs/proto3#options] > The option helps to get classes via reflection. > > protoc doesn't generate outer class if the option enables, but current > getNamespace() doesn't consider that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2332) Support java_multiple_files protobuf option
[ https://issues.apache.org/jira/browse/AVRO-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2332. - Resolution: Fixed > Support java_multiple_files protobuf option > --- > > Key: AVRO-2332 > URL: https://issues.apache.org/jira/browse/AVRO-2332 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ryo Okubo >Assignee: Ryo Okubo >Priority: Major > Fix For: 1.9.0 > > > Support java_multiple_files option, > [https://developers.google.com/protocol-buffers/docs/proto3#options] > The option helps to get classes via reflection. > > protoc doesn't generate outer class if the option enables, but current > getNamespace() doesn't consider that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2332) Support java_multiple_files protobuf option
[ https://issues.apache.org/jira/browse/AVRO-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2332: --- Assignee: Ryo Okubo > Support java_multiple_files protobuf option > --- > > Key: AVRO-2332 > URL: https://issues.apache.org/jira/browse/AVRO-2332 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ryo Okubo >Assignee: Ryo Okubo >Priority: Major > > Support java_multiple_files option, > [https://developers.google.com/protocol-buffers/docs/proto3#options] > The option helps to get classes via reflection. > > protoc doesn't generate outer class if the option enables, but current > getNamespace() doesn't consider that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2344) Handle google.protobuf.Timestamp as timestamp logical type in Avro
[ https://issues.apache.org/jira/browse/AVRO-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2344: --- Assignee: Ryo Okubo > Handle google.protobuf.Timestamp as timestamp logical type in Avro > --- > > Key: AVRO-2344 > URL: https://issues.apache.org/jira/browse/AVRO-2344 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ryo Okubo >Assignee: Ryo Okubo >Priority: Major > > Current avro-protobuf converts google.protobuf.Timestamp to simply record but > I'd like to get timestamp logical type. > > refs" > * proto > [https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/timestamp.proto] > * avro [https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2335) Remove Joda Time Library
[
https://issues.apache.org/jira/browse/AVRO-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16805172#comment-16805172
]
Nandor Kollar commented on AVRO-2335:
-
In my opinion we should let Avro users be able to use Joda time in 1.9 and
defer the removal to 1.10, I vote for marking Joda as deprecated and making
JSR-310 as default for 1.9.
> Remove Joda Time Library
>
>
> Key: AVRO-2335
> URL: https://issues.apache.org/jira/browse/AVRO-2335
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: David Mollitor
>Assignee: Fokko Driesprong
>Priority: Major
>
> Since [AVRO-2079] was completed, Avro supports Java's standard {{java.time}}
> libraries. Please remove the Joda Time implementations and the Joda Time
> dependency from Avro. Avro 2.0 perhaps?
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. *Users are now asked to migrate to java.time (JSR-310).*
> https://www.joda.org/joda-time/
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2362) Cannot Run spotless:apply From Root Pom
[
https://issues.apache.org/jira/browse/AVRO-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2362.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Cannot Run spotless:apply From Root Pom
> ---
>
> Key: AVRO-2362
> URL: https://issues.apache.org/jira/browse/AVRO-2362
> Project: Apache Avro
> Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 1.9.0
>
>
> I was running {{mvn install}} from the root directory and it failed because
> some of my code was not 'spotless'. In response, I ran {{mvn
> spotless:apply}} but it fails to run from the root avro project directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2362) Cannot Run spotless:apply From Root Pom
[
https://issues.apache.org/jira/browse/AVRO-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804136#comment-16804136
]
Nandor Kollar commented on AVRO-2362:
-
Try with {{mvn com.diffplug.spotless:spotless-maven-plugin:1.20.0:apply}}
> Cannot Run spotless:apply From Root Pom
> ---
>
> Key: AVRO-2362
> URL: https://issues.apache.org/jira/browse/AVRO-2362
> Project: Apache Avro
> Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Priority: Major
>
> I was running {{mvn install}} from the root directory and it failed because
> some of my code was not 'spotless'. In response, I ran {{mvn
> spotless:apply}} but it fails to run from the root avro project directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2144) 404 - Not found - Invalid Url for CSharp documentation
[ https://issues.apache.org/jira/browse/AVRO-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2144. - Resolution: Fixed Fix Version/s: 1.9.0 > 404 - Not found - Invalid Url for CSharp documentation > -- > > Key: AVRO-2144 > URL: https://issues.apache.org/jira/browse/AVRO-2144 > Project: Apache Avro > Issue Type: Bug > Components: doc >Affects Versions: 1.8.2 >Reporter: Selva Chinnasamy >Assignee: Selva Chinnasamy >Priority: Minor > Fix For: 1.9.0 > > > Invalid Url for C# Api documentation > Selecting the below link > [C# API|http://avro.apache.org/docs/current/api/csharp/index.html] > leads to 404. Below is the current result: > h1. Not Found > The requested URL /docs/current/api/csharp/index.html was not found on this > server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2269) Improve usability of Perf.java
[
https://issues.apache.org/jira/browse/AVRO-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2269.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Improve usability of Perf.java
> --
>
> Key: AVRO-2269
> URL: https://issues.apache.org/jira/browse/AVRO-2269
> Project: Apache Avro
> Issue Type: Test
> Components: java
>Reporter: Raymie Stata
>Assignee: Raymie Stata
>Priority: Major
> Fix For: 1.9.0
>
>
> The class {{org.apache.avro.ipc.io.Perf}} is Avro's performance test suite.
> This JIRA aims to make it easier to use. Specifically:
> * Added a file {{performance-testing.html}} with guidance on how to use the
> suite
> * Added script {{run-script.sh}} that uses {{Perf}} to run structured
> experiments.
> * Added tests for performance of resolution of unchanged unions and
> enumerations, which will be subject to future optimizations.
> * Tweaks to {{Perf}} for better experimentation (e.g., support for minimum as
> well as average aggregation).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2291) GenericData.Array not reusable after AVRO-2050
[ https://issues.apache.org/jira/browse/AVRO-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2291. - Resolution: Fixed Fix Version/s: 1.9.0 > GenericData.Array not reusable after AVRO-2050 > -- > > Key: AVRO-2291 > URL: https://issues.apache.org/jira/browse/AVRO-2291 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Martin Jubelgas >Assignee: Martin Jubelgas >Priority: Minor > Fix For: 1.9.0 > > > The fix for AVRO-2050 left the reusing functionality of GenericData.Array > broken. By nulling all fields of the underlying array during > GenericData.Array.clear(), there was no more data to be reused in > GenericDatumReader.readArray(). > I've already posted a pull request to alleviate this issue in a backward > compatible matter as a possible solution. Comments welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2291) GenericData.Array not reusable after AVRO-2050
[ https://issues.apache.org/jira/browse/AVRO-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2291: --- Assignee: Martin Jubelgas > GenericData.Array not reusable after AVRO-2050 > -- > > Key: AVRO-2291 > URL: https://issues.apache.org/jira/browse/AVRO-2291 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Martin Jubelgas >Assignee: Martin Jubelgas >Priority: Minor > > The fix for AVRO-2050 left the reusing functionality of GenericData.Array > broken. By nulling all fields of the underlying array during > GenericData.Array.clear(), there was no more data to be reused in > GenericDatumReader.readArray(). > I've already posted a pull request to alleviate this issue in a backward > compatible matter as a possible solution. Comments welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2333) Remove commons-logging dependency
[
https://issues.apache.org/jira/browse/AVRO-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2333.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Remove commons-logging dependency
> -
>
> Key: AVRO-2333
> URL: https://issues.apache.org/jira/browse/AVRO-2333
> Project: Apache Avro
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.9.0
>
>
> Remove dependency on {{commons-logging}}. Now using {{SLF4J}} + {{Log4J}}.
> https://github.com/apache/avro/blob/master/lang/java/tools/pom.xml#L221-L225
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2341) Simplify ByteArrayByteSource Min Buffer Size
[ https://issues.apache.org/jira/browse/AVRO-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2341. - Resolution: Fixed Fix Version/s: 1.9.0 > Simplify ByteArrayByteSource Min Buffer Size > > > Key: AVRO-2341 > URL: https://issues.apache.org/jira/browse/AVRO-2341 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2345) Export getSchema() to bypass using reflection
[
https://issues.apache.org/jira/browse/AVRO-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2345.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Export getSchema() to bypass using reflection
> -
>
> Key: AVRO-2345
> URL: https://issues.apache.org/jira/browse/AVRO-2345
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ryo Okubo
>Assignee: Ryo Okubo
>Priority: Major
> Fix For: 1.9.0
>
>
> I'd like to use {{getSchema(descriptor)}} directly to enable 1) to get schema
> from protobuf instance doesn't have {{getDescriptor()}} created from
> reflection and 2) to pass not protoc-java classes e.g. Scala classes
> generated by [ScalaPB|https://github.com/scalapb/ScalaPB] has
> {{javaDescriptor()}} instread of {{getDescriptor()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AVRO-2345) Export getSchema() to bypass using reflection
[
https://issues.apache.org/jira/browse/AVRO-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-2345:
---
Assignee: Ryo Okubo
> Export getSchema() to bypass using reflection
> -
>
> Key: AVRO-2345
> URL: https://issues.apache.org/jira/browse/AVRO-2345
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ryo Okubo
>Assignee: Ryo Okubo
>Priority: Major
>
> I'd like to use {{getSchema(descriptor)}} directly to enable 1) to get schema
> from protobuf instance doesn't have {{getDescriptor()}} created from
> reflection and 2) to pass not protoc-java classes e.g. Scala classes
> generated by [ScalaPB|https://github.com/scalapb/ScalaPB] has
> {{javaDescriptor()}} instread of {{getDescriptor()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2348) Do Not Wrap JSON Strings in StringReader for Parsing
[
https://issues.apache.org/jira/browse/AVRO-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2348.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Do Not Wrap JSON Strings in StringReader for Parsing
>
>
> Key: AVRO-2348
> URL: https://issues.apache.org/jira/browse/AVRO-2348
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.9.0
>
>
> {code}
> public JsonParser createParser(String content)
> throws IOException,
>JsonParseException
> Method for constructing parser for parsing contents of given String.
> {code}
> https://fasterxml.github.io/jackson-core/javadoc/2.3.0/com/fasterxml/jackson/core/JsonFactory.html#createParser(java.lang.String)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2347) Use Java8 Map API for SpecificData
[ https://issues.apache.org/jira/browse/AVRO-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2347. - Resolution: Fixed Fix Version/s: 1.9.0 > Use Java8 Map API for SpecificData > -- > > Key: AVRO-2347 > URL: https://issues.apache.org/jira/browse/AVRO-2347 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2349) Review Usage of Arrays.asList
[
https://issues.apache.org/jira/browse/AVRO-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2349.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Review Usage of Arrays.asList
> -
>
> Key: AVRO-2349
> URL: https://issues.apache.org/jira/browse/AVRO-2349
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.9.0
>
>
> The method {{Arrays.asList}} is used in several different places in the
> project and is used differently each place. I have reviewed all uses of it
> and made it consistent and removed it in a couple of places that would be
> better implemented with a different approach.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2350) Use Switch Statement for Strings in LogicalTypes
[
https://issues.apache.org/jira/browse/AVRO-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2350.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Use Switch Statement for Strings in LogicalTypes
>
>
> Key: AVRO-2350
> URL: https://issues.apache.org/jira/browse/AVRO-2350
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.9.0
>
>
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L59-L101
> {quote}
> The switch statement compares the String object in its expression with the
> expressions associated with each case label as if it were using the
> String.equals method; consequently, the comparison of String objects in
> switch statements is case sensitive. The Java compiler generates generally
> more efficient bytecode from switch statements that use String objects than
> from chained if-then-else statements.
> https://docs.oracle.com/javase/7/docs/technotes/guides/language/strings-switch.html
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2351) Long/Int in Binary Decoder
[
https://issues.apache.org/jira/browse/AVRO-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2351.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Long/Int in Binary Decoder
> --
>
> Key: AVRO-2351
> URL: https://issues.apache.org/jira/browse/AVRO-2351
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.9.0
>
>
> {code:java|title=BinaryDecoder.java}
> /**
>* Returns the number of items to follow in the current array or map.
> Returns
>* 0 if there are no more items in the current array and the array/map has
>* ended.
>*
>* @throws IOException
>*/
> protected long doReadItemCount() throws IOException {
> long result = readLong();
> if (result < 0) {
> readLong(); // Consume byte-count if present
> result = -result;
> }
> return result;
> }
> /**
>* Reads the count of items in the current array or map and skip those
> items,
>* if possible. If it could skip the items, keep repeating until there are
> no
>* more items left in the array or map. If items cannot be skipped (because
>* byte count to skip is not found in the stream) return the count of the
>* items found. The client needs to skip the items individually.
>*
>* @return Zero if there are no more items to skip and end of array/map is
>* reached. Positive number if some items are found that cannot be
>* skipped and the client needs to skip them individually.
>* @throws IOException
>*/
> private long doSkipItems() throws IOException {
> long result = readInt();
> while (result < 0) {
> long bytecount = readLong();
> doSkipBytes(bytecount);
> result = readInt();
> }
> return result;
> }
> {code}
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java#L370-L406
> The spec states that the sizes are {{long}} values.The {{doReadItemCount}}
> method has it correct and the {{doSkipItems}} method uses {{int}} values
> values.
> https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2352) BinaryDecoder isEnd
[ https://issues.apache.org/jira/browse/AVRO-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2352. - Resolution: Fixed Fix Version/s: 1.9.0 > BinaryDecoder isEnd > --- > > Key: AVRO-2352 > URL: https://issues.apache.org/jira/browse/AVRO-2352 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Fix For: 1.9.0 > > > * Add additional JavaDoc > * Simplify code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2340) Use JDK compareTo and hashcode Methods
[
https://issues.apache.org/jira/browse/AVRO-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2340.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Use JDK compareTo and hashcode Methods
> --
>
> Key: AVRO-2340
> URL: https://issues.apache.org/jira/browse/AVRO-2340
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.9.0
>
>
> There are a few different ways that {{hashCode()}} and {{compare}} are
> implemented. Just use the JDK methods.
> I set a breakpoint and ran the 'perf' program, but it did not hit, so these
> methods are not on the critical paths. Best to do this work in a consistent
> behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AVRO-2299) Get Plain Schema
[
https://issues.apache.org/jira/browse/AVRO-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-2299:
---
Assignee: Rumeshkrishnan
> Get Plain Schema
>
>
> Key: AVRO-2299
> URL: https://issues.apache.org/jira/browse/AVRO-2299
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.8.2
>Reporter: Rumeshkrishnan
>Assignee: Rumeshkrishnan
>Priority: Minor
> Labels: features
>
> {panel:title=Avro Schema Reserved Keys:}
> "doc", "fields", "items", "name", "namespace",
> "size", "symbols", "values", "type", "aliases", "default"
> {panel}
> AVRO also supports user defined properties for both Schema and Field.
> Is there way to get the schema with reserved property (key, value)?
> Input Schema:
> {code:java}
> {
> "name": "testSchema",
> "namespace": "com.avro",
> "type": "record",
> "fields": [
> {
> "name": "email",
> "type": "string",
> "doc": "email id",
> "user_field_prop": "x"
> }
> ],
> "user_schema_prop": "xx"
> }{code}
> Expected Plain Schema:
> {code:java}
> {
> "name": "testSchema",
> "namespace": "com.avro",
> "type": "record",
> "fields": [
> {
> "name": "email",
> "type": "string",
> "doc": "email id"
> }
> ]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2338) Remove Deprecated Use of JsonStringEncoder
[
https://issues.apache.org/jira/browse/AVRO-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2338.
-
Resolution: Fixed
Fix Version/s: 1.9.0
> Remove Deprecated Use of JsonStringEncoder
> --
>
> Key: AVRO-2338
> URL: https://issues.apache.org/jira/browse/AVRO-2338
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.9.0
>
>
> {code:java|title=SchemaBuilder.java}
> char[] quoted = JsonStringEncoder.getInstance().quoteAsString(s);
> {code}
> [Deprecated. Since 2.9.2 use BufferRecyclers.getJsonStringEncoder()
> instead|https://fasterxml.github.io/jackson-core/javadoc/2.9/com/fasterxml/jackson/core/io/JsonStringEncoder.html#getInstance--]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2339) Remove Unused LengthLimitedInputStream
[ https://issues.apache.org/jira/browse/AVRO-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2339. - Resolution: Fixed Fix Version/s: 1.9.0 > Remove Unused LengthLimitedInputStream > -- > > Key: AVRO-2339 > URL: https://issues.apache.org/jira/browse/AVRO-2339 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 1.9.0 > > > Package scoped class and it is not used at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2334) Update Docs to Reference Java 8 API
[ https://issues.apache.org/jira/browse/AVRO-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2334. - Resolution: Fixed Fix Version/s: 1.9.0 > Update Docs to Reference Java 8 API > --- > > Key: AVRO-2334 > URL: https://issues.apache.org/jira/browse/AVRO-2334 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Fix For: 1.9.0 > > > https://github.com/apache/avro/blob/master/lang/java/pom.xml#L113-L114 > > [https://github.com/apache/avro/blob/master/lang/java/pom.xml#L193] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2319) Bump grpc.version from 1.17.1 to 1.18.0
[ https://issues.apache.org/jira/browse/AVRO-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2319. - Resolution: Fixed > Bump grpc.version from 1.17.1 to 1.18.0 > --- > > Key: AVRO-2319 > URL: https://issues.apache.org/jira/browse/AVRO-2319 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.2 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2192) Remove paranamer dependency from Avro
[ https://issues.apache.org/jira/browse/AVRO-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2192: Hadoop Flags: Incompatible change > Remove paranamer dependency from Avro > - > > Key: AVRO-2192 > URL: https://issues.apache.org/jira/browse/AVRO-2192 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.2 >Reporter: RAJKUMAR NATARAJAN >Assignee: Ismaël Mejía >Priority: Minor > Fix For: 1.9.0 > > > Hi, > > Avro seems to be depend upon the com.thoughtworks.paranamer. Avro 1.7.7 > depends upon 2.7 and 1.8. depends on 2.8 paranamer versions. > > Please remove this dependency in the future versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2254) Unions with 2 records declared downward fail
[
https://issues.apache.org/jira/browse/AVRO-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685032#comment-16685032
]
Nandor Kollar commented on AVRO-2254:
-
[~zolyfarkas] the fix looks good to me, I can merge the PR if you open one, and
all test pass.
> Unions with 2 records declared downward fail
>
>
> Key: AVRO-2254
> URL: https://issues.apache.org/jira/browse/AVRO-2254
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Zoltan Farkas
>Priority: Major
>
> The following IDL will fail complaining that 2 same type is declared twice in
> the union:
> {code}
> @namespace("org.apache.avro.gen")
> protocol UnionFwd {
> record TestRecord {
> union {SR1, SR2} unionField;
> }
> record SR1 {
> string field;
> }
> record SR2 {
> string field;
> }
> }
> {code}
> the fix for this can be pretty simple:
> https://github.com/zolyfarkas/avro/commit/56b215f73f34cc80d505875c90217916b271abb5
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AVRO-2254) Unions with 2 records declared downward fail
[
https://issues.apache.org/jira/browse/AVRO-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685032#comment-16685032
]
Nandor Kollar edited comment on AVRO-2254 at 11/13/18 10:54 AM:
[~zolyfarkas] the fix looks good to me, I can merge the PR if you open one, and
every test passes.
was (Author: nkollar):
[~zolyfarkas] the fix looks good to me, I can merge the PR if you open one, and
all test pass.
> Unions with 2 records declared downward fail
>
>
> Key: AVRO-2254
> URL: https://issues.apache.org/jira/browse/AVRO-2254
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Zoltan Farkas
>Priority: Major
>
> The following IDL will fail complaining that 2 same type is declared twice in
> the union:
> {code}
> @namespace("org.apache.avro.gen")
> protocol UnionFwd {
> record TestRecord {
> union {SR1, SR2} unionField;
> }
> record SR1 {
> string field;
> }
> record SR2 {
> string field;
> }
> }
> {code}
> the fix for this can be pretty simple:
> https://github.com/zolyfarkas/avro/commit/56b215f73f34cc80d505875c90217916b271abb5
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2264) Bump Apache commons-lang3 to 3.8.1
[ https://issues.apache.org/jira/browse/AVRO-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-2264. - Resolution: Fixed > Bump Apache commons-lang3 to 3.8.1 > -- > > Key: AVRO-2264 > URL: https://issues.apache.org/jira/browse/AVRO-2264 > Project: Apache Avro > Issue Type: Task > Components: java >Affects Versions: 1.8.2 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2264) Bump Apache commons-lang3 to 3.8.1
[ https://issues.apache.org/jira/browse/AVRO-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2264: Fix Version/s: 1.9.0 > Bump Apache commons-lang3 to 3.8.1 > -- > > Key: AVRO-2264 > URL: https://issues.apache.org/jira/browse/AVRO-2264 > Project: Apache Avro > Issue Type: Task > Components: java >Affects Versions: 1.8.2 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2254) Unions with 2 records declared downward fail
[
https://issues.apache.org/jira/browse/AVRO-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681192#comment-16681192
]
Nandor Kollar commented on AVRO-2254:
-
[~zolyfarkas] looks like a bug, could you please create a PR with the fix?
> Unions with 2 records declared downward fail
>
>
> Key: AVRO-2254
> URL: https://issues.apache.org/jira/browse/AVRO-2254
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Zoltan Farkas
>Priority: Major
>
> The following IDL will fail complaining that 2 same type is declared twice in
> the union:
> {code}
> @namespace("org.apache.avro.gen")
> protocol UnionFwd {
> record TestRecord {
> union {SR1, SR2} unionField;
> }
> record SR1 {
> string field;
> }
> record SR2 {
> string field;
> }
> }
> {code}
> the fix for this can be pretty simple:
> https://github.com/zolyfarkas/avro/commit/56b215f73f34cc80d505875c90217916b271abb5
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2245) Java codec testing improvements
[
https://issues.apache.org/jira/browse/AVRO-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2245:
Fix Version/s: 1.9.0
> Java codec testing improvements
> ---
>
> Key: AVRO-2245
> URL: https://issues.apache.org/jira/browse/AVRO-2245
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Jacob Tolar
>Priority: Minor
> Fix For: 1.9.0
>
>
> In the Avro Java implementation, TestBZip2Codec and TestZstandardCodec are
> both laughably wrong.
> For example, the last lines of TestBZip2Codec:
> {code:java}
> byte[] outputByteArray = decompressedBuffer.array();
> for (int i = 0; i < inputByteSize; i++) {
> inputByteArray[i] = outputByteArray[i];
> }
> {code}
> There should be an assertEquals in there not an assignment statement. (And if
> you put assertEquals there, the test actually fails...).
> I will send a PR that replaces these tests with a correct parametrized test
> for all codecs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AVRO-2245) Java codec testing improvements
[
https://issues.apache.org/jira/browse/AVRO-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-2245:
---
Assignee: Jacob Tolar
> Java codec testing improvements
> ---
>
> Key: AVRO-2245
> URL: https://issues.apache.org/jira/browse/AVRO-2245
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Jacob Tolar
>Assignee: Jacob Tolar
>Priority: Minor
> Fix For: 1.9.0
>
>
> In the Avro Java implementation, TestBZip2Codec and TestZstandardCodec are
> both laughably wrong.
> For example, the last lines of TestBZip2Codec:
> {code:java}
> byte[] outputByteArray = decompressedBuffer.array();
> for (int i = 0; i < inputByteSize; i++) {
> inputByteArray[i] = outputByteArray[i];
> }
> {code}
> There should be an assertEquals in there not an assignment statement. (And if
> you put assertEquals there, the test actually fails...).
> I will send a PR that replaces these tests with a correct parametrized test
> for all codecs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2245) Java codec testing improvements
[
https://issues.apache.org/jira/browse/AVRO-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2245.
-
Resolution: Fixed
> Java codec testing improvements
> ---
>
> Key: AVRO-2245
> URL: https://issues.apache.org/jira/browse/AVRO-2245
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Jacob Tolar
>Assignee: Jacob Tolar
>Priority: Minor
> Fix For: 1.9.0
>
>
> In the Avro Java implementation, TestBZip2Codec and TestZstandardCodec are
> both laughably wrong.
> For example, the last lines of TestBZip2Codec:
> {code:java}
> byte[] outputByteArray = decompressedBuffer.array();
> for (int i = 0; i < inputByteSize; i++) {
> inputByteArray[i] = outputByteArray[i];
> }
> {code}
> There should be an assertEquals in there not an assignment statement. (And if
> you put assertEquals there, the test actually fails...).
> I will send a PR that replaces these tests with a correct parametrized test
> for all codecs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1126) Upgrade to Jackson 2+
[
https://issues.apache.org/jira/browse/AVRO-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-1126.
-
Resolution: Fixed
Hadoop Flags: Incompatible change
> Upgrade to Jackson 2+
> -
>
> Key: AVRO-1126
> URL: https://issues.apache.org/jira/browse/AVRO-1126
> Project: Avro
> Issue Type: Task
> Components: java
>Reporter: James Tyrrell
>Assignee: Charles Honton
>Priority: Critical
> Fix For: 1.9.0
>
>
> Quite annoyingly with Jackson 2+ the base package name has changed from
> org.codehaus.jackson to com.fasterxml.jackson so in addition to changing the
> dependencies from:
> {code:xml}
>
> org.codehaus.jackson
> jackson-core-asl
> ${jackson.version}
>
>
> org.codehaus.jackson
> jackson-mapper-asl
> ${jackson.version}
>
> {code}
> to:
> {code:xml}
>
> com.fasterxml.jackson.core
> jackson-core
> ${jackson.version}
>
>
> com.fasterxml.jackson.core
> jackson-databind
> ${jackson.version}
>
> {code}
> the base package in the code needs to be updated. More info can be found
> [here|http://wiki.fasterxml.com/JacksonUpgradeFrom19To20], I am happy to do
> the work just let me know what is preferable i.e. should I just attach a
> patch to this issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1605) Remove Jackson classes from public API
[ https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-1605: Resolution: Fixed Hadoop Flags: Incompatible change Status: Resolved (was: Patch Available) > Remove Jackson classes from public API > -- > > Key: AVRO-1605 > URL: https://issues.apache.org/jira/browse/AVRO-1605 > Project: Avro > Issue Type: Sub-task > Components: java >Affects Versions: 1.7.8 >Reporter: Tom White >Assignee: Gabor Szadovszky >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AVRO-2250) Release 1.9.0
Nandor Kollar created AVRO-2250: --- Summary: Release 1.9.0 Key: AVRO-2250 URL: https://issues.apache.org/jira/browse/AVRO-2250 Project: Avro Issue Type: Task Reporter: Nandor Kollar -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2120) NullPointerException thrown by Schema.Parser#parse(literal)
[ https://issues.apache.org/jira/browse/AVRO-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2120: Issue Type: Bug (was: Improvement) > NullPointerException thrown by Schema.Parser#parse(literal) > --- > > Key: AVRO-2120 > URL: https://issues.apache.org/jira/browse/AVRO-2120 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Sebastien Dubois >Assignee: Niels Basjes >Priority: Major > Fix For: 1.9.0, 1.8.3 > > > Calling the parse method with an invalid input (e.g., "") instead of a valid > schema throws a NullPointerException. > Expected behavior: SchemaParseException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-2079:
---
Assignee: Paul Vorbach
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Assignee: Paul Vorbach
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2079.
-
Resolution: Fixed
Hadoop Flags: Reviewed
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Assignee: Paul Vorbach
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2079:
Fix Version/s: 1.9.0
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2195) Add Zstandard Codec
[ https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561622#comment-16561622 ] Nandor Kollar commented on AVRO-2195: - Committed to master. Thanks [~benson.qiu] for the contribution, and [~nielsbasjes] for helping to review! > Add Zstandard Codec > --- > > Key: AVRO-2195 > URL: https://issues.apache.org/jira/browse/AVRO-2195 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.0 >Reporter: Benson Qiu >Assignee: Benson Qiu >Priority: Major > Labels: patch > Fix For: 1.9.0 > > Attachments: AVRO-2195.patch, AVRO-2195.patch.v2, AVRO-2195.v3.patch, > AVRO-2195.v4.patch > > > Inspired by AVRO-1373. The Zstandard algorithm is available in the > commons-library, which Avro projects already depend on. > In a quick test that I did, Zstandard had a better compression ratio than > deflate (compression level 9). > [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2195) Add Zstandard Codec
[ https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2195: Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Add Zstandard Codec > --- > > Key: AVRO-2195 > URL: https://issues.apache.org/jira/browse/AVRO-2195 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.0 >Reporter: Benson Qiu >Assignee: Benson Qiu >Priority: Major > Labels: patch > Fix For: 1.9.0 > > Attachments: AVRO-2195.patch, AVRO-2195.patch.v2, AVRO-2195.v3.patch, > AVRO-2195.v4.patch > > > Inspired by AVRO-1373. The Zstandard algorithm is available in the > commons-library, which Avro projects already depend on. > In a quick test that I did, Zstandard had a better compression ratio than > deflate (compression level 9). > [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2195) Add Zstandard Codec
[ https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2195: --- Assignee: Benson Qiu > Add Zstandard Codec > --- > > Key: AVRO-2195 > URL: https://issues.apache.org/jira/browse/AVRO-2195 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.0 >Reporter: Benson Qiu >Assignee: Benson Qiu >Priority: Major > Labels: patch > Fix For: 1.9.0 > > Attachments: AVRO-2195.patch, AVRO-2195.patch.v2, AVRO-2195.v3.patch, > AVRO-2195.v4.patch > > > Inspired by AVRO-1373. The Zstandard algorithm is available in the > commons-library, which Avro projects already depend on. > In a quick test that I did, Zstandard had a better compression ratio than > deflate (compression level 9). > [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2164) Make Decimal a first class type.
[ https://issues.apache.org/jira/browse/AVRO-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550713#comment-16550713 ] Nandor Kollar commented on AVRO-2164: - [~BigAndy] I wanted to refer to this in the Decimal section of the spec: "For the purposes of schema resolution, two schemas that are decimal logical types _match_ if their scales and precisions match." This sounds like a rule for schema evolution, and scale and precision should match (the current doesn't check for this). > Make Decimal a first class type. > > > Key: AVRO-2164 > URL: https://issues.apache.org/jira/browse/AVRO-2164 > Project: Avro > Issue Type: Improvement > Components: logical types >Affects Versions: 1.8.2 >Reporter: Andy Coates >Priority: Major > > I'd be interested to hear the communities thoughts on making decimal a first > class type. > The current logical type encodes a decimal into a _bytes_ or _fixed_. This > encoding does not include any information about the scale, i.e. this encoding > is lossy. > There are open issues around the compatibility / evolvability of schemas > containing decimal logical types, (e.g. AVRO-2078 & AVRO-1721), that mean > reading data that was previously written with a different scale will result > in data corruption. > If these issues were fixed, with suitable compatibility checks put in place, > this would then make it impossible to evolve an Avro schema where the scale > needs to be changed. This inability to evolve the scale is very restrictive, > and can result in high overhead for organizations that _need_ to change the > scale, i.e. they may potentially need to copy their entire data set, > deserializing with the old scale and re-serializing with the new. > If _decimal_ were promoted to a first class type, this would allow the scale > to be captured in the serialized form, allow for schema evolution support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2146) getting Expected start-union. Got VALUE_STRING
[
https://issues.apache.org/jira/browse/AVRO-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550702#comment-16550702
]
Nandor Kollar commented on AVRO-2146:
-
[~fhuertas] you should specify in the Json data which union branch to use. Your
Json should look like this:
{code}
{
"id" : {
"int" : 141
}
}
{code}
See details in [doc|https://avro.apache.org/docs/1.8.2/spec.html#json_encoding].
> getting Expected start-union. Got VALUE_STRING
> --
>
> Key: AVRO-2146
> URL: https://issues.apache.org/jira/browse/AVRO-2146
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.2
> Environment: error message:
> Exception in thread "main" org.apache.avro.AvroTypeException: Expected
> start-union. Got VALUE_STRING
> at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:698)
> at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:441)
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:290)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:267)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
> at myJson2Avro.fromJasonToAvro(myJson2Avro.java:81)
> at myJson2Avro.main(myJson2Avro.java:48)
>Reporter: laki
>Priority: Major
>
> Here is the schema, no unions, but getting union error :
>
> {
> "type" : "record",
> "name" : "edm_generic_publisher_avro_schema",
> "namespace" : "edm.avro",
> "doc" : "The generic avro schema used by publishers to publish events to the
> enterprise streaming service",
> "fields" : [
> {"name" : "event",
> "type" : {
> "type" : "record",
> "name" : "event_meta_data",
> "fields" : [
> {"name" : "event_name", "type" : "string", "doc" : "The name of the event. In
> the CDC, this field is populated with the name of the data base table or
> segment."}
> ,
> {"name" : "operation_type", "type" : "string", "doc": "The operation or
> action that triggered the event. e.g., Insert, Update, Delete, etc."}
> ,
> {"name" : "transaction_identifier", "type" : "string", "default" : "NONE",
> "doc" : "A unique identifier that identifies a unit or work or transaction.
> Useful in relating multiple events together."}
> ,
> {"name" : "event_publication_timestamp_millis", "type" : "string", "doc":
> "timestamp when the event was published"}
> ,
>
> {"name" : "event_publisher", "type" : "string", "doc" : "The system or
> application that published the event"}
> ,
>
> {"name" : "event_publisher_identity", "type": "string", "default" : "NONE",
> "doc": "The identity (user) of the system or application that published the
> event"}
> ,
>
> {"name" : "event_timestamp_millis", "type" : "string", "default" : "NONE",
> "doc": "timestamp when the event occured"}
> ,
>
> {"name": "event_initiator", "type": "string", "default" : "NONE", "doc" :
> "The system or application that initiated the event"}
> ,
> {"name": "event_initiator_identity", "type" : "string", "default" : "NONE",
> "doc": "The system id or application id that initiated the event" }
> ]},
> "doc" : "The data about the published event"
> },
> { "name" : "contents",
> "type" : {
> "name": "data_field_groups",
> "type": "array",
> "items": {
> "type": "record",
> "name": "data_field_group",
> "fields" : [
> {"name": "data_group_name", "type": "string" }
> ,
> {
> "name": "data_fields",
> "type": {
> "type": "array",
> "items": {
> "name": "data_field",
> "type": "record",
> "fields":[
> {"name" : "data_field_name", "type" : "string", "doc" : "The field name"}
> ,
>
> {"name": "data_field_type", "type": "string", "doc" : "The data type is one
> of the following values: string, boolean, int, long, float, double or bytes"}
> ,
> {"name" : "data_field_value", "type" : ["string"], "doc" : "The value"}
> ]
> }
> }
[jira] [Commented] (AVRO-2195) Add Zstandard Codec
[ https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547902#comment-16547902 ] Nandor Kollar commented on AVRO-2195: - +1 for AVRO-2195.v4.patch > Add Zstandard Codec > --- > > Key: AVRO-2195 > URL: https://issues.apache.org/jira/browse/AVRO-2195 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.0 >Reporter: Benson Qiu >Priority: Major > Labels: patch > Attachments: AVRO-2195.patch, AVRO-2195.patch.v2, AVRO-2195.v3.patch, > AVRO-2195.v4.patch > > > Inspired by AVRO-1373. The Zstandard algorithm is available in the > commons-library, which Avro projects already depend on. > In a quick test that I did, Zstandard had a better compression ratio than > deflate (compression level 9). > [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2195) Add Zstandard Codec
[
https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533923#comment-16533923
]
Nandor Kollar commented on AVRO-2195:
-
Patch looks good to me, however the {{com.github.luben:zstd-jni}} has BSD
license, and I'm not sure if this is could be a problem or not. [~nielsbasjes]
what do you think? Do you have any objection against committing this?
> Add Zstandard Codec
> ---
>
> Key: AVRO-2195
> URL: https://issues.apache.org/jira/browse/AVRO-2195
> Project: Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.9.0
>Reporter: Benson Qiu
>Priority: Major
> Labels: patch
> Attachments: AVRO-2195.patch, AVRO-2195.patch.v2
>
>
> Inspired by AVRO-1373. The Zstandard algorithm is available in the
> commons-library, which Avro projects already depend on.
> In a quick test that I did, Zstandard had a better compression ratio than
> deflate (compression level 9).
> [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2195) Add Zstandard Codec
[
https://issues.apache.org/jira/browse/AVRO-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529553#comment-16529553
]
Nandor Kollar commented on AVRO-2195:
-
Avro has several parametrized test case with codecs as parameters, how about
adding Zstandard as an additional parameter to those tests? \{{TestDataFile}}
is a good example for it.
> Add Zstandard Codec
> ---
>
> Key: AVRO-2195
> URL: https://issues.apache.org/jira/browse/AVRO-2195
> Project: Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.9.0
>Reporter: Benson Qiu
>Priority: Major
> Labels: patch
> Attachments: AVRO-2195.patch
>
>
> Inspired by AVRO-1373. The Zstandard algorithm is available in the
> commons-library, which Avro projects already depend on.
> In a quick test that I did, Zstandard had a better compression ratio than
> deflate (compression level 9).
> [https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2173) remove CHANGES.txt
[ https://issues.apache.org/jira/browse/AVRO-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527608#comment-16527608 ] Nandor Kollar commented on AVRO-2173: - [~nielsbasjes] fine for me, let's drop CHANGES.txt from branch-1.8 too. > remove CHANGES.txt > -- > > Key: AVRO-2173 > URL: https://issues.apache.org/jira/browse/AVRO-2173 > Project: Avro > Issue Type: Improvement >Reporter: Doug Cutting >Priority: Major > > The CHANGES.txt file is not well maintained and redundant with information in > Jira. > Let's remove this file, and instead generate release notes from Jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2164) Make Decimal a first class type.
[ https://issues.apache.org/jira/browse/AVRO-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525022#comment-16525022 ] Nandor Kollar commented on AVRO-2164: - As of schema evolution, Avro [spec|https://avro.apache.org/docs/1.8.2/spec.html#Decimal] has a rule for decimal evolution: the scale and precision should not change. Like Ryan mentioned this can result in wrong interpretation of data, therefore in my opinion the code should enforce this rule (see AVRO-2078). However spec doesn't have any evolution rule for other types. What happens if the millisecond precision timestamp is promoted to microsecond precision? Should Avro care about it? What happens to the other way? What happens if reader tries to read a time field as date? > Make Decimal a first class type. > > > Key: AVRO-2164 > URL: https://issues.apache.org/jira/browse/AVRO-2164 > Project: Avro > Issue Type: Improvement > Components: logical types >Affects Versions: 1.8.2 >Reporter: Andy Coates >Priority: Major > > I'd be interested to hear the communities thoughts on making decimal a first > class type. > The current logical type encodes a decimal into a _bytes_ or _fixed_. This > encoding does not include any information about the scale, i.e. this encoding > is lossy. > There are open issues around the compatibility / evolvability of schemas > containing decimal logical types, (e.g. AVRO-2078 & AVRO-1721), that mean > reading data that was previously written with a different scale will result > in data corruption. > If these issues were fixed, with suitable compatibility checks put in place, > this would then make it impossible to evolve an Avro schema where the scale > needs to be changed. This inability to evolve the scale is very restrictive, > and can result in high overhead for organizations that _need_ to change the > scale, i.e. they may potentially need to copy their entire data set, > deserializing with the old scale and re-serializing with the new. > If _decimal_ were promoted to a first class type, this would allow the scale > to be captured in the serialized form, allow for schema evolution support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2184) Unable to decode JSON data file if a property is renamed in reader schema
[
https://issues.apache.org/jira/browse/AVRO-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2184:
Assignee: Nandor Kollar
Status: Patch Available (was: Open)
> Unable to decode JSON data file if a property is renamed in reader schema
> -
>
> Key: AVRO-2184
> URL: https://issues.apache.org/jira/browse/AVRO-2184
> Project: Avro
> Issue Type: Bug
>Reporter: Prateek Kohli
>Assignee: Nandor Kollar
>Priority: Major
> Attachments: TestAliasesInSchemaEvolution.java
>
>
> I am unable to decode JSON data file if a property is renamed in reader
> schema:
> As per the documentation it is a compatible change.
> Also, Datatype promotion is not being supported, if I try to change the
> datatype of favourite_number field in the writer's schema, decoding fails.
> All of the above scenarios are supported if I use Binary decoding instead of
> JSON.
> *Writer Schema :*
> {"namespace": "example.avro",
> "type": "record",
> "name": "User",
> "fields": [
> \{"name": "name", "type": "string"},
> \{"name": "favorite_number", "type": ["int", "null"]},
> \{"name": "favorite_color", "type": ["string", "null"]}
> ]}
>
> *Reader Schema :*
> {"namespace": "example.avro",
> "type": "record",
> "name": "User",
> "fields": [
> \{"name": "fname", "type": "string", "aliases" : [ "name" ]},
> \{"name": "favorite_number", "type": ["int", "null"]},
> \{"name": "favorite_color", "type": ["string", "null"]}
> ]}
>
> *I have written the below code to decode JSON data:*
> FileInputStream fin = new FileInputStream(file);
> byte fileContent[] = new byte[(int)file.length()];
> fin.read(fileContent);
> InputStream input = new ByteArrayInputStream(fileContent);
> DataInputStream din = new DataInputStream(input);
>
> while (true) {
> try {
> Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
> ResolvingDecoder resolvingDecoder =
> DecoderFactory.get().resolvingDecoder(writer,
> reader, decoder);
> Object datum = datumReader.read(null, resolvingDecoder);
> System.out.println(datum);
> } catch (EOFException eofException) {
> break;
> }
> }
> *Below is the Exception I get :*
> Exception in thread "main" org.apache.avro.AvroTypeException: Found
> example.avro.User, expecting example.avro.User, missing required field fname
> at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:292)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:196)
> at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
> at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
> at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
> at
> com.ericsson.avroTest.avroCheck.WithoutCodeTest.main(WithoutCodeTest.java:134)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2189) Missing closing quotation in SchemaBuilder docstring
[
https://issues.apache.org/jira/browse/AVRO-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16513711#comment-16513711
]
Nandor Kollar commented on AVRO-2189:
-
Merged to master, thanks for fixing this!
> Missing closing quotation in SchemaBuilder docstring
>
>
> Key: AVRO-2189
> URL: https://issues.apache.org/jira/browse/AVRO-2189
> Project: Avro
> Issue Type: Bug
>Reporter: Austin Hacker
>Assignee: Austin Hacker
>Priority: Trivial
> Fix For: 1.9.0
>
>
> org.apache.avro.SchemaBuilder has a typo in its docstring.
> {noformat}
> .namespace("org.apache.avro.ipc){noformat}
> should be
> {noformat}
> .namespace("org.apache.avro.ipc"){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
