[jira] [Commented] (AVRO-1847) IDL compiler should use BigDecimal to represent decimal logical type.
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952612#comment-15952612 ] Yibing Shi commented on AVRO-1847: -- [~vijayp...@gmail.com], I have no idea about the release date of 1.8.2. Maybe [~rdblue] or [~busbey] can answer your question. > IDL compiler should use BigDecimal to represent decimal logical type. > - > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Fix For: 1.9.0, 1.8.2 > > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch, > AVRO-1847.8.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[
https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713643#comment-15713643
]
Yibing Shi commented on AVRO-1961:
--
{quote}
Actually, on the setter/builder/constructor, we could even go a step further: a
required field with a default value could be with Optional as well. The
matching getter would be without Optional. It would make the concept of default
value type safe just as well. Sweet isn't it?
{quote}
And Optional object indicates that the value could be null. It has a different
meaning than having a default value. That is, you can skip passing in a value
for this field doesn't mean you can pass in a null value for this field.
Suggest not to apply this to fields with default values.
{quote}
1. Java 8 isn't required for Avro project for this to work
All the "magic" happen uniquely through the template: the code could still
compile with Java 6, but anyone using the "pojoWithOptional" flag should
consume the generated pojo with Java 8 or backport Optional.
{quote}
Is Java 8 is needed to test the pojoWithOptionalGenerationTest you are going to
add?
> [JAVA] Generate getters that return an Optional
> ---
>
> Key: AVRO-1961
> URL: https://issues.apache.org/jira/browse/AVRO-1961
> Project: Avro
> Issue Type: New Feature
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
>
> A colleague of mine indicated that having getters that return an Optional
> (java 8 thingy) would be very useful for him.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1962) Support UUID logical type
[
https://issues.apache.org/jira/browse/AVRO-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688400#comment-15688400
]
Yibing Shi commented on AVRO-1962:
--
[~xiongtx], sorry for the misleading info in mail list! UUID type can be
specified like below:
{code}
{
"type": "string",
"logicalType": "uuid"
}
{code}
> Support UUID logical type
> -
>
> Key: AVRO-1962
> URL: https://issues.apache.org/jira/browse/AVRO-1962
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Tianxiang Xiong
>
> The AVRO-1554 ticket seems to suggest that the issue of representing UUIDs is
> resolved with [logical
> types|http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types] in Avro
> 1.8.1. However, there is [no UUID logical type in Avro
> 1.8.1|https://github.com/apache/avro/blob/release-1.8.1/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj#L214-L244].
> The specification offers several examples of using logical types; decimals
> are represented as:
> {code}
> {
> "type": "bytes",
> "logicalType": "decimal",
> "precision": 4,
> "scale": 2
> }
> {code}
> No examples for UUID are offered, presumably because UUIDs are not supported.
> Thanks to [~Yibing]'s confirmation on the mailing list that this is the case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional
[ https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688256#comment-15688256 ] Yibing Shi commented on AVRO-1961: -- [~nielsbasjes], would you please provide more details of this request? Such as how the Optional type should be returned and what is the benefit of it. > [JAVA] Generate getters that return an Optional > --- > > Key: AVRO-1961 > URL: https://issues.apache.org/jira/browse/AVRO-1961 > Project: Avro > Issue Type: New Feature >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Minor > > A colleague of mine indicated that having getters that return an Optional > (java 8 thingy) would be very useful for him. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1927) If a default value is set, Avro allows null values in non-nullable fields.
[
https://issues.apache.org/jira/browse/AVRO-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522012#comment-15522012
]
Yibing Shi commented on AVRO-1927:
--
[~asmaier], currently the generated code doesn't check whether we are setting a
null value for a non nullable column. In your case, null is a valid Java String
value and thus it works. However, I believe AVRO would throw an exception when
you try to write your object. Have you tried that? Does it fail or succeed?
{code}
ByteArrayOutputStream os = new ByteArrayOutputStream(1024);
Encoder encoder = EncoderFactory.get().binaryEncoder(os, null);
SpecificDatumWriter writer = new
SpecificDatumWriter(MyObject.getClassSchema());
writer.write(myObject, encoder);
{code}
> If a default value is set, Avro allows null values in non-nullable fields.
> --
>
> Key: AVRO-1927
> URL: https://issues.apache.org/jira/browse/AVRO-1927
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: Andreas Maier
> Labels: newbie
>
> With an avro schema like
> {code}
> {
> "name": "myfield",
> "type": "string",
> "default": ""
> }
> {code}
> the following code should throw an exception
> {code}
> MyObject myObject = MyObject.newBuilder().setMyfield(null).build();
> {code}
> But instead the value of myfield is set to null, which causes an exception
> later when serializing myObject, because null is not a valid value for
> myfield.
> I believe in this case setMyfield(null) should throw an exception,
> independent of the value of default.
> See also
> https://stackoverflow.com/questions/38509279/generated-avro-builder-set-null-doesnt-overwrite-with-default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15466040#comment-15466040
]
Yibing Shi commented on AVRO-1891:
--
The patch looks great, [~rdblue]!
I add a few comments to it.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15463713#comment-15463713
]
Yibing Shi commented on AVRO-1891:
--
[~rdblue], you idea looks good to me!
{quote}
Maintain a thread-local reference to the current specific record class in
GenericDatumReader
{quote}
I am also thinking about using a thread local variable, because my patch makes
data reader/writer non thread-safe. We may need to keep a stack of class
references, which will be helpful when there are nested records.
{quote}
# Add conversion lookup methods to GenericDatumReader that delegate to
GenericData
# Override the conversion lookup methods in SpecificDatumReader that use the
current record class's set of conversions instead.
{quote}
This is a great idea! Looking forward to your patch!
(For backward compatibility, the conversion lookup methods in
SpecificDataumReader/Writer may need to fallback to those in
GenericDatumReader/Writer if the generated class doesn't have the static
conversion map -- for legacy classes I mean)
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(Paren
[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.4.patch
Upload a new patch that fixes the defaultValue bug and can be applied to master
branch
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460908#comment-15460908
]
Yibing Shi commented on AVRO-1891:
--
Just found the {{defaultValue}} method I added has a bug. I guess you were
actually mentioning to this bug when you said that my modification was
incompatible. I will fix it. Thanks for pointing this out.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.j
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460903#comment-15460903
]
Yibing Shi commented on AVRO-1891:
--
I have another look into the code, and find that we might have to extract
{{ConversionResolver}}, because:
# {{Generic/SpecificData}} are public classes, and they are not final.
Applications may have extended from them, which makes it hard to create them
(especially their children) on the fly, which is necessary if we want to add
conversions when writing/reading children of {{SpecificRecordBase}}. On the
other hand, {{ConversionResolver}} is a new class, we can define it as final,
which make it easy to create objects.
# creating {{Generic/SpecificData}} object in specific datum reader/writer is
confusing, especially when the reader/writer already has a {{data}} member.
Using {{ConversionResolver}} makes code much clearer.
# Actually we don't have to mark those add/get conversion methods in
{{GenericData}} as deprecated. They are still useful and provide a convenient
way for applications to manage conversions.
What are you opinions on this, [~cutting] and [~tomwhite]?
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(Paren
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460722#comment-15460722
]
Yibing Shi commented on AVRO-1891:
--
{quote}
I don't see the advantage of adding ConversionResolver. We'll be stuck with a
bunch of deprecated methods for a number of releases or we break things
incompatibly. Does it add any expressive power, or is it just done for
cleanliness?
{quote}
The idea of my patch is to extract find out the conversions used in a child of
{{SpecificRecordBase}} and merged them with those added to the
{{Generic/SpecificData}} in datum reader/writer so that the record fields can
be processed properly without users having to manually adding the conversions.
I extracted {{ConversionResolver}} out just to avoid creating heavy
{{Generic/SpecificData}} objects. [~tomwhite] mentioned that we could make
{{ConversionResolver}} non-public. I am thinking about making it an inner class
of GenericData. This way, we might be able to avoid exposing
{{ConversionResolver}} and marking a bunch of methods deprecated.
I will upload another patch later.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.Pa
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460708#comment-15460708
]
Yibing Shi commented on AVRO-1891:
--
{quote}
although this has the effect of creating another class that doesn't really need
to be public
{quote}
I will try to make this class non-public.
{quote}
The modifications to support union look fairly minor. Are any further changes
needed for arrays and maps?
{quote}
We need some changes to support writing records and record fields. The patch
has already covered this. There should be no further change needed for arrays
and maps.
{quote}
It looks like RecordBuilderBase has an incompatible change to defaultValue().
{quote}
I deleted the previous {{defaultValue}} method, thinking it is only needed in
by the generated code and now the generated code doesn't use it any more. I
forgot this was a protected method and there has been a release that exposes
this change. Deleting it would bring a backward compatibility problem. I will
add it back. Thanks for pointing this out.
And sorry for uploading a broken patch. I will try to correct it in next
version.
{quote}
We need a test for multiple nested logical types with generated code, as well
as ones for arrays and maps of logical types. (BTW the v3 patch doesn't apply
to latest master for me.)
{quote}
Yes, I am working on this. I am thinking of changing class
{{TestRecordWithLogicalTypes}} to make it contain more types of nested types.
{quote}
The idea was that you'd always include all known conversions (the same set that
SpecificCompiler uses)
{quote}
Users can pass a parameter to {{SpecificCompiler}} to control whether it uses
{{BidDecimal}} or {{ByteBuffer}} for decimal types. That is, the
{{DecimalConversion}} is not always used in generated code. (This is to keep
backwards compatibility. We may not be able to assume {{DecimalConversion}} is
always used.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.3.patch
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflec
[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.2.patch
Attach another patch that fixes the reading problems
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch,
> AVRO-1891.yshi.2.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.1.patch
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMet
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15457575#comment-15457575
]
Yibing Shi commented on AVRO-1891:
--
{quote}
When resolveUnion is called there is no SpecificRecordBase object to get the
conversions from.
{quote}
We can add another {{resolveUnion}} method that takes an extra parameter that
holds all the conversions, and overwrites the {{resolveUnion}} method in
{{SpecificDatumWriter}}.
I will my patch (not completed yet) as well. Please help have a look. The idea
is to overloads the {{writeRecord}} in {{SepecificDatumWriter}} and utilize the
conversions used in the generated specific record for writing data. A similar
thing is done in {{SepcificDatumReader}} as well.
BTW, if we go with your patch, we will have to check the {{SpecificCompiler}}
parameters to determine whether we should include {{DecimalConversion}} to the
variable Model$.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453755#comment-15453755
]
Yibing Shi commented on AVRO-1891:
--
{quote}
We could do this by adding a constructor to SpecificDatumWriter that takes a
SpecificRecord, so it uses the SpecificData instance from the generated class
(which should have the conversions added to it).
{quote}
# At this moment, only the generated {{SpecificRecord}} contains a
{{SpecificData}} member, and this member is private. {{SpecificRecord}} itself
doesn't provide any method that returns a related {{SpecificData}}. Adding such
a method is not backward compatible.
# We still need to consider the situation where a {{SpecificRecord}} is nested
in another {{SpecificRecord}}, and these 2 records may use different set of
conversions (for example one use decimal conversion while the other uses date
conversion). {{SpecificDatumWriter/Reader}} need to process such a scenario.
Currently I am thinking of moving the conversion management function away from
{{Specific/GenericData}} and put it into another class (say
ConversionResolver), and pass its objects as a parameter to Specific/Generic
datum readers/writers. But I haven't got time to carefully examine this idea.
Will find some time this week to do this.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.jun
[jira] [Commented] (AVRO-1901) creating a record called "Exception" leads to generated code not compiling
[
https://issues.apache.org/jira/browse/AVRO-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432801#comment-15432801
]
Yibing Shi commented on AVRO-1901:
--
[~radai], are you trying to define an error in your IDL? If yes, have you tried
the keyword {{error}} as shown in [this
doc|https://avro.apache.org/docs/1.8.1/idl.html#format_records]?
> creating a record called "Exception" leads to generated code not compiling
> --
>
> Key: AVRO-1901
> URL: https://issues.apache.org/jira/browse/AVRO-1901
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7, 1.8.1, 1.9.0
>Reporter: radai rosenblatt
>
> consider the following *.avsc file:
> {code:title=Exception.avsc}
> {
> "type": "record",
> "name": "Exception",
> "namespace": "com.acme",
> "fields" : [
> {"name": "value", "type": "int"}
> ],
> "doc": "Will not submit to your fascist naming conventions"
> }
> {code}
> the generated java code will contain:
> {code:title=Exception.java}
> ...
> @override
> public Exception build() {
>try {
> ...
> return record;
>} catch (Exception e) { //boom
> throw new org.apache.avro.AvroRuntimeException(e);
>}
> }
> {code}
> which fails to compile because Exception refers to the generated class (which
> does not extend Throwable)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411921#comment-15411921
]
Yibing Shi commented on AVRO-1891:
--
[~rdblue], [~busbey], could you please help confirm whether we need support
embedded complex type in specific records? Such as:
{code}
union { array, map }
{code}
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorI
[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types
[
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410955#comment-15410955
]
Yibing Shi commented on AVRO-1895:
--
The workaround in AVRO-1891 may not help here. DeepCopy hasn't applied any
logical type conversion logic yet.
That said, I suggest we hold on do anything on this till we decide how to
manage the conversions for specific records in AVRO-1891 and AVRO-1894.
> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
> Issue Type: Improvement
> Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(),
> source);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1894) GenericData does not add Logical Type conversions by default
[
https://issues.apache.org/jira/browse/AVRO-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410949#comment-15410949
]
Yibing Shi commented on AVRO-1894:
--
How about add all the default conversions into {{SpecificData}} but not
{{GenericData}}? We have applied a bunch of default conversions to
{{SpecificRecord}} in {{SpecificCompiler}}. Does it make sense the apply the
same set of conversions to {{SpecificData}}?
As for {{GenericData}}, adding a builder class is a good idea. We can cache the
created {{GenericData}} instances to reduce memory usage (reuse the same
instance if a same bunch of conversions are passed in).
> GenericData does not add Logical Type conversions by default
>
>
> Key: AVRO-1894
> URL: https://issues.apache.org/jira/browse/AVRO-1894
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>
> In order to support Decimal fields, the following line needs to be added:
> {code}
> new GenericData().addLogicalTypeConversion(new
> Conversions.DecimalConversion());
> {code}
> It would be more convenient if logical type conversions were added by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410947#comment-15410947
]
Yibing Shi commented on AVRO-1891:
--
Things can become even more complicated. AVRO supports multi-dimensional array,
and any complex type can be included in another complex type. This makes it
necessary for specific compiler to check all the levels of an idl definition
and find out all the used conversions. Currently the conversions are stored on
a per field basis. This may not be suitable for the scenarios mentioned above.
Instead, I propose the store all the conversions (from any level) in a set so
that application can use it when necessary.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTest
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406991#comment-15406991
]
Yibing Shi commented on AVRO-1891:
--
Hi [~rdblue], thanks for the reply!
I agree that we need to modify some methods in SpecificDatumReader/Writer to
use conversions contained in SepcificRecord.
Problem is that currently the {{SpecificCompiler}} doesn't generate conversions
for complex type fields:
https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L647-L672
The complex types themselves don't contain a "logical type". Instead, it is
their element types that can contain a logical type. We may need to modify the
compiler to include these scenarios.
I have a quick thought about this, ARRAY and MAP type are OK. They contain only
one element type. But for UNION, it is possible that there are multiple element
types in it. [The current {{Conversion}}
array|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L647-L672]
in {{SpecificRecord}} cannot hold this. I was considering changing it as
{{java.util.List>[] conversions}}, but looks like
this is a non compatible change. Now I tend to add another field specially for
union fields (need to change {{getConversion}} to determine which field to look
at -- bad for performance}}. What do you think about this?
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4Clas
[jira] [Comment Edited] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405931#comment-15405931
]
Yibing Shi edited comment on AVRO-1891 at 8/3/16 1:52 PM:
--
Had a quick look at this. The scope of this problem is bigger than I expected.
It looks like that current {{SpecificCompiler}} and
{{SpecificDatumWriter/Reader}} haven't considered the logical type conversion
in some complex types (array, map, union) yet. Will try to find a way to fix
this problem.
was (Author: yibing):
Had a quick look at this. The scope of this problem is bigger than I expected.
It looks like that current {{SpecificCompiler}} and
{{SpecificDatumWriter/Reader}} haven't considered the logical type conversion
in some complex types (array, map, union) yet. Will try to find a good way to
fix this problem.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUn
[jira] [Comment Edited] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405931#comment-15405931
]
Yibing Shi edited comment on AVRO-1891 at 8/3/16 1:46 PM:
--
Had a quick look at this. The scope of this problem is bigger than I expected.
It looks like that current {{SpecificCompiler}} and
{{SpecificDatumWriter/Reader}} haven't considered the logical type conversion
in some complex types (array, map, union) yet. Will try to find a good way to
fix this problem.
was (Author: yibing):
Had a quick look at this. The scope of this problem is bigger than I expected.
It looks like that current {{SpecificCompiler}} and
{{SpecificDatumWriter/Reader}} haven't considered the conversion of logical
types yet. Will try to find a good way to fix this problem.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.in
[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type
[
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405931#comment-15405931
]
Yibing Shi commented on AVRO-1891:
--
Had a quick look at this. The scope of this problem is bigger than I expected.
It looks like that current {{SpecificCompiler}} and
{{SpecificDatumWriter/Reader}} haven't considered the conversion of logical
types yet. Will try to find a good way to fix this problem.
> Generated Java code fails with union containing logical type
>
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
> "type": "record",
> "name": "RecordV1",
> "namespace": "org.brasslock.event",
> "fields": [
> { "name": "first", "type": ["null", {"type": "long",
> "logicalType":"timestamp-millis"}]}
> ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder =
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at
[jira] [Commented] (AVRO-1890) Java objects compiled from AVDL disregard default values
[
https://issues.apache.org/jira/browse/AVRO-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400569#comment-15400569
]
Yibing Shi commented on AVRO-1890:
--
[~ryonday], it is recommended to use record builder to create the specific
records.
{code}
fasdf.Wat.newBuilder().build()
{code}
This builder should be able to assign the default values for the fields.
> Java objects compiled from AVDL disregard default values
>
>
> Key: AVRO-1890
> URL: https://issues.apache.org/jira/browse/AVRO-1890
> Project: Avro
> Issue Type: Bug
>Reporter: Ryon Day
>Priority: Minor
>
> Given the schema
> {code}
> @namespace("fasdf")
> protocol foo {
> enum Domain {
> COM,
> ORG,
> BIZ,
> MIL
> }
> record Wat {
> Domain domain = "COM";
> string foo = "asfdasdf";
> }
> }
> {code}
> The resulting {{Wat}} Java class does not use the defined default values:
> {code}
> @Test
> public void asdfasdf() throws Exception {
> fasdf.Wat x = new Wat();
> System.out.print(x);
> }
> {code}
> This test results in:
> {code}
> {"domain": null, "foo": null}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1817) Allow enums to be "promoted" to strings
[
https://issues.apache.org/jira/browse/AVRO-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398756#comment-15398756
]
Yibing Shi commented on AVRO-1817:
--
[~busbey], I am not sure whether this task is feasible, especially when binary
encoder/writer is used.
AFAICS, in {{GenericDatumWriter}}, an enum value is written as its offset in
schema enum symbols list.
{code}
protected void writeEnum(Schema schema, Object datum, Encoder out)
throws IOException {
if (!data.isEnum(datum))
throw new AvroTypeException("Not an enum: "+datum);
out.writeEnum(schema.getEnumOrdinal(datum.toString()));
}
{code}
If {{BinaryEncoder}} is used, this offset is write *through* to the data file,
without any flags added to it.
{code}
public void writeEnum(int e) throws IOException {
this.writeInt(e);
}
{code}
In datum reader and decoder, it is very hard, if not impossible, to figure out
whether the data to read is actually an enum or an actual string. Things can be
even more complicated if unicode string is considered.
> Allow enums to be "promoted" to strings
> ---
>
> Key: AVRO-1817
> URL: https://issues.apache.org/jira/browse/AVRO-1817
> Project: Avro
> Issue Type: Improvement
> Components: java, spec
>Reporter: Michael Overmeyer
>Priority: Minor
>
> We should consider adding a resolution rule that can promote an enum to a
> string using the enum's symbol.
> I have an Avro schema that has a field with an enum type. However, I have
> realized that an enum is not the type I actually wanted. I would much rather
> have the type of the field be a string. I went to change this, but of course
> this type of change (enum -> string) is not within the bounds of Avro's
> schema evolution. Therefore a reader with this changed schema is not be able
> to read an object written with the old schema.
> For example, if the writer schema was:
> enum Colour {
> RED, YELLOW, GREEN
> }
> protocol stoplight {
> Colour colour;
> }
> And the reader schema was:
> protocol stoplight {
> string colour;
> }
> Then when you access the colour field of your object, you get the string
> representation of the enum value's symbol .
> For example, Colour.RED => "RED", Colour.YELLOW => "YELLOW", Colour.GREEN =>
> "GREEN"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388623#comment-15388623
]
Yibing Shi commented on AVRO-1877:
--
Thanks for the tip, [~busbey]!
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements
[
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi reassigned AVRO-1883:
Assignee: Yibing Shi
> Schema validator cannot find broken backwards compatibility in Union type
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
> {
> "type": "record",
> "name": "rec1",
> "fields": [
> {
> "name": "age",
> "type": "long"
> }
> ]
> },
> {
> "type": "record",
> "name": "rec2",
> "fields": [
> {
> "name": "username",
> "type": "string"
> }
> ]
> }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
> {
> "type": "record",
> "name": "rec1",
> "fields": [
> {
> "name": "age",
> "type": "long"
> },
> {
> "name": "address",
> "type": "string"
> }
> ]
> },
> {
> "type": "record",
> "name": "rec2",
> "fields": [
> {
> "name": "username",
> "type": "string"
> }
> ]
> }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the
> {{address}} field of {{rec1}} in the second one is not nullable. However, if
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements
[
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1883:
-
Status: Patch Available (was: Open)
> Schema validator cannot find broken backwards compatibility in Union type
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
> {
> "type": "record",
> "name": "rec1",
> "fields": [
> {
> "name": "age",
> "type": "long"
> }
> ]
> },
> {
> "type": "record",
> "name": "rec2",
> "fields": [
> {
> "name": "username",
> "type": "string"
> }
> ]
> }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
> {
> "type": "record",
> "name": "rec1",
> "fields": [
> {
> "name": "age",
> "type": "long"
> },
> {
> "name": "address",
> "type": "string"
> }
> ]
> },
> {
> "type": "record",
> "name": "rec2",
> "fields": [
> {
> "name": "username",
> "type": "string"
> }
> ]
> }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the
> {{address}} field of {{rec1}} in the second one is not nullable. However, if
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements
[
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1883:
-
Attachment: AVRO-1883.1.patch
> Schema validator cannot find broken backwards compatibility in Union type
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
> {
> "type": "record",
> "name": "rec1",
> "fields": [
> {
> "name": "age",
> "type": "long"
> }
> ]
> },
> {
> "type": "record",
> "name": "rec2",
> "fields": [
> {
> "name": "username",
> "type": "string"
> }
> ]
> }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
> {
> "type": "record",
> "name": "rec1",
> "fields": [
> {
> "name": "age",
> "type": "long"
> },
> {
> "name": "address",
> "type": "string"
> }
> ]
> },
> {
> "type": "record",
> "name": "rec2",
> "fields": [
> {
> "name": "username",
> "type": "string"
> }
> ]
> }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the
> {{address}} field of {{rec1}} in the second one is not nullable. However, if
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements
Yibing Shi created AVRO-1883:
Summary: Schema validator cannot find broken backwards
compatibility in Union type elements
Key: AVRO-1883
URL: https://issues.apache.org/jira/browse/AVRO-1883
Project: Avro
Issue Type: Bug
Affects Versions: 1.8.1
Reporter: Yibing Shi
Priority: Critical
Consider below 2 schemas:
*Schema 1*:
{noformat}
[
{
"type": "record",
"name": "rec1",
"fields": [
{
"name": "age",
"type": "long"
}
]
},
{
"type": "record",
"name": "rec2",
"fields": [
{
"name": "username",
"type": "string"
}
]
}
]
{noformat}
*Schema 2*:
{noformat}
[
{
"type": "record",
"name": "rec1",
"fields": [
{
"name": "age",
"type": "long"
},
{
"name": "address",
"type": "string"
}
]
},
{
"type": "record",
"name": "rec2",
"fields": [
{
"name": "username",
"type": "string"
}
]
}
]
{noformat}
The {{rec1}} field in these 2 unions are not compatible, because the
{{address}} field of {{rec1}} in the second one is not nullable. However, if we
check them with validate like below, validator doesn't return any error:
{code}
final SchemaValidator backwardValidator = new
SchemaValidatorBuilder().canReadStrategy().validateLatest();
final Schema schema1 = new Schema.Parser().parse(schema1Str);
final Schema schema2 = new Schema.Parser().parse(schema2Str);
backwardValidator.validate(schema2, Arrays.asList(schema1));
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1877:
-
Attachment: AVRO-1877.3.patch
Upload a new patch with code refactored
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369995#comment-15369995
]
Yibing Shi edited comment on AVRO-1877 at 7/10/16 11:29 PM:
{quote}
Honestly, I like the old code better.
{quote}
The problem with the old code is that it checks for conversions twice: once in
{{javaUnbox}}, once again in {{javaType}}
That said, I have a look at the code again and the extra test for UNION does
smells bad. I will refactor the code a bit more and upload another patch here.
{quote}
As for enableDecimalLogicalType. If truly the intent is to use converted
logical types at all, then this should be renamed to enableLogicalType. It's
confusing to say it's for Decimal but mean it for all.
{quote}
As said before, {{enableDecimalLogicalType}} should only applies to decimal
type, not the date/time logical types, which should always use Joda/Java
date/time classes instead of raw types if these classes are present.
Please have a look at the unit tests added in {{TestSpecificCompiler}}
(testJavaTypeWithDecimalLogicalTypeDisabled/Enabled). They clearly show the
intent of {{enableDecimalLogicalType}}
{quote}
More importantly, after applying your AVRO-1847, if you have date or
timestamp-millis without setting enableDecmialLogicalType, the compiled
SpecificRecord code will NOT compile.
{quote}
This problem is caused because the problem tracked in this JIRA (AVRO-1877).
Please try apply AVRO-1877 on top of AVRO-1847 and see whether it fixes the
problem for you.
was (Author: yibing):
{quote}
Honestly, I like the old code better.
{quote}
The problem with the old code is that it checks for conversions twice: once in
{{javaUnbox}}, once again in {{javaType}}
That said, I have a look at the code again and the extra test for UNION does
smells bad. I will refactor the code a bit more and upload another patch here.
{quote}
As for enableDecimalLogicalType. If truly the intent is to use converted
logical types at all, then this should be renamed to enableLogicalType. It's
confusing to say it's for Decimal but mean it for all.
{quote}
As said before, {{enableDecimalLogicalType}} should only applies to decimal
type, not the date/time logical types, which should always use Joda/Java
date/time classes instead of raw types if these classes are present.
{quote}
More importantly, after applying your AVRO-1847, if you have date or
timestamp-millis without setting enableDecmialLogicalType, the compiled
SpecificRecord code will NOT compile.
{quote}
This problem is caused because the problem tracked in this JIRA (AVRO-1877).
Please try apply AVRO-1877 on top of AVRO-1847 and see whether it fixes the
problem for you.
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369995#comment-15369995
]
Yibing Shi commented on AVRO-1877:
--
{quote}
Honestly, I like the old code better.
{quote}
The problem with the old code is that it checks for conversions twice: once in
{{javaUnbox}}, once again in {{javaType}}
That said, I have a look at the code again and the extra test for UNION does
smells bad. I will refactor the code a bit more and upload another patch here.
{quote}
As for enableDecimalLogicalType. If truly the intent is to use converted
logical types at all, then this should be renamed to enableLogicalType. It's
confusing to say it's for Decimal but mean it for all.
{quote}
As said before, {{enableDecimalLogicalType}} should only applies to decimal
type, not the date/time logical types, which should always use Joda/Java
date/time classes instead of raw types if these classes are present.
{quote}
More importantly, after applying your AVRO-1847, if you have date or
timestamp-millis without setting enableDecmialLogicalType, the compiled
SpecificRecord code will NOT compile.
{quote}
This problem is caused because the problem tracked in this JIRA (AVRO-1877).
Please try apply AVRO-1877 on top of AVRO-1847 and see whether it fixes the
problem for you.
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369646#comment-15369646
]
Yibing Shi commented on AVRO-1877:
--
[~mwong], I have uploaded another patch, but fixes the problem in a different
way than your suggestion. Please help have another look.
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1877:
-
Attachment: AVRO-1877.2.patch
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369631#comment-15369631
]
Yibing Shi commented on AVRO-1877:
--
{quote}
The way you call javaType from javaUnbox in this latest patch is not correct.
For example, if you have a {{union \{null, double\}}}, javaType() will turn it
into a boxed java.lang.Double that is nullable. Your changes to javaUnbox will
then incorrectly convert that to a primitive type double. My suggestion is to
revert to the old code and maybe refactor the duplicate SPECIFIC conversion
part into a separate method.
{quote}
Very good point! I will correct this. Thanks for the idea!
{quote}
The problem is that enableDecimalLogicalType only applies to the Decimal
logical type. For the other logical types, such as date or timestamp-millis,
they would be converted even if you didn't enableDecimalLogicalType.
{quote}
This is done on purpose. It is to keep backwards compatibility with 1.8.0. When
date/time logical types were added in 1.8.0, they always use logical types
instead of backing raw types in SpecificRecord. Please see [this
comment|https://issues.apache.org/jira/browse/AVRO-1847?focusedCommentId=15327757&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15327757]
for more details.
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367651#comment-15367651
]
Yibing Shi edited comment on AVRO-1877 at 7/8/16 1:14 PM:
--
Hi [~rdblue], it was my fault. I accidentally deleted a part of method
{{javaUnbox}}. I am not sure how it happened though.
Before AVRO-1847, the javaUnbox method is like below:
{code}
public String javaType(Schema schema) {
Conversion conversion = SPECIFIC
.getConversionFor(schema.getLogicalType());
if (conversion != null) {
return conversion.getConvertedType().getName();
}
switch (schema.getType()) {
..
{code}
However, AVRO-1847 somehow deleted the conversion part and made it like below:
{code}
public String javaType(Schema schema) {
switch (schema.getType()) {
..
{code}
As a result, for logical type timestamp_ms, date and time_ms, method
{{javaUnbox}} returns "int" instead of JODA classes, which breaks the generated
Specific Record.
I submitted a patch in this JIRA. Instead of using the original code, I rely on
method {{javaType}} to do the conversion to avoid maintain the same code in 2
places, especially when now we have more and more parameters to control whether
to use upper level classes for a logical type. I also added 2 unit tests in
this patch to cover this function. Please let me know whether you think this is
appropriate or not.
was (Author: yibing):
Hi [~rdblue], it was my fault. I accidentally deleted a part of method
{{javaUnbox}}. I am not sure how it happened though.
Before AVRO-1847, the javaUnbox method is like below:
{code}
public String javaType(Schema schema) {
Conversion conversion = SPECIFIC
.getConversionFor(schema.getLogicalType());
if (conversion != null) {
return conversion.getConvertedType().getName();
}
switch (schema.getType()) {
..
{code}
However, AVRO-1847 somehow deleted the conversion part and made it like below:
{code}
public String javaType(Schema schema) {
switch (schema.getType()) {
..
{code}
I submitted a patch in this JIRA. Instead of using the original code, I rely on
method {{javaType}} to do the conversion to avoid maintain the same code in 2
places, especially when now we have more and more parameters to control whether
to use upper level classes for a logical type. I also added 2 unit tests in
this patch to cover this function. Please let me know whether you think this is
appropriate or not.
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367651#comment-15367651
]
Yibing Shi commented on AVRO-1877:
--
Hi [~rdblue], it was my fault. I accidentally deleted a part of method
{{javaUnbox}}. I am not sure how it happened though.
Before AVRO-1847, the javaUnbox method is like below:
{code}
public String javaType(Schema schema) {
Conversion conversion = SPECIFIC
.getConversionFor(schema.getLogicalType());
if (conversion != null) {
return conversion.getConvertedType().getName();
}
switch (schema.getType()) {
..
{code}
However, AVRO-1847 somehow deleted the conversion part and made it like below:
{code}
public String javaType(Schema schema) {
switch (schema.getType()) {
..
{code}
I submitted a patch in this JIRA. Instead of using the original code, I rely on
method {{javaType}} to do the conversion to avoid maintain the same code in 2
places, especially when now we have more and more parameters to control whether
to use upper level classes for a logical type. I also added 2 unit tests in
this patch to cover this function. Please let me know whether you think this is
appropriate or not.
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1877:
-
Status: Patch Available (was: Open)
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1877:
-
Attachment: AVRO-1877.1.patch
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
[
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1877:
-
Hadoop Flags: (was: Incompatible change)
> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
>
> AVRO-1847 accidentally removes the logical type conversion logic in method
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
> specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
Yibing Shi created AVRO-1877:
Summary: AVRO-1847 accidentally breaks javaUnbox method in
SpecificCompiler
Key: AVRO-1877
URL: https://issues.apache.org/jira/browse/AVRO-1877
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.9.0
Reporter: Yibing Shi
Priority: Blocker
AVRO-1847 accidentally removes the logical type conversion logic in method
{{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in
specific compiler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8
[
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362617#comment-15362617
]
Yibing Shi commented on AVRO-1811:
--
{quote}
What's the current behavior if we try to turn a SpecificRecord that contains
either representation into a GenericRecord
{quote}
I believe it depends on how to do the conversion.
If we first serialize the SpecificRecord (no matter using high/low level
representation), and then deserialize it into a GenericRecord, it should work
fine , but the logical type fields will use low level representations in
GenericRecord, unless conversion objects are explicitly added to GenericRecord
before reading the underlying data.
If we use {{GenericRecord.deepCopy}} directly to copy a SpecificRecord with
high level logical type representations (BigDecimal etc.), I believe it would
fail, because {{GenericData.deepCopy}} doesn't understand the high level
representations at all. A CCE would be thrown out. Copying a SpecificRecord
with low level representation should be fine because it is just the same as
before Logical Type is adopted in.
Actually, we may also face problems when deepCopy a GenericData to
SpecificData. The setField method in SpecificData needs a value that matches
field type. If the logical type field uses high level representation, I expect
this copying would fail, because deepCopy at this moment only returns low level
representations.
I haven't tested these behaviours though. Will do some test when I have time.
The patch uploaded here should be able to solve the immediate problem. Thanks
for your acceptance of it as a short-term solution. I am just afraid we may
face other similar problems in other scenarios. Maybe we should create another
JIRA to track the wider "convertible" values problem?
> SpecificData.deepCopy() cannot be used if schema compiler generated Java
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them
> generate fields of type {{string}} with the Java standard {{String}} type,
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every
> {{string}} field in a schema like so:
> {code}
> {
> "name": "some_string",
> "doc": "a field that is guaranteed to compile to java.lang.String",
> "type": [
> "null",
> {
> "type": "string",
> "avro.java.string": "String"
> }
> ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by
> this annotation by volume; for teams using heterogenous clients, they may to
> want to avoid Java-specific annotation in their schema files, or may not
> think to use it unless there exist Java exploiters of the schema at the time
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects
> using the {{SpecificCompiler}}'s string type selection. This option actually
> alters the schema carried by the object's {{SCHEMA$}} field to have the above
> annotation in it, ensuring that when used by the Java API, the String type
> will be used.
> Unfortunately, this method is not interoperable with GenericRecords created
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out
> of the {{GenericRecord}}
> There is a unit test that demonstrate this
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are lit
[jira] [Commented] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8
[
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361264#comment-15361264
]
Yibing Shi commented on AVRO-1811:
--
Hi [~rdblue], I have another consideration about this problem. Above change is
ugly. It only fixes the record copy problem. We may still face the same problem
in other scenarios. Maybe we should change the specific compiler instead to
make the {{setField}} method be able to accept different types of values (for
example, can accept both String and Utf8 for a string field)?
> SpecificData.deepCopy() cannot be used if schema compiler generated Java
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them
> generate fields of type {{string}} with the Java standard {{String}} type,
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every
> {{string}} field in a schema like so:
> {code}
> {
> "name": "some_string",
> "doc": "a field that is guaranteed to compile to java.lang.String",
> "type": [
> "null",
> {
> "type": "string",
> "avro.java.string": "String"
> }
> ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by
> this annotation by volume; for teams using heterogenous clients, they may to
> want to avoid Java-specific annotation in their schema files, or may not
> think to use it unless there exist Java exploiters of the schema at the time
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects
> using the {{SpecificCompiler}}'s string type selection. This option actually
> alters the schema carried by the object's {{SCHEMA$}} field to have the above
> annotation in it, ensuring that when used by the Java API, the String type
> will be used.
> Unfortunately, this method is not interoperable with GenericRecords created
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out
> of the {{GenericRecord}}
> There is a unit test that demonstrate this
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be
> cast to java.lang.String}}
> {panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8
[
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1811:
-
Attachment: AVRO-1811.1.patch
> SpecificData.deepCopy() cannot be used if schema compiler generated Java
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them
> generate fields of type {{string}} with the Java standard {{String}} type,
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every
> {{string}} field in a schema like so:
> {code}
> {
> "name": "some_string",
> "doc": "a field that is guaranteed to compile to java.lang.String",
> "type": [
> "null",
> {
> "type": "string",
> "avro.java.string": "String"
> }
> ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by
> this annotation by volume; for teams using heterogenous clients, they may to
> want to avoid Java-specific annotation in their schema files, or may not
> think to use it unless there exist Java exploiters of the schema at the time
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects
> using the {{SpecificCompiler}}'s string type selection. This option actually
> alters the schema carried by the object's {{SCHEMA$}} field to have the above
> annotation in it, ensuring that when used by the Java API, the String type
> will be used.
> Unfortunately, this method is not interoperable with GenericRecords created
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out
> of the {{GenericRecord}}
> There is a unit test that demonstrate this
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be
> cast to java.lang.String}}
> {panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8
[
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1811:
-
Affects Version/s: 1.8.1
Status: Patch Available (was: Open)
> SpecificData.deepCopy() cannot be used if schema compiler generated Java
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1, 1.8.0
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them
> generate fields of type {{string}} with the Java standard {{String}} type,
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every
> {{string}} field in a schema like so:
> {code}
> {
> "name": "some_string",
> "doc": "a field that is guaranteed to compile to java.lang.String",
> "type": [
> "null",
> {
> "type": "string",
> "avro.java.string": "String"
> }
> ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by
> this annotation by volume; for teams using heterogenous clients, they may to
> want to avoid Java-specific annotation in their schema files, or may not
> think to use it unless there exist Java exploiters of the schema at the time
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects
> using the {{SpecificCompiler}}'s string type selection. This option actually
> alters the schema carried by the object's {{SCHEMA$}} field to have the above
> annotation in it, ensuring that when used by the Java API, the String type
> will be used.
> Unfortunately, this method is not interoperable with GenericRecords created
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out
> of the {{GenericRecord}}
> There is a unit test that demonstrate this
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be
> cast to java.lang.String}}
> {panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8
[
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360955#comment-15360955
]
Yibing Shi commented on AVRO-1811:
--
Hi [~rdblue], this problem may not be very easy. It happens when deep copying
an object that uses {{Utf8}} as string representation into a specific record
which uses {{java.lang.String}} instead. We can simplify the issue with below
unit test:
{code}
@Test
public void testStringDeepCopy() throws IOException {
FooBarSpecificRecord specificRecord = FooBarSpecificRecord.newBuilder()
.setId(1)
.setName("test_record_specific")
.setNicknames(new ArrayList())
.setRelatedids(new ArrayList())
.build();
GenericRecordBuilder builder = new
GenericRecordBuilder(specificRecord.getSchema());
GenericRecord genericRecord = builder
.set("id", 1)
.set("name", new Utf8("test_record_specific"))
.set("nicknames", new ArrayList())
.set("relatedids", new ArrayList())
.build();
FooBarSpecificRecord copiedFromGeneric = (FooBarSpecificRecord)
SpecificData.get().deepCopy(
FooBarSpecificRecord.getClassSchema(), genericRecord);
assertEquals("Should get an equal record by deep copying the generic
record",
specificRecord, copiedFromGeneric);
}
{code}
This unit test fails in master branch with below exception:
{noformat}
java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
java.lang.String
at
org.apache.avro.FooBarSpecificRecord.put(FooBarSpecificRecord.java:70)
at org.apache.avro.generic.GenericData.setField(GenericData.java:659)
at org.apache.avro.generic.GenericData.setField(GenericData.java:676)
at org.apache.avro.generic.GenericData.deepCopy(GenericData.java:1081)
at
org.apache.avro.generic.TestGenericData.testStringDeepCopy(TestGenericData.java:450)
{noformat}
Looking at [the
code|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java#L1079-L1081]:
{code}
Object newValue = deepCopy(f.schema(),
getField(value, name, pos, oldState));
setField(newRecord, name, pos, newValue, newState);
{code}
Variable newValue is an {{Utf8}} object because {{getField}} of the generic
record uses it for field "name", while {{setField}} method in
{{FooBarSpecificRecord}} expects a {{java.lang.string}} as input parameter.
This causes above exception.
Method "deepCopy" is defined with below signature:
{code}
public T deepCopy(Schema schema, T value) {
{code}
which makes the problem a bit complicated. It can only return an object of
exactly the same type as input parameter "value". That is, if we pass in an
{{Utf8}} object as "value", we cannot return a {{java.lang.String}} object.
The only way I can find to solve this problem is to finish the conversion
before calling {{setField}} when deep copying records. That is, change above
code to something like below:
{code}
Object newValue = deepCopy(f.schema(),
getField(value, name, pos, oldState));
if (STRING_TYPE_STRING.equals(f.schema().getProp(STRING_PROP))) {
if (! (newValue instanceof java.lang.String)) {
newValue = newValue.toString();
}
} else {
if (newValue instanceof java.lang.String) {
newValue = new Utf8((String) newValue);
}
}
setField(newRecord, name, pos, newValue, newState);
{code}
I will submit a patch based on this thought.
Further thoughts:
# [~ryonday], why did you have incompatible schemas in Specific Data and
Generic Data in the unit test you provided
[here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]?
How did you generate the avsc files?
# [~rdblue], this problem reminds me of the Logical Types. A logical type field
can have 2 representations: one uses raw type and the other uses upper level
java types. For instance, for "decimal" type, we can use both ByteBuffer and
BigDecimal. Do we need support copying between them?
> SpecificData.deepCopy() cannot be used if schema compiler generated Java
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Ryon Day
>Priority: Critical
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them
> generate fields of ty
[jira] [Assigned] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8
[
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi reassigned AVRO-1811:
Assignee: Yibing Shi
> SpecificData.deepCopy() cannot be used if schema compiler generated Java
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them
> generate fields of type {{string}} with the Java standard {{String}} type,
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every
> {{string}} field in a schema like so:
> {code}
> {
> "name": "some_string",
> "doc": "a field that is guaranteed to compile to java.lang.String",
> "type": [
> "null",
> {
> "type": "string",
> "avro.java.string": "String"
> }
> ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by
> this annotation by volume; for teams using heterogenous clients, they may to
> want to avoid Java-specific annotation in their schema files, or may not
> think to use it unless there exist Java exploiters of the schema at the time
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects
> using the {{SpecificCompiler}}'s string type selection. This option actually
> alters the schema carried by the object's {{SCHEMA$}} field to have the above
> annotation in it, ensuring that when used by the Java API, the String type
> will be used.
> Unfortunately, this method is not interoperable with GenericRecords created
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out
> of the {{GenericRecord}}
> There is a unit test that demonstrate this
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be
> cast to java.lang.String}}
> {panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1869) Decimal conversion from ByteBuffer ignores buffer position and limit
[
https://issues.apache.org/jira/browse/AVRO-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352944#comment-15352944
]
Yibing Shi commented on AVRO-1869:
--
Thanks [~rdblue]!
> Decimal conversion from ByteBuffer ignores buffer position and limit
>
>
> Key: AVRO-1869
> URL: https://issues.apache.org/jira/browse/AVRO-1869
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Fix For: 1.8.2
>
>
> The decimal conversion from a ByteBuffer currently copies the buffer contents
> into an array, but then calls {{array}} on the buffer and uses the buffer's
> internal array rather than the copied bytes. The result is that the
> constructed Decimal value uses whatever the buffer contained without using
> its position or limit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346370#comment-15346370 ] Yibing Shi commented on AVRO-1847: -- [~busbey] and [~rdblue], thank you for your help on this! > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Fix For: 1.9.0, 1.8.2 > > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch, > AVRO-1847.8.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.8.patch
I forgot to apply the decimal type flag to
{{SpecificCompiler.conversionInstance}}. Upload a new patch that corrects this
problem, and adds 2 unit tests to cover this.
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch,
> AVRO-1847.8.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337885#comment-15337885 ] Yibing Shi commented on AVRO-1847: -- Thank you very much for your help, [~busbey]! [~rdblue], please let me know if you have anything else to add. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337884#comment-15337884 ] Yibing Shi commented on AVRO-1847: -- Yes, I did add the test for UUID. It always use CharSequence as type. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.7.patch
Upload a new patch:
# Fixed all the javadoc errors pointed out by [~busbey]
# Use {{IllegalArugumentException}} instead of {{AvroRuntimeException}} in
{{Conversions.convert...}} functions
# Add javadoc to {{GenericDatumWriter/Reader.convert}} to describe the
behaviour change (different exception type is thrown out now). [~rdblue], would
you please help confirm whether this is OK for you?
And as [~busbey] said, I have added below test cases in
{{TestSpecificCompiler}}:
# testJavaTypeWithDecimalLogicalTypeEnabled
# testJavaTypeWithoutDecimalLogicalTypeEnabled
to test whether the decimal logical type enablement would affect other logical
types. [~rdblue], please help confirm whether it is enough, or you want to add
more tests?
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Attachment: AVRO-1847.6.patch Upload a new patch > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331268#comment-15331268
]
Yibing Shi edited comment on AVRO-1847 at 6/15/16 2:01 PM:
---
# I will add javadoc to those public functions
# I will change the test case names.
# I will correct the help messages for compiler tool. Do you think it worth to
use Apache Commons Options package to process the parameters? (maybe leave it
for a later?)
# The null and "null" thing is a typo. Sorry for this stupid mistake. A null
value will cause problems for SpecificRecordCompiler.
# I will make the parameter in compiler and maven plugin only affects decimal
type, which will minimize the impact of this patch. Thanks for the suggestion!
# For the invariant problem in {{RecordBuilderBase.defaultValue(Field field,
Conversion conversion)}}, I will add a checking to make sure that logical
type is not null before call the {{convertToLogicalType}} method.
# For the convert to logical/raw type methods, I will change it a bit to have
them return null if the original value passed in is null.
# The focus here is the {{convert}} method in GenericDatumWriter. You are
correct about the invariant here. But as [~rdblue] explains below, this used to
be not a problem because the caller within GenericDatumWriter won't pass in any
null pointer. I tends not to add the checking here. One reason is it is good
for performance. The second reason is that a null schema or logical type is
definitely a bad error, and I don't think any children would try to capture or
process it.
was (Author: yibing):
# I will add javadoc to those public functions
# I will change the test case names.
# I will correct the help messages for compiler tool. Do you think it worth to
use Apache Commons Options package to process the parameters? (maybe leave it
for a later?)
# The null and "null" thing is a typo. Sorry for this stupid mistake. A null
value will cause problems for SpecificRecordCompiler.
# I will make the parameter in compiler and maven plugin only affects decimal
type, which will minimize the impact of this patch. Thanks for the suggestion
# The focus here is the {{convert}} method in GenericDatumWriter. You are
correct about the invariant here. But as [~rdblue] explains below, this used to
be not a problem because the caller within GenericDatumWriter won't pass in any
null pointer. I tends not to add the checking here. One reason is it is good
for performance. The second reason is that a null schema or logical type is
definitely a bad error, and I don't think any children would try to capture or
process it.
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331759#comment-15331759
]
Yibing Shi commented on AVRO-1847:
--
Method {{GenericDatumWriter.convert}} is a protected method from the very
beginning. Changing it to private would be a behaviour change. I prefer not to
make this change. Please let me know you opinion.
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331271#comment-15331271 ] Yibing Shi commented on AVRO-1847: -- I had a very busy day. Will work on this tonight and upload a new patch. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch, AVRO-1847.5.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331268#comment-15331268
]
Yibing Shi commented on AVRO-1847:
--
# I will add javadoc to those public functions
# I will change the test case names.
# I will correct the help messages for compiler tool. Do you think it worth to
use Apache Commons Options package to process the parameters? (maybe leave it
for a later?)
# The null and "null" thing is a typo. Sorry for this stupid mistake. A null
value will cause problems for SpecificRecordCompiler.
# I will make the parameter in compiler and maven plugin only affects decimal
type, which will minimize the impact of this patch. Thanks for the suggestion
# The focus here is the {{convert}} method in GenericDatumWriter. You are
correct about the invariant here. But as [~rdblue] explains below, this used to
be not a problem because the caller within GenericDatumWriter won't pass in any
null pointer. I tends not to add the checking here. One reason is it is good
for performance. The second reason is that a null schema or logical type is
definitely a bad error, and I don't think any children would try to capture or
process it.
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Attachment: AVRO-1847.5.patch Upload the new patch that contains all the changes based on code review. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch, AVRO-1847.5.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329515#comment-15329515 ] Yibing Shi commented on AVRO-1847: -- Oops! Have corrected it. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329521#comment-15329521
]
Yibing Shi commented on AVRO-1847:
--
As mentioned above, I don't think we need a separate flag for date/time types
since their behaviour has never changed. I just make sure to use JODA/Java
classes for these types no matter the value of this parameter. This parameter
has also been renamed as {{enableLogicalTypesForNonDateTimeTypes}}
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329522#comment-15329522 ] Yibing Shi commented on AVRO-1847: -- Same as above > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329511#comment-15329511 ] Yibing Shi commented on AVRO-1847: -- I didn't think of childrens of this class may need to check its return value. I thought an exception made more sense here because there is an unknown logical type was found. Now I agree with you that return null makes their life easier. Have changed the code back according to your suggestion. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329501#comment-15329501
]
Yibing Shi commented on AVRO-1847:
--
Since we always try to use Joda or Java classes for date/time types, I prefer
not to add a new flag. Instead, I just check the whether the the logical type
is one of the date/time types. If yes, I apply the conversions if they are
present.
{code}
public String javaType(Schema schema) {
if (useLogicalTypesForNonDateTimeTypes
|| LogicalTypes.isDateTimeType(schema.getLogicalType())) {
Conversion conversion = SPECIFIC
.getConversionFor(schema.getLogicalType());
if (conversion != null) {
return conversion.getConvertedType().getName();
}
}
{code}
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329495#comment-15329495 ] Yibing Shi commented on AVRO-1847: -- I have removed the passthrough behaviour in convert to/from methods. As a result, this part of code keeps intact. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329494#comment-15329494
]
Yibing Shi commented on AVRO-1847:
--
The {{conver}} method is newly added by this patch. It didn't exist in master
branch. However, the exception it throws does get out through method {{write}}.
I agree with [~busbey] that we should keep the exceptions the same as before.
On the other hand, it also makes sense to make the to convert methods in
{{Conversions}} behave similarly. So I prefer throwing AvroRuntimeException in
{{convertToRawType}}, but catch this exception in
{{GenericDatumWriter.convert}} and throw its cause out to keep backwards
compatibility. Maybe we can change this feature in later versions?
The code is like below:
{code}
protected Object convert(Schema schema, LogicalType logicalType,
Conversion conversion, Object datum) {
try {
if (conversion == null) {
return datum;
} else {
return Conversions.convertToRawType(datum, schema, logicalType,
conversion);
}
} catch (AvroRuntimeException e) {
Throwable cause = e.getCause();
if (cause != null && cause.getClass() == ClassCastException.class) {
// This is to keep backwards compatibility. The convert function here
used to
// throw CCE. When prompted to Conversions, it throws
AvroRuntimeException. So,
// if the cause is a CCE, rethrow it in case any child class checks it.
This
// behaviour can be changed later in future versions to make it
consistent with
// reading path, which throws AvroRuntimeException
throw (ClassCastException)cause;
} else {
throw e;
}
}
}
{code}
Please let me know your opinion.
> IDL compiler uses ByteBuffer for decimal type even if logical type is
> supported
>
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch,
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class
> (Conversions.DecimalConversion) has also been added for decimal type.
> However, the IDL compiler still uses ByteBuffer for decimal types, which is
> not the same behaviour as data, time or timestamp type (added in AVRO-1684).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328665#comment-15328665 ] Yibing Shi edited comment on AVRO-1847 at 6/14/16 1:18 PM: --- Agree. Since these 2 methods are now public, I add checking for all the parameters. If any of them is null, an AvroRuntimeException will be thrown out. IMO, adding the pass through function to these functions makes them coupled with callers. Please let me know if you don't think so. was (Author: yibing): Agree. Will do that. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328665#comment-15328665 ] Yibing Shi commented on AVRO-1847: -- Agree. Will do that. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi >Assignee: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Attachment: AVRO-1847.4.patch Add another patch, which add the backward compatibility of logical type support. That is, by default the compiler still defines class member types based on "type" in schema (such as ByteBuffer for decimal), but also add a parameter which enables defining members based on "logicalType" (such as BigDecimal for decimal). > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, > AVRO-1847.4.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Attachment: AVRO-1847.3.patch Attach another patch, which fixes the problem where classes generated by IDL compiler cannot be constructed via builder properly. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311336#comment-15311336 ] Yibing Shi commented on AVRO-1847: -- Understood! Thanks for the response! > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309321#comment-15309321 ] Yibing Shi commented on AVRO-1847: -- [~rdblue], could you please help have a look at the new patch? > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Attachment: AVRO-1847.2.patch [~rdblue], I have attached another patch that adds the missing parts mentioned in your previous comment. Could you please help have another look? > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290055#comment-15290055 ] Yibing Shi commented on AVRO-1847: -- Hi [~rdblue], thanks for the guide! I will modify the patch and submit it again. > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
Yibing Shi created AVRO-1847: Summary: IDL compiler uses ByteBuffer for decimal type even if logical type is supported Key: AVRO-1847 URL: https://issues.apache.org/jira/browse/AVRO-1847 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.8.0 Reporter: Yibing Shi Version 1.8.0 has added the support of logical types. A conversion class (Conversions.DecimalConversion) has also been added for decimal type. However, the IDL compiler still uses ByteBuffer for decimal types, which is not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Attachment: AVRO-1847.1.patch > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Status: Patch Available (was: Open) > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
