[jira] [Commented] (AVRO-1847) IDL compiler should use BigDecimal to represent decimal logical type.

2017-04-02 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952612#comment-15952612
 ] 

Yibing Shi commented on AVRO-1847:
--

[~vijayp...@gmail.com], I have no idea about the release date of 1.8.2. Maybe 
[~rdblue] or [~busbey] can answer your question.

> IDL compiler should use BigDecimal to represent decimal logical type.
> -
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: New Feature
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch, 
> AVRO-1847.8.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional

2016-12-01 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713643#comment-15713643
 ] 

Yibing Shi commented on AVRO-1961:
--

{quote}
Actually, on the setter/builder/constructor, we could even go a step further: a 
required field with a default value could be with Optional as well. The 
matching getter would be without Optional. It would make the concept of default 
value type safe just as well. Sweet isn't it?
{quote}
And Optional object indicates that the value could be null. It has a different 
meaning than having a default value. That is, you can skip passing in a value 
for this field doesn't mean you can pass in a null value for this field. 
Suggest not to apply this to fields with default values.

{quote}
1. Java 8 isn't required for Avro project for this to work
All the "magic" happen uniquely through the template: the code could still 
compile with Java 6, but anyone using the "pojoWithOptional" flag should 
consume the generated pojo with Java 8 or backport Optional.
{quote}
Is Java 8 is needed to test the pojoWithOptionalGenerationTest you are going to 
add?

> [JAVA] Generate getters that return an Optional
> ---
>
> Key: AVRO-1961
> URL: https://issues.apache.org/jira/browse/AVRO-1961
> Project: Avro
>  Issue Type: New Feature
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
>
> A colleague of mine indicated that having getters that return an Optional 
> (java 8 thingy) would be very useful for him.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1962) Support UUID logical type

2016-11-22 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688400#comment-15688400
 ] 

Yibing Shi commented on AVRO-1962:
--

[~xiongtx], sorry for the misleading info in mail list! UUID type can be 
specified like below:

{code}
{
  "type": "string",
  "logicalType": "uuid"
}
{code}

> Support UUID logical type
> -
>
> Key: AVRO-1962
> URL: https://issues.apache.org/jira/browse/AVRO-1962
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Tianxiang Xiong
>
> The AVRO-1554 ticket seems to suggest that the issue of representing UUIDs is 
> resolved with [logical 
> types|http://avro.apache.org/docs/1.8.1/spec.html#Logical+Types] in Avro
> 1.8.1. However, there is [no UUID logical type in Avro 
> 1.8.1|https://github.com/apache/avro/blob/release-1.8.1/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj#L214-L244].
> The specification offers several examples of using logical types; decimals
> are represented as:
> {code}
> {
>   "type": "bytes",
>   "logicalType": "decimal",
>   "precision": 4,
>   "scale": 2
> }
> {code}
> No examples for UUID are offered, presumably because UUIDs are not supported.
> Thanks to [~Yibing]'s confirmation on the mailing list that this is the case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1961) [JAVA] Generate getters that return an Optional

2016-11-22 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688256#comment-15688256
 ] 

Yibing Shi commented on AVRO-1961:
--

[~nielsbasjes], would you please provide more details of this request? Such as 
how the Optional type should be returned and what is the benefit of it.

> [JAVA] Generate getters that return an Optional
> ---
>
> Key: AVRO-1961
> URL: https://issues.apache.org/jira/browse/AVRO-1961
> Project: Avro
>  Issue Type: New Feature
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Minor
>
> A colleague of mine indicated that having getters that return an Optional 
> (java 8 thingy) would be very useful for him.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1927) If a default value is set, Avro allows null values in non-nullable fields.

2016-09-25 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522012#comment-15522012
 ] 

Yibing Shi commented on AVRO-1927:
--

[~asmaier], currently the generated code doesn't check whether we are setting a 
null value for a non nullable column. In your case, null is a valid Java String 
value and thus it works. However, I believe AVRO would throw an exception when 
you try to write your object. Have you tried that? Does it fail or succeed?

{code}
ByteArrayOutputStream os = new ByteArrayOutputStream(1024);
Encoder encoder = EncoderFactory.get().binaryEncoder(os, null);
SpecificDatumWriter writer = new 
SpecificDatumWriter(MyObject.getClassSchema());
writer.write(myObject, encoder);
{code}

> If a default value is set, Avro allows null values in non-nullable fields.
> --
>
> Key: AVRO-1927
> URL: https://issues.apache.org/jira/browse/AVRO-1927
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: Andreas Maier
>  Labels: newbie
>
> With an avro schema like
> {code}
> {
>   "name": "myfield",
>   "type": "string",
>   "default": ""
> }
> {code}
> the following code should throw an exception
> {code}
> MyObject myObject = MyObject.newBuilder().setMyfield(null).build();
> {code}
> But instead the value of myfield is set to null, which causes an exception 
> later when serializing myObject, because null is not a valid value for 
> myfield. 
> I believe in this case setMyfield(null) should throw an exception, 
> independent of the value of default. 
> See also
> https://stackoverflow.com/questions/38509279/generated-avro-builder-set-null-doesnt-overwrite-with-default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-05 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15466040#comment-15466040
 ] 

Yibing Shi commented on AVRO-1891:
--

The patch looks great, [~rdblue]! 
I add a few comments to it.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-04 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15463713#comment-15463713
 ] 

Yibing Shi commented on AVRO-1891:
--

[~rdblue], you idea looks good to me!

{quote}
Maintain a thread-local reference to the current specific record class in 
GenericDatumReader
{quote}
I am also thinking about using a thread local variable, because my patch makes 
data reader/writer non thread-safe. We may need to keep a stack of class 
references, which will be helpful when there are nested records.

{quote}
# Add conversion lookup methods to GenericDatumReader that delegate to 
GenericData
# Override the conversion lookup methods in SpecificDatumReader that use the 
current record class's set of conversions instead.
{quote}
This is a great idea! Looking forward to your patch!
(For backward compatibility, the conversion lookup methods in 
SpecificDataumReader/Writer may need to fallback to those in 
GenericDatumReader/Writer if the generated class doesn't have the static 
conversion map -- for legacy classes I mean)

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(Paren

[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-03 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.4.patch

Upload a new patch that fixes the defaultValue bug and can be applied to master 
branch

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch, AVRO-1891.yshi.4.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460908#comment-15460908
 ] 

Yibing Shi commented on AVRO-1891:
--

Just found the {{defaultValue}} method I added has a bug. I guess you were 
actually mentioning to this bug when you said that my modification was 
incompatible. I will fix it. Thanks for pointing this out.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.j

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460903#comment-15460903
 ] 

Yibing Shi commented on AVRO-1891:
--

I have another look into the code, and find that we might have to extract 
{{ConversionResolver}}, because:
# {{Generic/SpecificData}} are public classes, and they are not final. 
Applications may have extended from them, which makes it hard to create them 
(especially their children) on the fly, which is necessary if we want to add 
conversions when writing/reading children of {{SpecificRecordBase}}. On the 
other hand, {{ConversionResolver}} is a new class, we can define it as final, 
which make it easy to create objects.
# creating {{Generic/SpecificData}} object in specific datum reader/writer is 
confusing, especially when the reader/writer already has a {{data}} member. 
Using {{ConversionResolver}} makes code much clearer.
# Actually we don't have to mark those add/get conversion methods in 
{{GenericData}} as deprecated. They are still useful and provide a convenient 
way for applications to manage conversions.

What are you opinions on this, [~cutting] and [~tomwhite]?

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(Paren

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460722#comment-15460722
 ] 

Yibing Shi commented on AVRO-1891:
--

{quote}
I don't see the advantage of adding ConversionResolver. We'll be stuck with a 
bunch of deprecated methods for a number of releases or we break things 
incompatibly. Does it add any expressive power, or is it just done for 
cleanliness?
{quote}

The idea of my patch is to extract find out the conversions used in a child of 
{{SpecificRecordBase}} and merged them with those added to the 
{{Generic/SpecificData}} in datum reader/writer so that the record fields can 
be processed properly without users having to manually adding the conversions. 
I extracted {{ConversionResolver}} out just to avoid creating heavy 
{{Generic/SpecificData}} objects. [~tomwhite] mentioned that we could make 
{{ConversionResolver}} non-public. I am thinking about making it an inner class 
of GenericData. This way, we might be able to avoid exposing 
{{ConversionResolver}} and marking a bunch of methods deprecated.
I will upload another patch later.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.Pa

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460708#comment-15460708
 ] 

Yibing Shi commented on AVRO-1891:
--

{quote}
although this has the effect of creating another class that doesn't really need 
to be public
{quote}
I will try to make this class non-public.

{quote}
The modifications to support union look fairly minor. Are any further changes 
needed for arrays and maps?
{quote}
We need some changes to support writing records and record fields. The patch 
has already covered this. There should be no further change needed for arrays 
and maps.

{quote}
It looks like RecordBuilderBase has an incompatible change to defaultValue().
{quote}
I deleted the previous {{defaultValue}} method, thinking it is only needed in 
by the generated code and now the generated code doesn't use it any more. I 
forgot this was a protected method and there has been a release that exposes 
this change. Deleting it would bring a backward compatibility problem. I will 
add it back. Thanks for pointing this out.
And sorry for uploading a broken patch. I will try to correct it in next 
version.

{quote}
We need a test for multiple nested logical types with generated code, as well 
as ones for arrays and maps of logical types. (BTW the v3 patch doesn't apply 
to latest master for me.)
{quote}
Yes, I am working on this. I am thinking of changing class 
{{TestRecordWithLogicalTypes}} to make it contain more types of nested types.

{quote}
The idea was that you'd always include all known conversions (the same set that 
SpecificCompiler uses)
{quote}
Users can pass a parameter to {{SpecificCompiler}} to control whether it uses 
{{BidDecimal}} or {{ByteBuffer}} for decimal types. That is, the 
{{DecimalConversion}} is not always used in generated code. (This is to keep 
backwards compatibility.  We may not be able to assume {{DecimalConversion}} is 
always used.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp

[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-02 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.3.patch

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch, AVRO-1891.yshi.3.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflec

[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-02 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.2.patch

Attach another patch that fixes the reading problems

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch, 
> AVRO-1891.yshi.2.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j

[jira] [Updated] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-01 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1891:
-
Attachment: AVRO-1891.yshi.1.patch

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch, AVRO-1891.yshi.1.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMet

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-09-01 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15457575#comment-15457575
 ] 

Yibing Shi commented on AVRO-1891:
--

{quote}
When resolveUnion is called there is no SpecificRecordBase object to get the 
conversions from.
{quote}
We can add another {{resolveUnion}} method that takes an extra parameter that 
holds all the conversions, and overwrites the {{resolveUnion}} method in 
{{SpecificDatumWriter}}.

I will my patch (not completed yet) as well. Please help have a look. The idea 
is to overloads the {{writeRecord}} in {{SepecificDatumWriter}} and utilize the 
conversions used in the generated specific record  for writing data. A similar 
thing is done in {{SepcificDatumReader}} as well.

BTW, if we go with your patch, we will have to check the {{SpecificCompiler}} 
parameters to determine whether we should include {{DecimalConversion}} to the 
variable Model$. 

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
> Attachments: AVRO-1891.patch
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-08-31 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453755#comment-15453755
 ] 

Yibing Shi commented on AVRO-1891:
--

{quote}
We could do this by adding a constructor to SpecificDatumWriter that takes a 
SpecificRecord, so it uses the SpecificData instance from the generated class 
(which should have the conversions added to it). 
{quote}

# At this moment, only the generated {{SpecificRecord}} contains a 
{{SpecificData}} member, and this member is private. {{SpecificRecord}} itself 
doesn't provide any method that returns a related {{SpecificData}}. Adding such 
a method is not backward compatible.
# We still need to consider the situation where a {{SpecificRecord}} is nested 
in another {{SpecificRecord}}, and these 2 records may use different set of 
conversions (for example one use decimal conversion while the other uses date 
conversion). {{SpecificDatumWriter/Reader}} need to process such a scenario.

Currently I am thinking of moving the conversion management function away from 
{{Specific/GenericData}} and put it into another class (say 
ConversionResolver), and pass its objects as a parameter to Specific/Generic 
datum readers/writers. But I haven't got time to carefully examine this idea. 
Will find some time this week to do this.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.jun

[jira] [Commented] (AVRO-1901) creating a record called "Exception" leads to generated code not compiling

2016-08-23 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432801#comment-15432801
 ] 

Yibing Shi commented on AVRO-1901:
--

[~radai], are you trying to define an error in your IDL? If yes, have you tried 
the keyword {{error}} as shown in [this 
doc|https://avro.apache.org/docs/1.8.1/idl.html#format_records]?

> creating a record called "Exception" leads to generated code not compiling
> --
>
> Key: AVRO-1901
> URL: https://issues.apache.org/jira/browse/AVRO-1901
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.7.7, 1.8.1, 1.9.0
>Reporter: radai rosenblatt
>
> consider the following *.avsc file:
> {code:title=Exception.avsc}
> {
>   "type": "record",
>   "name": "Exception",
>   "namespace": "com.acme",
>   "fields" : [
> {"name": "value", "type": "int"}
>   ],
>   "doc": "Will not submit to your fascist naming conventions"
> }
> {code}
> the generated java code will contain:
> {code:title=Exception.java}
> ...
> @override
> public Exception build() {
>try {
>   ...
>   return record;
>} catch (Exception e) { //boom
>   throw new org.apache.avro.AvroRuntimeException(e);
>}
> }
> {code}
> which fails to compile because Exception refers to the generated class (which 
> does not extend Throwable)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-08-08 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411921#comment-15411921
 ] 

Yibing Shi commented on AVRO-1891:
--

[~rdblue], [~busbey], could you please help confirm whether we need support 
embedded complex type in specific records? Such as:
{code}
union { array, map }
{code}

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorI

[jira] [Commented] (AVRO-1895) DeepCopy does not work with logical types

2016-08-07 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410955#comment-15410955
 ] 

Yibing Shi commented on AVRO-1895:
--

The workaround in AVRO-1891 may not help here. DeepCopy hasn't applied any 
logical type conversion logic yet. 
That said, I suggest we hold on do anything on this till we decide how to 
manage the conversions for specific records in AVRO-1891 and AVRO-1894.

> DeepCopy does not work with logical types
> -
>
> Key: AVRO-1895
> URL: https://issues.apache.org/jira/browse/AVRO-1895
> Project: Avro
>  Issue Type: Improvement
>  Components: logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>
> AvroSchema is taken from a compiled avsc file which contains a decimal field.
> {code}
> AvroSchema.Builder builder = AvroSchema.newBuilder();
> BigDecimal bd = new BigDecimal(new BigInteger("155"), 3);
> campaignBuilder.setDecimalField(bd);
> AvroSchema source = builder.build();
> //This line causes an exception
> AvroSchema.Builder builder1 = AvroSchema.newBuilder(source);
> {code}
> Exception:
> {code}
> InvocationTargetException: java.math.BigDecimal cannot be cast to 
> java.nio.ByteBuffer
> {code}
> The same failure happens with GenericData as well:
> {code}
> GenericRecord copy = GenericData.get().deepCopy(AvroSchema.getClassSchema(), 
> source);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1894) GenericData does not add Logical Type conversions by default

2016-08-07 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410949#comment-15410949
 ] 

Yibing Shi commented on AVRO-1894:
--

How about add all the default conversions into {{SpecificData}} but not 
{{GenericData}}? We have applied a bunch of default conversions to 
{{SpecificRecord}} in {{SpecificCompiler}}. Does it make sense the apply the 
same set of conversions to {{SpecificData}}?
As for {{GenericData}}, adding a builder class is a good idea. We can cache the 
created {{GenericData}} instances to reduce memory usage (reuse the same 
instance if a same bunch of conversions are passed in).

> GenericData does not add Logical Type conversions by default
> 
>
> Key: AVRO-1894
> URL: https://issues.apache.org/jira/browse/AVRO-1894
> Project: Avro
>  Issue Type: Improvement
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Taras Bobrovytsky
>
> In order to support Decimal fields, the following line needs to be added:
> {code}
> new GenericData().addLogicalTypeConversion(new 
> Conversions.DecimalConversion());
> {code}
> It would be more convenient if logical type conversions were added by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-08-07 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410947#comment-15410947
 ] 

Yibing Shi commented on AVRO-1891:
--

Things can become even more complicated. AVRO supports multi-dimensional array, 
and any complex type can be included in another complex type. This makes it 
necessary for specific compiler to check all the levels of an idl definition 
and find out all the used conversions. Currently the conversions are stored on 
a per field basis. This may not be suitable for the scenarios mentioned above. 
Instead, I propose the store all the conversions (from any level) in a set so 
that application can use it when necessary.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTest

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-08-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406991#comment-15406991
 ] 

Yibing Shi commented on AVRO-1891:
--

Hi [~rdblue], thanks for the reply!
I agree that we need to modify some methods in SpecificDatumReader/Writer to 
use conversions contained in SepcificRecord.
Problem is that currently the {{SpecificCompiler}} doesn't generate conversions 
for complex type fields:

https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L647-L672

The complex types themselves don't contain a "logical type". Instead, it is 
their element types that can contain a logical type. We may need to modify the 
compiler to include these scenarios.

I have a quick thought about this, ARRAY and MAP type are OK. They contain only 
one element type. But for UNION, it is possible that there are multiple element 
types in it. [The current {{Conversion}} 
array|https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L647-L672]
 in {{SpecificRecord}} cannot hold this. I was considering changing it as 
{{java.util.List>[] conversions}}, but looks like 
this is a non compatible change. Now I tend to add another field specially for 
union fields (need to change {{getConversion}} to determine which field to look 
at -- bad for performance}}. What do you think about this?

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4Clas

[jira] [Comment Edited] (AVRO-1891) Generated Java code fails with union containing logical type

2016-08-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405931#comment-15405931
 ] 

Yibing Shi edited comment on AVRO-1891 at 8/3/16 1:52 PM:
--

Had a quick look at this. The scope of this problem is bigger than I expected. 
It looks like that current {{SpecificCompiler}} and 
{{SpecificDatumWriter/Reader}} haven't considered the logical type conversion 
in some complex types (array, map, union) yet.  Will try to find a way to fix 
this problem.


was (Author: yibing):
Had a quick look at this. The scope of this problem is bigger than I expected. 
It looks like that current {{SpecificCompiler}} and 
{{SpecificDatumWriter/Reader}} haven't considered the logical type conversion 
in some complex types (array, map, union) yet.  Will try to find a good way to 
fix this problem.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUn

[jira] [Comment Edited] (AVRO-1891) Generated Java code fails with union containing logical type

2016-08-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405931#comment-15405931
 ] 

Yibing Shi edited comment on AVRO-1891 at 8/3/16 1:46 PM:
--

Had a quick look at this. The scope of this problem is bigger than I expected. 
It looks like that current {{SpecificCompiler}} and 
{{SpecificDatumWriter/Reader}} haven't considered the logical type conversion 
in some complex types (array, map, union) yet.  Will try to find a good way to 
fix this problem.


was (Author: yibing):
Had a quick look at this. The scope of this problem is bigger than I expected. 
It looks like that current {{SpecificCompiler}} and 
{{SpecificDatumWriter/Reader}} haven't considered the conversion of logical 
types yet. Will try to find a good way to fix this problem.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.in

[jira] [Commented] (AVRO-1891) Generated Java code fails with union containing logical type

2016-08-03 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405931#comment-15405931
 ] 

Yibing Shi commented on AVRO-1891:
--

Had a quick look at this. The scope of this problem is bigger than I expected. 
It looks like that current {{SpecificCompiler}} and 
{{SpecificDatumWriter/Reader}} haven't considered the conversion of logical 
types yet. Will try to find a good way to fix this problem.

> Generated Java code fails with union containing logical type
> 
>
> Key: AVRO-1891
> URL: https://issues.apache.org/jira/browse/AVRO-1891
> Project: Avro
>  Issue Type: Bug
>  Components: java, logical types
>Affects Versions: 1.8.1
>Reporter: Ross Black
>Priority: Blocker
> Fix For: 1.8.3
>
>
> Example schema:
> {code}
> {
>   "type": "record",
>   "name": "RecordV1",
>   "namespace": "org.brasslock.event",
>   "fields": [
> { "name": "first", "type": ["null", {"type": "long", 
> "logicalType":"timestamp-millis"}]}
>   ]
> }
> {code}
> The avro compiler generates a field using the relevant joda class:
> {code}
> public org.joda.time.DateTime first
> {code}
> Running the following code to perform encoding:
> {code}
> final RecordV1 record = new 
> RecordV1(DateTime.parse("2016-07-29T10:15:30.00Z"));
> final DatumWriter datumWriter = new 
> SpecificDatumWriter<>(record.getSchema());
> final ByteArrayOutputStream stream = new ByteArrayOutputStream(8192);
> final BinaryEncoder encoder = 
> EncoderFactory.get().directBinaryEncoder(stream, null);
> datumWriter.write(record, encoder);
> encoder.flush();
> final byte[] bytes = stream.toByteArray();
> {code}
> fails with the exception stacktrace:
> {code}
>  org.apache.avro.AvroRuntimeException: Unknown datum type 
> org.joda.time.DateTime: 2016-07-29T10:15:30.000Z
> at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:741)
> at 
> org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:293)
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:706)
> at 
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at 
> org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at 
> org.brasslock.avro.compiler.GeneratedRecordTest.shouldEncodeLogicalTypeInUnion(GeneratedRecordTest.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
> at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> at

[jira] [Commented] (AVRO-1890) Java objects compiled from AVDL disregard default values

2016-07-30 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400569#comment-15400569
 ] 

Yibing Shi commented on AVRO-1890:
--

[~ryonday], it is recommended to use record builder to create the specific 
records. 
{code}
fasdf.Wat.newBuilder().build()
{code}
This builder should be able to assign the default values for the fields.


> Java objects compiled from AVDL disregard default values
> 
>
> Key: AVRO-1890
> URL: https://issues.apache.org/jira/browse/AVRO-1890
> Project: Avro
>  Issue Type: Bug
>Reporter: Ryon Day
>Priority: Minor
>
> Given the schema
> {code}
> @namespace("fasdf")
> protocol foo {
> enum Domain {
> COM,
> ORG,
> BIZ,
> MIL
> }
> record Wat {
> Domain domain = "COM";
> string foo = "asfdasdf";
> }
> }
> {code}
> The resulting {{Wat}} Java class does not use the defined default values:
> {code}
> @Test
> public void asdfasdf() throws Exception {
> fasdf.Wat x = new Wat();
> System.out.print(x);
> }
> {code}
> This test results in:
> {code}
> {"domain": null, "foo": null}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1817) Allow enums to be "promoted" to strings

2016-07-28 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398756#comment-15398756
 ] 

Yibing Shi commented on AVRO-1817:
--

[~busbey], I am not sure whether this task is feasible, especially when binary 
encoder/writer is used.
AFAICS, in {{GenericDatumWriter}}, an enum value is written as its offset in 
schema enum symbols list.
{code}
  protected void writeEnum(Schema schema, Object datum, Encoder out)
throws IOException {
if (!data.isEnum(datum))
  throw new AvroTypeException("Not an enum: "+datum);
out.writeEnum(schema.getEnumOrdinal(datum.toString()));
  }
{code}

If {{BinaryEncoder}} is used, this offset is write *through* to the data file, 
without any flags added to it.
{code}
  public void writeEnum(int e) throws IOException {
this.writeInt(e);
  }
{code}

In datum reader and decoder, it is very hard, if not impossible, to figure out 
whether the data to read is actually an enum or an actual string. Things can be 
even more complicated if unicode string is considered.

> Allow enums to be "promoted" to strings
> ---
>
> Key: AVRO-1817
> URL: https://issues.apache.org/jira/browse/AVRO-1817
> Project: Avro
>  Issue Type: Improvement
>  Components: java, spec
>Reporter: Michael Overmeyer
>Priority: Minor
>
> We should consider adding a resolution rule that can promote an enum to a 
> string using the enum's symbol.
> I have an Avro schema that has a field with an enum type. However, I have 
> realized that an enum is not the type I actually wanted. I would much rather 
> have the type of the field be a string. I went to change this, but of course 
> this type of change (enum -> string) is not within the bounds of Avro's 
> schema evolution. Therefore a reader with this changed schema is not be able 
> to read an object written with the old schema.
> For example, if the writer schema was:
> enum Colour {
>   RED, YELLOW, GREEN 
> }
> protocol stoplight {
>   Colour colour;
> }
> And the reader schema was:
> protocol stoplight {
>   string colour;
> }
> Then when you access the colour field of your object, you get the string 
> representation of the enum value's symbol .
> For example, Colour.RED => "RED", Colour.YELLOW => "YELLOW", Colour.GREEN => 
> "GREEN"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-21 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388623#comment-15388623
 ] 

Yibing Shi commented on AVRO-1877:
--

Thanks for the tip, [~busbey]!

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Blocker
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements

2016-07-17 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi reassigned AVRO-1883:


Assignee: Yibing Shi

> Schema validator cannot find broken backwards compatibility in Union type 
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   },
>   {
> "name": "address",
> "type": "string"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the 
> {{address}} field of {{rec1}} in the second one is not nullable. However, if 
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new 
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements

2016-07-17 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1883:
-
Status: Patch Available  (was: Open)

> Schema validator cannot find broken backwards compatibility in Union type 
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   },
>   {
> "name": "address",
> "type": "string"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the 
> {{address}} field of {{rec1}} in the second one is not nullable. However, if 
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new 
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements

2016-07-17 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1883:
-
Attachment: AVRO-1883.1.patch

> Schema validator cannot find broken backwards compatibility in Union type 
> elements
> --
>
> Key: AVRO-1883
> URL: https://issues.apache.org/jira/browse/AVRO-1883
> Project: Avro
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1883.1.patch
>
>
> Consider below 2 schemas:
> *Schema 1*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> *Schema 2*:
> {noformat}
> [
>   {
> "type": "record",
> "name": "rec1",
> "fields": [
>   {
> "name": "age",
> "type": "long"
>   },
>   {
> "name": "address",
> "type": "string"
>   }
> ]
>   },
>   {
> "type": "record",
> "name": "rec2",
> "fields": [
>   {
> "name": "username",
> "type": "string"
>   }
> ]
>   }
> ]
> {noformat}
> The {{rec1}} field in these 2 unions are not compatible, because the 
> {{address}} field of {{rec1}} in the second one is not nullable. However, if 
> we check them with validate like below, validator doesn't return any error:
> {code}
> final SchemaValidator backwardValidator = new 
> SchemaValidatorBuilder().canReadStrategy().validateLatest();
> final Schema schema1 = new Schema.Parser().parse(schema1Str);
> final Schema schema2 = new Schema.Parser().parse(schema2Str);
> backwardValidator.validate(schema2, Arrays.asList(schema1));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (AVRO-1883) Schema validator cannot find broken backwards compatibility in Union type elements

2016-07-17 Thread Yibing Shi (JIRA)
Yibing Shi created AVRO-1883:


 Summary: Schema validator cannot find broken backwards 
compatibility in Union type elements
 Key: AVRO-1883
 URL: https://issues.apache.org/jira/browse/AVRO-1883
 Project: Avro
  Issue Type: Bug
Affects Versions: 1.8.1
Reporter: Yibing Shi
Priority: Critical


Consider below 2 schemas:

*Schema 1*:
{noformat}
[
  {
"type": "record",
"name": "rec1",
"fields": [
  {
"name": "age",
"type": "long"
  }
]
  },
  {
"type": "record",
"name": "rec2",
"fields": [
  {
"name": "username",
"type": "string"
  }
]
  }
]
{noformat}

*Schema 2*:
{noformat}
[
  {
"type": "record",
"name": "rec1",
"fields": [
  {
"name": "age",
"type": "long"
  },
  {
"name": "address",
"type": "string"
  }
]
  },
  {
"type": "record",
"name": "rec2",
"fields": [
  {
"name": "username",
"type": "string"
  }
]
  }
]
{noformat}

The {{rec1}} field in these 2 unions are not compatible, because the 
{{address}} field of {{rec1}} in the second one is not nullable. However, if we 
check them with validate like below, validator doesn't return any error:
{code}
final SchemaValidator backwardValidator = new 
SchemaValidatorBuilder().canReadStrategy().validateLatest();
final Schema schema1 = new Schema.Parser().parse(schema1Str);
final Schema schema2 = new Schema.Parser().parse(schema2Str);
backwardValidator.validate(schema2, Arrays.asList(schema1));
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-10 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1877:
-
Attachment: AVRO-1877.3.patch

Upload a new patch with code refactored 

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch, AVRO-1877.3.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-10 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369995#comment-15369995
 ] 

Yibing Shi edited comment on AVRO-1877 at 7/10/16 11:29 PM:


{quote}
Honestly, I like the old code better.
{quote}
The problem with the old code is that it checks for conversions twice: once in 
{{javaUnbox}}, once again in {{javaType}}
That said, I have a look at the code again and the extra test for UNION does 
smells bad. I will refactor the code a bit more and upload another patch here.

{quote}
As for enableDecimalLogicalType. If truly the intent is to use converted 
logical types at all, then this should be renamed to enableLogicalType. It's 
confusing to say it's for Decimal but mean it for all.
{quote}
As said before, {{enableDecimalLogicalType}} should only applies to decimal 
type, not the date/time logical types, which should always use Joda/Java 
date/time classes instead of raw types if these classes are present.
Please have a look at the unit tests added in {{TestSpecificCompiler}} 
(testJavaTypeWithDecimalLogicalTypeDisabled/Enabled). They clearly show the 
intent of {{enableDecimalLogicalType}}

{quote}
More importantly, after applying your AVRO-1847, if you have date or 
timestamp-millis without setting enableDecmialLogicalType, the compiled 
SpecificRecord code will NOT compile.
{quote}
This problem is caused because the problem tracked in this JIRA (AVRO-1877). 
Please try apply AVRO-1877 on top of AVRO-1847 and see whether it fixes the 
problem for you.


was (Author: yibing):
{quote}
Honestly, I like the old code better.
{quote}
The problem with the old code is that it checks for conversions twice: once in 
{{javaUnbox}}, once again in {{javaType}}
That said, I have a look at the code again and the extra test for UNION does 
smells bad. I will refactor the code a bit more and upload another patch here.

{quote}
As for enableDecimalLogicalType. If truly the intent is to use converted 
logical types at all, then this should be renamed to enableLogicalType. It's 
confusing to say it's for Decimal but mean it for all.
{quote}
As said before, {{enableDecimalLogicalType}} should only applies to decimal 
type, not the date/time logical types, which should always use Joda/Java 
date/time classes instead of raw types if these classes are present.

{quote}
More importantly, after applying your AVRO-1847, if you have date or 
timestamp-millis without setting enableDecmialLogicalType, the compiled 
SpecificRecord code will NOT compile.
{quote}
This problem is caused because the problem tracked in this JIRA (AVRO-1877). 
Please try apply AVRO-1877 on top of AVRO-1847 and see whether it fixes the 
problem for you.

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-10 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369995#comment-15369995
 ] 

Yibing Shi commented on AVRO-1877:
--

{quote}
Honestly, I like the old code better.
{quote}
The problem with the old code is that it checks for conversions twice: once in 
{{javaUnbox}}, once again in {{javaType}}
That said, I have a look at the code again and the extra test for UNION does 
smells bad. I will refactor the code a bit more and upload another patch here.

{quote}
As for enableDecimalLogicalType. If truly the intent is to use converted 
logical types at all, then this should be renamed to enableLogicalType. It's 
confusing to say it's for Decimal but mean it for all.
{quote}
As said before, {{enableDecimalLogicalType}} should only applies to decimal 
type, not the date/time logical types, which should always use Joda/Java 
date/time classes instead of raw types if these classes are present.

{quote}
More importantly, after applying your AVRO-1847, if you have date or 
timestamp-millis without setting enableDecmialLogicalType, the compiled 
SpecificRecord code will NOT compile.
{quote}
This problem is caused because the problem tracked in this JIRA (AVRO-1877). 
Please try apply AVRO-1877 on top of AVRO-1847 and see whether it fixes the 
problem for you.

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-10 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369646#comment-15369646
 ] 

Yibing Shi commented on AVRO-1877:
--

[~mwong], I have uploaded another patch, but fixes the problem in a different 
way than your suggestion. Please help have another look.

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-10 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1877:
-
Attachment: AVRO-1877.2.patch

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch, AVRO-1877.2.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-10 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369631#comment-15369631
 ] 

Yibing Shi commented on AVRO-1877:
--

{quote}
The way you call javaType from javaUnbox in this latest patch is not correct. 
For example, if you have a {{union \{null, double\}}}, javaType() will turn it 
into a boxed java.lang.Double that is nullable. Your changes to javaUnbox will 
then incorrectly convert that to a primitive type double. My suggestion is to 
revert to the old code and maybe refactor the duplicate SPECIFIC conversion 
part into a separate method.
{quote}
Very good point! I will correct this. Thanks for the idea!

{quote}
The problem is that enableDecimalLogicalType only applies to the Decimal 
logical type. For the other logical types, such as date or timestamp-millis, 
they would be converted even if you didn't enableDecimalLogicalType. 
{quote}
This is done on purpose. It is to keep backwards compatibility with 1.8.0. When 
date/time logical types were added in 1.8.0, they always use logical types 
instead of backing raw types in SpecificRecord. Please see [this 
comment|https://issues.apache.org/jira/browse/AVRO-1847?focusedCommentId=15327757&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15327757]
 for more details.

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-08 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367651#comment-15367651
 ] 

Yibing Shi edited comment on AVRO-1877 at 7/8/16 1:14 PM:
--

Hi [~rdblue], it was my fault. I accidentally deleted a part of method 
{{javaUnbox}}. I am not sure how it happened though. 
Before AVRO-1847, the javaUnbox method is like below:
{code}
  public String javaType(Schema schema) {
Conversion conversion = SPECIFIC
.getConversionFor(schema.getLogicalType());
if (conversion != null) {
  return conversion.getConvertedType().getName();
}

switch (schema.getType()) {
..
{code}

However, AVRO-1847 somehow deleted the conversion part and made it like below:
{code}
  public String javaType(Schema schema) {
switch (schema.getType()) {
..
{code}

As a result, for logical type timestamp_ms, date and time_ms, method 
{{javaUnbox}} returns "int" instead of JODA classes, which breaks the generated 
Specific Record.

I submitted a patch in this JIRA. Instead of using the original code, I rely on 
method {{javaType}} to do the conversion to avoid maintain the same code in 2 
places, especially when now we have more and more parameters to control whether 
to use upper level classes for a logical type. I also added 2 unit tests in 
this patch to cover this function. Please let me know whether you think this is 
appropriate or not.



was (Author: yibing):
Hi [~rdblue], it was my fault. I accidentally deleted a part of method 
{{javaUnbox}}. I am not sure how it happened though. 
Before AVRO-1847, the javaUnbox method is like below:
{code}
  public String javaType(Schema schema) {
Conversion conversion = SPECIFIC
.getConversionFor(schema.getLogicalType());
if (conversion != null) {
  return conversion.getConvertedType().getName();
}

switch (schema.getType()) {
..
{code}

However, AVRO-1847 somehow deleted the conversion part and made it like below:
{code}
  public String javaType(Schema schema) {
switch (schema.getType()) {
..
{code}

I submitted a patch in this JIRA. Instead of using the original code, I rely on 
method {{javaType}} to do the conversion to avoid maintain the same code in 2 
places, especially when now we have more and more parameters to control whether 
to use upper level classes for a logical type. I also added 2 unit tests in 
this patch to cover this function. Please let me know whether you think this is 
appropriate or not.


> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-08 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367651#comment-15367651
 ] 

Yibing Shi commented on AVRO-1877:
--

Hi [~rdblue], it was my fault. I accidentally deleted a part of method 
{{javaUnbox}}. I am not sure how it happened though. 
Before AVRO-1847, the javaUnbox method is like below:
{code}
  public String javaType(Schema schema) {
Conversion conversion = SPECIFIC
.getConversionFor(schema.getLogicalType());
if (conversion != null) {
  return conversion.getConvertedType().getName();
}

switch (schema.getType()) {
..
{code}

However, AVRO-1847 somehow deleted the conversion part and made it like below:
{code}
  public String javaType(Schema schema) {
switch (schema.getType()) {
..
{code}

I submitted a patch in this JIRA. Instead of using the original code, I rely on 
method {{javaType}} to do the conversion to avoid maintain the same code in 2 
places, especially when now we have more and more parameters to control whether 
to use upper level classes for a logical type. I also added 2 unit tests in 
this patch to cover this function. Please let me know whether you think this is 
appropriate or not.


> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-07 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1877:
-
Status: Patch Available  (was: Open)

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-07 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1877:
-
Attachment: AVRO-1877.1.patch

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
> Attachments: AVRO-1877.1.patch
>
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-07 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1877:
-
Hadoop Flags:   (was: Incompatible change)

> AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler
> --
>
> Key: AVRO-1877
> URL: https://issues.apache.org/jira/browse/AVRO-1877
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Yibing Shi
>Priority: Blocker
>
> AVRO-1847 accidentally removes the logical type conversion logic in method 
> {{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
> specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (AVRO-1877) AVRO-1847 accidentally breaks javaUnbox method in SpecificCompiler

2016-07-07 Thread Yibing Shi (JIRA)
Yibing Shi created AVRO-1877:


 Summary: AVRO-1847 accidentally breaks javaUnbox method in 
SpecificCompiler
 Key: AVRO-1877
 URL: https://issues.apache.org/jira/browse/AVRO-1877
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.9.0
Reporter: Yibing Shi
Priority: Blocker


AVRO-1847 accidentally removes the logical type conversion logic in method 
{{SpecificCompiler.javaUnbox}}, which breaks the data time type compiling in 
specific compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8

2016-07-05 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362617#comment-15362617
 ] 

Yibing Shi commented on AVRO-1811:
--

{quote}
What's the current behavior if we try to turn a SpecificRecord that contains 
either representation into a GenericRecord
{quote}
I believe it depends on how to do the conversion.
If we first serialize the SpecificRecord (no matter using high/low level 
representation), and then deserialize it into a GenericRecord, it should work 
fine , but the logical type fields will use low level representations in 
GenericRecord, unless conversion objects are explicitly added to GenericRecord 
before reading the underlying data.
If we use {{GenericRecord.deepCopy}} directly to copy a SpecificRecord with 
high level logical type representations (BigDecimal etc.), I believe it would 
fail, because {{GenericData.deepCopy}} doesn't understand the high level 
representations at all. A CCE would be thrown out. Copying a SpecificRecord 
with low level representation should be fine because it is just the same as 
before Logical Type is adopted in.
Actually, we may also face problems when deepCopy a GenericData to 
SpecificData. The setField method in SpecificData needs a value that matches 
field type. If the logical type field uses high level representation, I expect 
this copying would fail, because deepCopy at this moment only returns low level 
representations.
I haven't tested these behaviours though. Will do some test when I have time.

The patch uploaded here should be able to solve the immediate problem. Thanks 
for your acceptance of it as a short-term solution. I am just afraid we may 
face other similar problems in other scenarios. Maybe we should create another 
JIRA to track the wider "convertible" values problem?

> SpecificData.deepCopy() cannot be used if schema compiler generated Java 
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them 
> generate fields of type {{string}} with the Java standard {{String}} type, 
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific 
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order 
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every 
> {{string}} field in a schema like so:
> {code}
> {
>   "name": "some_string",
>   "doc": "a field that is guaranteed to compile to java.lang.String",
>   "type": [
> "null",
> {
>   "type": "string",
>   "avro.java.string": "String"
> }
>   ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by 
> this annotation by volume; for teams using heterogenous clients, they may to 
> want to avoid  Java-specific annotation in their schema files, or may not 
> think to use it unless there exist Java exploiters of the schema at the time 
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects  
> using the {{SpecificCompiler}}'s string type selection. This option actually 
> alters the schema carried by the object's {{SCHEMA$}} field to have the above 
> annotation in it, ensuring that when used by the Java API, the String type 
> will be used. 
> Unfortunately, this method is not interoperable with GenericRecords created 
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the 
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the 
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out 
> of the {{GenericRecord}} 
> There is a unit test that demonstrate this 
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are lit

[jira] [Commented] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8

2016-07-04 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361264#comment-15361264
 ] 

Yibing Shi commented on AVRO-1811:
--

Hi [~rdblue], I have another consideration about this problem. Above change is 
ugly. It only fixes the record copy problem. We may still face the same problem 
in other scenarios. Maybe we should change the specific compiler instead to 
make the {{setField}} method be able to accept different types of values (for 
example, can accept both String and Utf8 for a string field)?

> SpecificData.deepCopy() cannot be used if schema compiler generated Java 
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them 
> generate fields of type {{string}} with the Java standard {{String}} type, 
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific 
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order 
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every 
> {{string}} field in a schema like so:
> {code}
> {
>   "name": "some_string",
>   "doc": "a field that is guaranteed to compile to java.lang.String",
>   "type": [
> "null",
> {
>   "type": "string",
>   "avro.java.string": "String"
> }
>   ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by 
> this annotation by volume; for teams using heterogenous clients, they may to 
> want to avoid  Java-specific annotation in their schema files, or may not 
> think to use it unless there exist Java exploiters of the schema at the time 
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects  
> using the {{SpecificCompiler}}'s string type selection. This option actually 
> alters the schema carried by the object's {{SCHEMA$}} field to have the above 
> annotation in it, ensuring that when used by the Java API, the String type 
> will be used. 
> Unfortunately, this method is not interoperable with GenericRecords created 
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the 
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the 
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out 
> of the {{GenericRecord}} 
> There is a unit test that demonstrate this 
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion 
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be 
> cast to java.lang.String}}
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8

2016-07-04 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1811:
-
Attachment: AVRO-1811.1.patch

> SpecificData.deepCopy() cannot be used if schema compiler generated Java 
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them 
> generate fields of type {{string}} with the Java standard {{String}} type, 
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific 
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order 
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every 
> {{string}} field in a schema like so:
> {code}
> {
>   "name": "some_string",
>   "doc": "a field that is guaranteed to compile to java.lang.String",
>   "type": [
> "null",
> {
>   "type": "string",
>   "avro.java.string": "String"
> }
>   ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by 
> this annotation by volume; for teams using heterogenous clients, they may to 
> want to avoid  Java-specific annotation in their schema files, or may not 
> think to use it unless there exist Java exploiters of the schema at the time 
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects  
> using the {{SpecificCompiler}}'s string type selection. This option actually 
> alters the schema carried by the object's {{SCHEMA$}} field to have the above 
> annotation in it, ensuring that when used by the Java API, the String type 
> will be used. 
> Unfortunately, this method is not interoperable with GenericRecords created 
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the 
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the 
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out 
> of the {{GenericRecord}} 
> There is a unit test that demonstrate this 
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion 
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be 
> cast to java.lang.String}}
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8

2016-07-04 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1811:
-
Affects Version/s: 1.8.1
   Status: Patch Available  (was: Open)

> SpecificData.deepCopy() cannot be used if schema compiler generated Java 
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1, 1.8.0
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
> Attachments: AVRO-1811.1.patch
>
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them 
> generate fields of type {{string}} with the Java standard {{String}} type, 
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific 
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order 
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every 
> {{string}} field in a schema like so:
> {code}
> {
>   "name": "some_string",
>   "doc": "a field that is guaranteed to compile to java.lang.String",
>   "type": [
> "null",
> {
>   "type": "string",
>   "avro.java.string": "String"
> }
>   ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by 
> this annotation by volume; for teams using heterogenous clients, they may to 
> want to avoid  Java-specific annotation in their schema files, or may not 
> think to use it unless there exist Java exploiters of the schema at the time 
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects  
> using the {{SpecificCompiler}}'s string type selection. This option actually 
> alters the schema carried by the object's {{SCHEMA$}} field to have the above 
> annotation in it, ensuring that when used by the Java API, the String type 
> will be used. 
> Unfortunately, this method is not interoperable with GenericRecords created 
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the 
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the 
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out 
> of the {{GenericRecord}} 
> There is a unit test that demonstrate this 
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion 
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be 
> cast to java.lang.String}}
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8

2016-07-04 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360955#comment-15360955
 ] 

Yibing Shi commented on AVRO-1811:
--

Hi [~rdblue], this problem may not be very easy. It happens when deep copying 
an object that uses {{Utf8}} as string representation into a specific record 
which uses {{java.lang.String}} instead. We can simplify the issue with below 
unit test:
{code}
  @Test
  public void testStringDeepCopy() throws IOException {
FooBarSpecificRecord specificRecord = FooBarSpecificRecord.newBuilder()
.setId(1)
.setName("test_record_specific")
.setNicknames(new ArrayList())
.setRelatedids(new ArrayList())
.build();

GenericRecordBuilder builder = new 
GenericRecordBuilder(specificRecord.getSchema());
GenericRecord genericRecord = builder
.set("id", 1)
.set("name", new Utf8("test_record_specific"))
.set("nicknames", new ArrayList())
.set("relatedids", new ArrayList())
.build();

FooBarSpecificRecord copiedFromGeneric = (FooBarSpecificRecord) 
SpecificData.get().deepCopy(
FooBarSpecificRecord.getClassSchema(), genericRecord);

assertEquals("Should get an equal record by deep copying the generic 
record",
specificRecord, copiedFromGeneric);
  }
{code}

This unit test fails in master branch with below exception:
{noformat}
java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to 
java.lang.String
at 
org.apache.avro.FooBarSpecificRecord.put(FooBarSpecificRecord.java:70)
at org.apache.avro.generic.GenericData.setField(GenericData.java:659)
at org.apache.avro.generic.GenericData.setField(GenericData.java:676)
at org.apache.avro.generic.GenericData.deepCopy(GenericData.java:1081)
at 
org.apache.avro.generic.TestGenericData.testStringDeepCopy(TestGenericData.java:450)
{noformat}

Looking at [the 
code|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java#L1079-L1081]:
{code}
  Object newValue = deepCopy(f.schema(),
 getField(value, name, pos, oldState));
  setField(newRecord, name, pos, newValue, newState);
{code}
Variable newValue is an {{Utf8}} object because {{getField}} of the generic 
record uses it for field "name", while {{setField}} method in 
{{FooBarSpecificRecord}} expects a {{java.lang.string}} as input parameter. 
This causes above exception.

Method "deepCopy" is defined with below signature:
{code}
  public  T deepCopy(Schema schema, T value) {
{code}
which makes the problem a bit complicated. It can only return an object of 
exactly the same type as input parameter "value". That is, if we pass in an 
{{Utf8}} object as "value", we cannot return a {{java.lang.String}} object. 

The only way I can find to solve this problem is to finish the conversion 
before calling {{setField}} when deep copying records. That is, change above 
code to something like below:
{code}
  Object newValue = deepCopy(f.schema(),
 getField(value, name, pos, oldState));
  if (STRING_TYPE_STRING.equals(f.schema().getProp(STRING_PROP))) {
if (! (newValue instanceof java.lang.String)) {
  newValue = newValue.toString();
}
  } else {
if (newValue instanceof java.lang.String) {
  newValue = new Utf8((String) newValue);
}
  }
  setField(newRecord, name, pos, newValue, newState);
{code}

I will submit a patch based on this thought. 

Further thoughts:
# [~ryonday], why did you have incompatible schemas in Specific Data and 
Generic Data in the unit test you provided 
[here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]?
 How did you generate the avsc files?
# [~rdblue], this problem reminds me of the Logical Types. A logical type field 
can have 2 representations: one uses raw type and the other uses upper level 
java types. For instance, for "decimal" type, we can use both ByteBuffer and 
BigDecimal. Do we need support copying between them?



> SpecificData.deepCopy() cannot be used if schema compiler generated Java 
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Ryon Day
>Priority: Critical
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them 
> generate fields of ty

[jira] [Assigned] (AVRO-1811) SpecificData.deepCopy() cannot be used if schema compiler generated Java objects with Strings instead of UTF8

2016-07-04 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi reassigned AVRO-1811:


Assignee: Yibing Shi

> SpecificData.deepCopy() cannot be used if schema compiler generated Java 
> objects with Strings instead of UTF8
> -
>
> Key: AVRO-1811
> URL: https://issues.apache.org/jira/browse/AVRO-1811
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Ryon Day
>Assignee: Yibing Shi
>Priority: Critical
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> When the Avro compiler creates Java objects, you have the option to have them 
> generate fields of type {{string}} with the Java standard {{String}} type, 
> for wide interoperability with existing Java applications and APIs.
> By default, however, the compiler outputs these fields in the Avro-specific 
> {{Utf8}} type, requiring frequent usage of the {{toString()}} method in order 
> for default domain objects to be used with the majority of Java libraries.
> There are two ways to get around this. The first is to annotate every 
> {{string}} field in a schema like so:
> {code}
> {
>   "name": "some_string",
>   "doc": "a field that is guaranteed to compile to java.lang.String",
>   "type": [
> "null",
> {
>   "type": "string",
>   "avro.java.string": "String"
> }
>   ]
> },
> {code}
> Unfortunately, long schemas containing many string fields can be dominated by 
> this annotation by volume; for teams using heterogenous clients, they may to 
> want to avoid  Java-specific annotation in their schema files, or may not 
> think to use it unless there exist Java exploiters of the schema at the time 
> the schema is proposed and written.
> The other solution to the problem is to compile the schema into Java objects  
> using the {{SpecificCompiler}}'s string type selection. This option actually 
> alters the schema carried by the object's {{SCHEMA$}} field to have the above 
> annotation in it, ensuring that when used by the Java API, the String type 
> will be used. 
> Unfortunately, this method is not interoperable with GenericRecords created 
> by libraries that use the _original_ schema.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> # Create a schema with several {{string}} fields.
> # Parse the schema using the standard Avro schema parser
> # Create Java domain objects for that schema ensuring usage of the 
> {{java.lang.String}} string type.
> # Create a message of some sort that ends up as a {{GenericRecord}} of the 
> original schema
> # Attempt to use {{SpecificData.deepCopy()}} to make a {{SpecificRecord}} out 
> of the {{GenericRecord}} 
> There is a unit test that demonstrate this 
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/1.8.0/src/test/java/com/ryonday/avro/test/v180/AvroDeepCopyTest.java]
> {panel}
> {panel:title=Expected Results|titleBGColor=#AD3|bgColor=#DDD}
> As the schemas are literally identical aside from string type, the conversion 
> should work (and does work for schema that are exactly identical).
> {panel}
> {panel:title=Actual Results|titleBGColor=#D55|bgColor=#DDD}
> {{ClassCastException}} with the message {{org.apache.avro.util.Utf8 cannot be 
> cast to java.lang.String}}
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1869) Decimal conversion from ByteBuffer ignores buffer position and limit

2016-06-28 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352944#comment-15352944
 ] 

Yibing Shi commented on AVRO-1869:
--

Thanks [~rdblue]!

> Decimal conversion from ByteBuffer ignores buffer position and limit
> 
>
> Key: AVRO-1869
> URL: https://issues.apache.org/jira/browse/AVRO-1869
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0, 1.8.1
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Fix For: 1.8.2
>
>
> The decimal conversion from a ByteBuffer currently copies the buffer contents 
> into an array, but then calls {{array}} on the buffer and uses the buffer's 
> internal array rather than the copied bytes. The result is that the 
> constructed Decimal value uses whatever the buffer contained without using 
> its position or limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-23 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346370#comment-15346370
 ] 

Yibing Shi commented on AVRO-1847:
--

[~busbey] and [~rdblue], thank you for your help on this!

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Fix For: 1.9.0, 1.8.2
>
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch, 
> AVRO-1847.8.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-19 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.8.patch

I forgot to apply the decimal type flag to 
{{SpecificCompiler.conversionInstance}}. Upload a new patch that corrects this 
problem, and adds 2 unit tests to cover this.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch, 
> AVRO-1847.8.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-18 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337885#comment-15337885
 ] 

Yibing Shi commented on AVRO-1847:
--

Thank you very much for your help, [~busbey]!
[~rdblue], please let me know if you have anything else to add.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-18 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337884#comment-15337884
 ] 

Yibing Shi commented on AVRO-1847:
--

Yes, I did add the test for UUID. It always use CharSequence as type.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-17 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.7.patch

Upload a new patch:
# Fixed all the javadoc errors pointed out by [~busbey]
# Use {{IllegalArugumentException}} instead of {{AvroRuntimeException}} in 
{{Conversions.convert...}} functions
# Add javadoc to {{GenericDatumWriter/Reader.convert}} to describe the 
behaviour change (different exception type is thrown out now). [~rdblue], would 
you please help confirm whether this is OK for you?

And as [~busbey] said, I have added below test cases in 
{{TestSpecificCompiler}}:
# testJavaTypeWithDecimalLogicalTypeEnabled
# testJavaTypeWithoutDecimalLogicalTypeEnabled
to test whether the decimal logical type enablement would affect other logical 
types. [~rdblue], please help confirm whether it is enough, or you want to add 
more tests?

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch, AVRO-1847.7.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-15 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.6.patch

Upload a new patch

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch, AVRO-1847.6.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-15 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331268#comment-15331268
 ] 

Yibing Shi edited comment on AVRO-1847 at 6/15/16 2:01 PM:
---

# I will add javadoc to those public functions 
# I will change the test case names.
# I will correct the help messages for compiler tool. Do you think it worth to 
use Apache Commons Options package to process the parameters? (maybe leave it 
for a later?)
# The null and "null" thing is a typo. Sorry for this stupid mistake. A null 
value will cause problems for SpecificRecordCompiler.
# I will make the parameter in compiler and maven plugin only affects decimal 
type, which will minimize the impact of this patch. Thanks for the suggestion!
# For the invariant problem in {{RecordBuilderBase.defaultValue(Field field, 
Conversion conversion)}}, I will add a checking to make sure that logical 
type is not null before call the {{convertToLogicalType}} method.
# For the convert to logical/raw type methods, I will change it a bit to have 
them return null if the original value passed in is null. 
# The focus here is the {{convert}} method in GenericDatumWriter. You are 
correct about the invariant here. But as [~rdblue] explains below, this used to 
be not a problem because the caller within GenericDatumWriter won't pass in any 
null pointer. I tends not to add the checking here. One reason is it is good 
for performance. The second reason is that a null schema or logical type is 
definitely a bad error, and I don't think any children would try to capture or 
process it.


was (Author: yibing):
# I will add javadoc to those public functions 
# I will change the test case names.
# I will correct the help messages for compiler tool. Do you think it worth to 
use Apache Commons Options package to process the parameters? (maybe leave it 
for a later?)
# The null and "null" thing is a typo. Sorry for this stupid mistake. A null 
value will cause problems for SpecificRecordCompiler.
# I will make the parameter in compiler and maven plugin only affects decimal 
type, which will minimize the impact of this patch. Thanks for the suggestion
# The focus here is the {{convert}} method in GenericDatumWriter. You are 
correct about the invariant here. But as [~rdblue] explains below, this used to 
be not a problem because the caller within GenericDatumWriter won't pass in any 
null pointer. I tends not to add the checking here. One reason is it is good 
for performance. The second reason is that a null schema or logical type is 
definitely a bad error, and I don't think any children would try to capture or 
process it.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-15 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331759#comment-15331759
 ] 

Yibing Shi commented on AVRO-1847:
--

Method {{GenericDatumWriter.convert}} is a protected method from the very 
beginning. Changing it to private would be a behaviour change. I prefer not to 
make this change. Please let me know you opinion.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-15 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331271#comment-15331271
 ] 

Yibing Shi commented on AVRO-1847:
--

I had a very busy day. Will work on this tonight and upload a new patch.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-15 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331268#comment-15331268
 ] 

Yibing Shi commented on AVRO-1847:
--

# I will add javadoc to those public functions 
# I will change the test case names.
# I will correct the help messages for compiler tool. Do you think it worth to 
use Apache Commons Options package to process the parameters? (maybe leave it 
for a later?)
# The null and "null" thing is a typo. Sorry for this stupid mistake. A null 
value will cause problems for SpecificRecordCompiler.
# I will make the parameter in compiler and maven plugin only affects decimal 
type, which will minimize the impact of this patch. Thanks for the suggestion
# The focus here is the {{convert}} method in GenericDatumWriter. You are 
correct about the invariant here. But as [~rdblue] explains below, this used to 
be not a problem because the caller within GenericDatumWriter won't pass in any 
null pointer. I tends not to add the checking here. One reason is it is good 
for performance. The second reason is that a null schema or logical type is 
definitely a bad error, and I don't think any children would try to capture or 
process it.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.5.patch

Upload the new patch that contains all the changes based on code review.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch, AVRO-1847.5.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329515#comment-15329515
 ] 

Yibing Shi commented on AVRO-1847:
--

Oops! Have corrected it.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329521#comment-15329521
 ] 

Yibing Shi commented on AVRO-1847:
--

As mentioned above, I don't think we need a separate flag for date/time types 
since their behaviour has never changed. I just make sure to use JODA/Java 
classes for these types no matter the value of this parameter. This parameter 
has also been renamed as {{enableLogicalTypesForNonDateTimeTypes}}

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329522#comment-15329522
 ] 

Yibing Shi commented on AVRO-1847:
--

Same as above

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329511#comment-15329511
 ] 

Yibing Shi commented on AVRO-1847:
--

I didn't think of childrens of this class may need to check its return value. I 
thought an exception made more sense here because there is an unknown logical 
type was found. Now I agree with you that return null makes their life easier. 
Have changed the code back according to your suggestion.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329501#comment-15329501
 ] 

Yibing Shi commented on AVRO-1847:
--

Since we always try to use Joda or Java classes for date/time types, I prefer 
not to add a new flag. Instead, I just check the whether the the logical type 
is one of the date/time types. If yes, I apply the conversions if they are 
present.
{code}
  public String javaType(Schema schema) {
if (useLogicalTypesForNonDateTimeTypes
|| LogicalTypes.isDateTimeType(schema.getLogicalType())) {
  Conversion conversion = SPECIFIC
  .getConversionFor(schema.getLogicalType());
  if (conversion != null) {
return conversion.getConvertedType().getName();
  }
}
{code}

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329495#comment-15329495
 ] 

Yibing Shi commented on AVRO-1847:
--

I have removed the passthrough behaviour in convert to/from methods. As a 
result, this part of code keeps intact.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329494#comment-15329494
 ] 

Yibing Shi commented on AVRO-1847:
--

The {{conver}} method is newly added by this patch. It didn't exist in master 
branch. However, the exception it throws does get out through method {{write}}. 
I agree with [~busbey] that we should keep the exceptions the same as before.
On the other hand, it also makes sense to make the to convert methods in 
{{Conversions}} behave similarly. So I prefer throwing AvroRuntimeException in 
{{convertToRawType}}, but catch this exception in 
{{GenericDatumWriter.convert}} and throw its cause out to keep backwards 
compatibility. Maybe we can change this feature in later versions?
The code is like below:
{code}
  protected  Object convert(Schema schema, LogicalType logicalType,
   Conversion conversion, Object datum) {
try {
  if (conversion == null) {
return datum;
  } else {
return Conversions.convertToRawType(datum, schema, logicalType, 
conversion);
  }
} catch (AvroRuntimeException e) {
  Throwable cause = e.getCause();
  if (cause != null && cause.getClass() == ClassCastException.class) {
// This is to keep backwards compatibility. The convert function here 
used to
// throw CCE. When prompted to Conversions, it throws 
AvroRuntimeException. So,
// if the cause is a CCE, rethrow it in case any child class checks it. 
This
// behaviour can be changed later in future versions to make it 
consistent with
// reading path, which throws AvroRuntimeException
throw (ClassCastException)cause;
  } else {
throw e;
  }
}
  }
{code}
Please let me know your opinion.


> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-14 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328665#comment-15328665
 ] 

Yibing Shi edited comment on AVRO-1847 at 6/14/16 1:18 PM:
---

Agree. Since these 2 methods are now public, I add checking for all the 
parameters. If any of them is null, an AvroRuntimeException will be thrown out. 
IMO, adding the pass through function to these functions makes them coupled 
with callers. Please let me know if you don't think so.


was (Author: yibing):
Agree. Will do that.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-13 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328665#comment-15328665
 ] 

Yibing Shi commented on AVRO-1847:
--

Agree. Will do that.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-08 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.4.patch

Add another patch, which add the backward compatibility of logical type 
support. That is, by default the compiler still defines class member types 
based on "type" in schema (such as ByteBuffer for decimal), but also add a 
parameter which enables defining members based on "logicalType" (such as 
BigDecimal for decimal).

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch, 
> AVRO-1847.4.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-06 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.3.patch

Attach another patch, which fixes the problem where classes generated by IDL 
compiler cannot be constructed via builder properly.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch, AVRO-1847.3.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-06-01 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311336#comment-15311336
 ] 

Yibing Shi commented on AVRO-1847:
--

Understood! Thanks for the response!

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-05-31 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309321#comment-15309321
 ] 

Yibing Shi commented on AVRO-1847:
--

[~rdblue], could you please help have a look at the new patch?

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-05-21 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.2.patch

[~rdblue], I have attached another patch that adds the missing parts mentioned 
in your previous comment. Could you please help have another look?

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-05-18 Thread Yibing Shi (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290055#comment-15290055
 ] 

Yibing Shi commented on AVRO-1847:
--

Hi [~rdblue], thanks for the guide! I will modify the patch and submit it again.

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-05-18 Thread Yibing Shi (JIRA)
Yibing Shi created AVRO-1847:


 Summary: IDL compiler uses ByteBuffer for decimal type even if 
logical type is supported 
 Key: AVRO-1847
 URL: https://issues.apache.org/jira/browse/AVRO-1847
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.8.0
Reporter: Yibing Shi


Version 1.8.0 has added the support of logical types. A conversion class 
(Conversions.DecimalConversion) has also been added for decimal type. However, 
the IDL compiler still uses ByteBuffer for decimal types, which is not the same 
behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-05-18 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Attachment: AVRO-1847.1.patch

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported

2016-05-18 Thread Yibing Shi (JIRA)

 [ 
https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yibing Shi updated AVRO-1847:
-
Status: Patch Available  (was: Open)

> IDL compiler uses ByteBuffer for decimal type even if logical type is 
> supported 
> 
>
> Key: AVRO-1847
> URL: https://issues.apache.org/jira/browse/AVRO-1847
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Yibing Shi
> Attachments: AVRO-1847.1.patch
>
>
> Version 1.8.0 has added the support of logical types. A conversion class 
> (Conversions.DecimalConversion) has also been added for decimal type. 
> However, the IDL compiler still uses ByteBuffer for decimal types, which is 
> not the same behaviour as data, time or timestamp type (added in AVRO-1684). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)