Re: Feature freeze Apache Avro 1.9

[Ivan Greene]

Somehow only now after I sent this last message did I realize the
conversion can already be controlled by adding the LogicalType to
ReflectData as in TestReflectLogicalTypes.java :) So never mind on that
second part.

However it's worth making a consideration about whether we are interested
in revising the UUID logicalType to be 16-byte fixed rather than a 36-byte
string before it is documented in this release.

-- Ivan

On Mon, Apr 1, 2019 at 8:30 PM Ivan Greene 
wrote:

> Another outstanding bit for the release in my mind is the state of the
> UUID logicalType. As Ryan commented here:
> https://issues.apache.org/jira/browse/AVRO-2021?focusedCommentId=16242524=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16242524
> there is currently an implemented logical type of 'uuid', however it uses
> the naive toString/fromString rather than a more compact 16-byte fixed. I
> think we should think about whether we really need this logical type as it
> doesn't provide much benefit as it is. What do you think?
>
> Related is UUID support for ReflectData, which is currently nil (we cannot
> even annotate @Stringable; UUID has no String constructor). Basically the
> only option for using UUIDs in ReflectData is the user implementing their
> own CustomEncoding. A long unanswered query on another Jira is here:
> https://issues.apache.org/jira/browse/AVRO-1497?focusedCommentId=16283656=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16283656
> We could either:
> - Provide CustomEncoding implementation(s), such as UUIDAsStringEncoding
> and/or UUIDAsFixedEncoding
> - Add support for LogicalTypes in ReflectData definitions, through a new
> annotation a la @AvroLogicalType("timestamp-millis"), etc. where we can use
> the uuid logicalType if we decide to keep that in
>
> Taking a look at the history it seems as though LogicalTypes and
> CustomEncodings share some of the same intentions. It's not clear what the
> better option is in this case. CustomEncoding predates LogicalType by a few
> years, but CustomEncoding is more Java-centric.
>
> Please let me know what the thoughts are. I should have some time to work
> on what we decide on this week. Would like to get these items in for 1.9.
>
> -- Ivan
>

Re: Feature freeze Apache Avro 1.9

[Ivan Greene]

Another outstanding bit for the release in my mind is the state of the UUID
logicalType. As Ryan commented here:
https://issues.apache.org/jira/browse/AVRO-2021?focusedCommentId=16242524=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16242524
there is currently an implemented logical type of 'uuid', however it uses
the naive toString/fromString rather than a more compact 16-byte fixed. I
think we should think about whether we really need this logical type as it
doesn't provide much benefit as it is. What do you think?

Related is UUID support for ReflectData, which is currently nil (we cannot
even annotate @Stringable; UUID has no String constructor). Basically the
only option for using UUIDs in ReflectData is the user implementing their
own CustomEncoding. A long unanswered query on another Jira is here:
https://issues.apache.org/jira/browse/AVRO-1497?focusedCommentId=16283656=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16283656
We could either:
- Provide CustomEncoding implementation(s), such as UUIDAsStringEncoding
and/or UUIDAsFixedEncoding
- Add support for LogicalTypes in ReflectData definitions, through a new
annotation a la @AvroLogicalType("timestamp-millis"), etc. where we can use
the uuid logicalType if we decide to keep that in

Taking a look at the history it seems as though LogicalTypes and
CustomEncodings share some of the same intentions. It's not clear what the
better option is in this case. CustomEncoding predates LogicalType by a few
years, but CustomEncoding is more Java-centric.

Please let me know what the thoughts are. I should have some time to work
on what we decide on this week. Would like to get these items in for 1.9.

-- Ivan

Jenkins build is back to normal : AvroJava #635

[Apache Jenkins Server]

See 

[jira] [Updated] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raman Gupta updated AVRO-2366:
--
Description: 
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure at compile-time if the default does not match the 
schema. For example, a field definition of:

{{
{"name": "name", "type": "string", "default": null}
}}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{
Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a {"type":"string","avro.java.string":"String"}
at org.apache.avro.Schema.validateDefault(Schema.java:1482)
at org.apache.avro.Schema.access$300(Schema.java:84)
at org.apache.avro.Schema$Field.(Schema.java:493)
at org.apache.avro.Schema$Field.(Schema.java:485)
at org.apache.avro.Schema$Field.(Schema.java:504)
at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)
at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)
at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)
at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)
}}

  was:
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure at compile-time if the default does not match the 
schema. For example, a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a \{"type":"string","avro.java.string":"String"}}}
{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}


> 1.9 
> 
>
> Key: AVRO-2366
> URL: https://issues.apache.org/jira/browse/AVRO-2366
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Raman Gupta
>Priority: Major
>
> Followup to AVRO-2035.
> I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to 
> false, and still get a failure at compile-time if the default does not match 
> the schema. For example, a field definition of:
> {{
> {"name": "name", "type": "string", "default": null}
> }}
> Should this definition succeed if `setValidateDefaults(false)` is called? If 
> not, should the `setValidateDefaults` method just be removed?
> The exception is:
> {{
> Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
> null not a {"type":"string","avro.java.string":"String"}
>   at org.apache.avro.Schema.validateDefault(Schema.java:1482)
>   at org.apache.avro.Schema.access$300(Schema.java:84)
>   at org.apache.avro.Schema$Field.(Schema.java:493)
>   at org.apache.avro.Schema$Field.(Schema.java:485)
>   at org.apache.avro.Schema$Field.(Schema.java:504)
>   at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)
>   at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)
>   at 
> org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)
>   at 
> org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)
> }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raman Gupta updated AVRO-2366:
--
Description: 
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure at compile-time if the default does not match the 
schema. For example, a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a \{"type":"string","avro.java.string":"String"}}}
{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

  was:
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure at compile-time if the default does not match the 
schema. For example, a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a \{"type":"string","avro.java.string":"String"}}}
{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 


> 1.9 
> 
>
> Key: AVRO-2366
> URL: https://issues.apache.org/jira/browse/AVRO-2366
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Raman Gupta
>Priority: Major
>
> Followup to AVRO-2035.
> I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to 
> false, and still get a failure at compile-time if the default does not match 
> the schema. For example, a field definition of:
> {"name": "name", "type": "string", "default": null}
> Should this definition succeed if `setValidateDefaults(false)` is called? If 
> not, should the `setValidateDefaults` method just be removed?
> The exception is:
> {{Caused by: org.apache.avro.AvroTypeException: Invalid default for field 
> name: null not a \{"type":"string","avro.java.string":"String"}}}
> {{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
> {{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raman Gupta updated AVRO-2366:
--
Description: 
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure at compile-time if the default does not match the 
schema. For example, a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a \{"type":"string","avro.java.string":"String"}}}
{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 

  was:
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure at compile-time if the default does not match the 
schema. For example, a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{\{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
 \{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 


> 1.9 
> 
>
> Key: AVRO-2366
> URL: https://issues.apache.org/jira/browse/AVRO-2366
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Raman Gupta
>Priority: Major
>
> Followup to AVRO-2035.
> I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to 
> false, and still get a failure at compile-time if the default does not match 
> the schema. For example, a field definition of:
> {"name": "name", "type": "string", "default": null}
> Should this definition succeed if `setValidateDefaults(false)` is called? If 
> not, should the `setValidateDefaults` method just be removed?
> The exception is:
> {{Caused by: org.apache.avro.AvroTypeException: Invalid default for field 
> name: null not a \{"type":"string","avro.java.string":"String"}}}
> {{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
> {{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raman Gupta updated AVRO-2366:
--
Description: 
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure at compile-time if the default does not match the 
schema. For example, a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{\{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
 \{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 

  was:
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure if the default does not match the schema. For example, 
a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a {"type":"string","avro.java.string":"String"}}}
 \{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
 \{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 


> 1.9 
> 
>
> Key: AVRO-2366
> URL: https://issues.apache.org/jira/browse/AVRO-2366
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Raman Gupta
>Priority: Major
>
> Followup to AVRO-2035.
> I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to 
> false, and still get a failure at compile-time if the default does not match 
> the schema. For example, a field definition of:
> {"name": "name", "type": "string", "default": null}
> Should this definition succeed if `setValidateDefaults(false)` is called? If 
> not, should the `setValidateDefaults` method just be removed?
> The exception is:
> {\{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
>  \{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
>  \{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
>  \{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
>  \{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raman Gupta updated AVRO-2366:
--
Description: 
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure if the default does not match the schema. For example, 
a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a {"type":"string","avro.java.string":"String"}}}
 \{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
 \{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
 \{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
 \{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 

  was:
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure if the default does not match the schema. For example, 
a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a \{"type":"string","avro.java.string":"String"}}}
{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 


> 1.9 
> 
>
> Key: AVRO-2366
> URL: https://issues.apache.org/jira/browse/AVRO-2366
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Raman Gupta
>Priority: Major
>
> Followup to AVRO-2035.
> I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to 
> false, and still get a failure if the default does not match the schema. For 
> example, a field definition of:
> {"name": "name", "type": "string", "default": null}
> Should this definition succeed if `setValidateDefaults(false)` is called? If 
> not, should the `setValidateDefaults` method just be removed?
> The exception is:
> {{Caused by: org.apache.avro.AvroTypeException: Invalid default for field 
> name: null not a {"type":"string","avro.java.string":"String"}}}
>  \{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
>  \{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
>  \{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
>  \{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
>  \{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
>  \{{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raman Gupta updated AVRO-2366:
--
Description: 
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure if the default does not match the schema. For example, 
a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?

The exception is:

{{Caused by: org.apache.avro.AvroTypeException: Invalid default for field name: 
null not a \{"type":"string","avro.java.string":"String"}}}
{{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
{{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
{{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
{{ at 
org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}

 

  was:
Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure if the default does not match the schema. For example, 
a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?


> 1.9 
> 
>
> Key: AVRO-2366
> URL: https://issues.apache.org/jira/browse/AVRO-2366
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Raman Gupta
>Priority: Major
>
> Followup to AVRO-2035.
> I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to 
> false, and still get a failure if the default does not match the schema. For 
> example, a field definition of:
> {"name": "name", "type": "string", "default": null}
> Should this definition succeed if `setValidateDefaults(false)` is called? If 
> not, should the `setValidateDefaults` method just be removed?
> The exception is:
> {{Caused by: org.apache.avro.AvroTypeException: Invalid default for field 
> name: null not a \{"type":"string","avro.java.string":"String"}}}
> {{ at org.apache.avro.Schema.validateDefault(Schema.java:1482)}}
> {{ at org.apache.avro.Schema.access$300(Schema.java:84)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:493)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:485)}}
> {{ at org.apache.avro.Schema$Field.(Schema.java:504)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:689)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.addStringType(SpecificCompiler.java:668)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compile(SpecificCompiler.java:598)}}
> {{ at 
> org.apache.avro.compiler.specific.SpecificCompiler.compileToDestination(SpecificCompiler.java:527)}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-813) EOFException is thrown during normal operation

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807222#comment-16807222
 ] 

Hudson commented on AVRO-813:
-

FAILURE: Integrated in Jenkins build AvroJava #634 (See 
[https://builds.apache.org/job/AvroJava/634/])
[AVRO-813] Keep the InputStream contract of -1 for eof Patch from Bruno (dkulp: 
[https://github.com/apache/avro/commit/a5e0f4ad01703eb4a724e71dfcf99306819c6bf3])
* (edit) 
lang/java/avro/src/main/java/org/apache/avro/util/ByteBufferInputStream.java


> EOFException is thrown during normal operation
> --
>
> Key: AVRO-813
> URL: https://issues.apache.org/jira/browse/AVRO-813
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.5.0
>Reporter: Bruno Dumon
>Assignee: Bruno Dumon
>Priority: Major
>  Labels: memex
> Fix For: 1.9.0
>
> Attachments: avro-813-patch.txt
>
>
> In an application that uses Avro as RPC mechanism (with the NettyTransceiver, 
> but that's irrelevant), I've noticed in jprofiler that during normal 
> operation quite some time was spent creating EOFExceptions:
> {noformat}
>   5.4% - 2,004 ms org.apache.avro.ipc.generic.GenericResponder.readRequest
>   5.0% - 1,871 ms org.apache.avro.generic.GenericDatumReader.read
>   4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.read
>   4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.readRecord
>   4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.read
>   4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.readRecord
>   4.3% - 1,596 ms org.apache.avro.generic.GenericDatumReader.read
>   2.8% - 1,048 ms org.apache.avro.generic.GenericDatumReader.readArray
>   1.3% - 477 ms org.apache.avro.io.ValidatingDecoder.arrayNext
>   1.3% - 471 ms org.apache.avro.io.BinaryDecoder.arrayNext
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.doReadItemCount
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.readLong
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.ensureBounds
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill
>   1.3% - 466 ms 
> org.apache.avro.io.BinaryDecoder$InputStreamByteSource.tryReadRaw
>   1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.read
>   1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.getBuffer
>   1.3% - 466 ms java.io.EOFException.
>   1.3% - 466 ms java.io.IOException.
>   1.2% - 460 ms java.lang.Exception.
>   1.2% - 460 ms java.lang.Throwable.
>   1.2% - 460 ms java.lang.Throwable.fillInStackTrace
> {noformat}
> These exceptions are produced by the ByteBufferInputStream (which modifies 
> InputStream's contract: return -1 at eof), but are catched higher up by the 
> tryReadRaw method.
> What happens is this:
> The message in question has an (empty) array at the end of its message, thus 
> the reader tries to read the size of this array in BinaryDecoder.readLong. 
> This calls ensureBounds(10), whose contract is that it should read 10 bytes 
> if they are available, and otherwise be quiet. ensureBounds calls via 
> compactAndFill the tryReadRaw method. It is this method which catches the 
> EOFException, because it only 'tries' to read so many bytes.
> Note that InputStreamByteSource.readRaw (without the 'try' part) does itself 
> check if read < 0 in order to throw EOFException, making the throwing of 
> EOFException in ByteBufferInputStream unnecessary (for this particular usage).
> There was some talk about EOFException in AVRO-392 too, though it seems this 
> particular common case was not mentioned there. When using Avro RPC, or more 
> in general, when using Avro to read small messages rather than large files, 
> it seems like one can very easily run into this EOFException situation, which 
> hurts performance.
> I'll attach a patch which simply removes the throwing of EOFException in 
> ByteBufferInputStream, but this will likely break other cases which rely on 
> the EOFException being thrown (haven't researched this to the bottom).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: AvroJava #634

[Apache Jenkins Server]

See 

Changes:

[dkulp] [AVRO-813] Keep the InputStream contract of -1 for eof Patch from Bruno

--
[...truncated 267.21 KB...]
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[ERROR] 
testCompileSchemasWithVariousFieldTypes(org.apache.avro.tool.TestSpecificCompilerTool)
  Time elapsed: 0.24 s  <<< FAILURE!
org.junit.ComparisonFailure: Found file: 
target/compiler/output-string/avro/examples/baseball/FieldTest.java does not 
match expected file: 
src/test/compiler/output-string/avro/examples/baseball/FieldTest.java 
expected:<...rg.apache.avro.data.[TimeConversions.TimestampMillisConversion()); 
   MODEL$.addLogicalTypeConversion(new 
org.apache.avro.data.TimeConversions.TimeMillisConversion());  }  private 
static final BinaryMessageEncoder ENCODER =  new 
BinaryMessageEncoder(MODEL$, SCHEMA$);  private static final 
BinaryMessageDecoder DECODER =  new 
BinaryMessageDecoder(MODEL$, SCHEMA$);  /**   * Return the 
BinaryMessageEncoder instance used by this class.   * @return the message 
encoder used by this class   */  public static BinaryMessageEncoder 
getEncoder() {return ENCODER;  }  /**   * Return the BinaryMessageDecoder 
instance used by this class.   * @return the message decoder used by this class 
  */  public static BinaryMessageDecoder getDecoder() {return 
DECODER;  }  /**   * Create a new BinaryMessageDecoder instance for this class 
that uses the specified {@link SchemaStore}.   * @param resolver a {@link 
SchemaStore} used to find schemas by fingerprint   * @return a 
BinaryMessageDecoder instance for this class backed by the given SchemaStore   
*/  public static BinaryMessageDecoder createDecoder(SchemaStore 
resolver) {return new BinaryMessageDecoder(MODEL$, SCHEMA$, 
resolver);  }  /**   * Serializes this FieldTest to a ByteBuffer.   * @return a 
buffer holding the serialized data for this instance   * @throws 
java.io.IOException if this instance could not be serialized   */  public 
java.nio.ByteBuffer toByteBuffer() throws java.io.IOException {return 
ENCODER.encode(this);  }  /**   * Deserializes a FieldTest from a ByteBuffer.   
* @param b a byte buffer holding serialized data for an instance of this class  
 * @return a FieldTest instance decoded from the given buffer   * @throws 
java.io.IOException if the given bytes could not be deserialized into an 
instance of this class   */  public static FieldTest fromByteBuffer(  
java.nio.ByteBuffer b) throws java.io.IOException {return 
DECODER.decode(b);  }  /** The number of the player */   private int number;   
private java.lang.String last_name;   private java.time.Instant timestamp;   
private java.time.Instant timestampMicros;   private java.time.LocalTime 
timeMillis;   private java.time.LocalTime timeMicros;  /**   * Default 
constructor.  Note that this does not initialize fields   * to their default 
values from the schema.  If that is desired then   * one should use 
newBuilder().   */  public FieldTest() {}  /**   * All-args 
constructor.   * @param number The number of the player   * @param last_name 
The new value for last_name   * @param timestamp The new value for timestamp   
* @param timestampMicros The new value for timestampMicros   * @param 
timeMillis The new value for timeMillis   * @param timeMicros The new value for 
timeMicros   */  public FieldTest(java.lang.Integer number, java.lang.String 
last_name, java.time.Instant timestamp, java.time.Instant timestampMicros, 
java.time.LocalTime timeMillis, java.time.LocalTime timeMicros) {
this.number = number;this.last_name = last_name;this.timestamp = 
timestamp.truncatedTo(java.time.temporal.ChronoUnit.MILLIS);
this.timestampMicros = 
timestampMicros.truncatedTo(java.time.temporal.ChronoUnit.MICROS);
this.timeMillis = timeMillis.truncatedTo(java.time.temporal.ChronoUnit.MILLIS); 
   this.timeMicros = 
timeMicros.truncatedTo(java.time.temporal.ChronoUnit.MICROS);  }  public 
org.apache.avro.specific.SpecificData getSpecificData() { return MODEL$; }  
public org.apache.avro.Schema getSchema() { return SCHEMA$; }  // Used by 
DatumWriter.  Applications should not call.  public java.lang.Object get(int 
field$) {switch (field$) {case 0: return number;case 1: return 
last_name;case 2: return timestamp;case 3: return timestampMicros;
case 4: return timeMillis;case 5: return timeMicros;default: throw new 
org.apache.avro.AvroRuntimeException("Bad index");}  }  private static 
final org.apache.avro.Conversion[] conversions =  new 
org.apache.avro.Conversion[] {  null,  null,  new 
org.apache.avro.data.TimeConversions.TimestampMillisConversion(),  new 
org.apache.avro.data.TimeConversions.TimestampMicrosConversion(),  new 
org.apache.avro.data.TimeConversions.TimeMillisConversion(),  new 

[jira] [Updated] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raman Gupta updated AVRO-2366:
--
Affects Version/s: 1.9.0

> 1.9 
> 
>
> Key: AVRO-2366
> URL: https://issues.apache.org/jira/browse/AVRO-2366
> Project: Apache Avro
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Raman Gupta
>Priority: Major
>
> Followup to AVRO-2035.
> I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to 
> false, and still get a failure if the default does not match the schema. For 
> example, a field definition of:
> {"name": "name", "type": "string", "default": null}
> Should this definition succeed if `setValidateDefaults(false)` is called? If 
> not, should the `setValidateDefaults` method just be removed?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2366) 1.9

[Raman Gupta (JIRA)]

Raman Gupta created AVRO-2366:
-

 Summary: 1.9 
 Key: AVRO-2366
 URL: https://issues.apache.org/jira/browse/AVRO-2366
 Project: Apache Avro
  Issue Type: Bug
Reporter: Raman Gupta


Followup to AVRO-2035.

I am testing 1.9 SNAPSHOT. I set `Schema.Parser.setValidateDefaults` to false, 
and still get a failure if the default does not match the schema. For example, 
a field definition of:

{"name": "name", "type": "string", "default": null}

Should this definition succeed if `setValidateDefaults(false)` is called? If 
not, should the `setValidateDefaults` method just be removed?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-1658) Add avroDoc on reflect

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-1658.
---
Resolution: Fixed

> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Raymie Stata
>Priority: Major
>  Labels: reflection
> Fix For: 1.9.0
>
> Attachments: 
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch, 
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch, 
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some 
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in 
> org.apache.avro.reflect



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-813) EOFException is thrown during normal operation

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-813.
--
Resolution: Fixed

> EOFException is thrown during normal operation
> --
>
> Key: AVRO-813
> URL: https://issues.apache.org/jira/browse/AVRO-813
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.5.0
>Reporter: Bruno Dumon
>Assignee: Bruno Dumon
>Priority: Major
>  Labels: memex
> Fix For: 1.9.0
>
> Attachments: avro-813-patch.txt
>
>
> In an application that uses Avro as RPC mechanism (with the NettyTransceiver, 
> but that's irrelevant), I've noticed in jprofiler that during normal 
> operation quite some time was spent creating EOFExceptions:
> {noformat}
>   5.4% - 2,004 ms org.apache.avro.ipc.generic.GenericResponder.readRequest
>   5.0% - 1,871 ms org.apache.avro.generic.GenericDatumReader.read
>   4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.read
>   4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.readRecord
>   4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.read
>   4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.readRecord
>   4.3% - 1,596 ms org.apache.avro.generic.GenericDatumReader.read
>   2.8% - 1,048 ms org.apache.avro.generic.GenericDatumReader.readArray
>   1.3% - 477 ms org.apache.avro.io.ValidatingDecoder.arrayNext
>   1.3% - 471 ms org.apache.avro.io.BinaryDecoder.arrayNext
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.doReadItemCount
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.readLong
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.ensureBounds
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill
>   1.3% - 466 ms 
> org.apache.avro.io.BinaryDecoder$InputStreamByteSource.tryReadRaw
>   1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.read
>   1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.getBuffer
>   1.3% - 466 ms java.io.EOFException.
>   1.3% - 466 ms java.io.IOException.
>   1.2% - 460 ms java.lang.Exception.
>   1.2% - 460 ms java.lang.Throwable.
>   1.2% - 460 ms java.lang.Throwable.fillInStackTrace
> {noformat}
> These exceptions are produced by the ByteBufferInputStream (which modifies 
> InputStream's contract: return -1 at eof), but are catched higher up by the 
> tryReadRaw method.
> What happens is this:
> The message in question has an (empty) array at the end of its message, thus 
> the reader tries to read the size of this array in BinaryDecoder.readLong. 
> This calls ensureBounds(10), whose contract is that it should read 10 bytes 
> if they are available, and otherwise be quiet. ensureBounds calls via 
> compactAndFill the tryReadRaw method. It is this method which catches the 
> EOFException, because it only 'tries' to read so many bytes.
> Note that InputStreamByteSource.readRaw (without the 'try' part) does itself 
> check if read < 0 in order to throw EOFException, making the throwing of 
> EOFException in ByteBufferInputStream unnecessary (for this particular usage).
> There was some talk about EOFException in AVRO-392 too, though it seems this 
> particular common case was not mentioned there. When using Avro RPC, or more 
> in general, when using Avro to read small messages rather than large files, 
> it seems like one can very easily run into this EOFException situation, which 
> hurts performance.
> I'll attach a patch which simply removes the throwing of EOFException in 
> ByteBufferInputStream, but this will likely break other cases which rely on 
> the EOFException being thrown (haven't researched this to the bottom).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-813) EOFException is thrown during normal operation

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807183#comment-16807183
 ] 

ASF subversion and git services commented on AVRO-813:
--

Commit a5e0f4ad01703eb4a724e71dfcf99306819c6bf3 in avro's branch 
refs/heads/master from Daniel Kulp
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=a5e0f4a ]

[AVRO-813] Keep the InputStream contract of -1 for eof
Patch from Bruno Dumon applied


> EOFException is thrown during normal operation
> --
>
> Key: AVRO-813
> URL: https://issues.apache.org/jira/browse/AVRO-813
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.5.0
>Reporter: Bruno Dumon
>Assignee: Bruno Dumon
>Priority: Major
>  Labels: memex
> Fix For: 1.9.0
>
> Attachments: avro-813-patch.txt
>
>
> In an application that uses Avro as RPC mechanism (with the NettyTransceiver, 
> but that's irrelevant), I've noticed in jprofiler that during normal 
> operation quite some time was spent creating EOFExceptions:
> {noformat}
>   5.4% - 2,004 ms org.apache.avro.ipc.generic.GenericResponder.readRequest
>   5.0% - 1,871 ms org.apache.avro.generic.GenericDatumReader.read
>   4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.read
>   4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.readRecord
>   4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.read
>   4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.readRecord
>   4.3% - 1,596 ms org.apache.avro.generic.GenericDatumReader.read
>   2.8% - 1,048 ms org.apache.avro.generic.GenericDatumReader.readArray
>   1.3% - 477 ms org.apache.avro.io.ValidatingDecoder.arrayNext
>   1.3% - 471 ms org.apache.avro.io.BinaryDecoder.arrayNext
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.doReadItemCount
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.readLong
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder.ensureBounds
>   1.3% - 466 ms org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill
>   1.3% - 466 ms 
> org.apache.avro.io.BinaryDecoder$InputStreamByteSource.tryReadRaw
>   1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.read
>   1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.getBuffer
>   1.3% - 466 ms java.io.EOFException.
>   1.3% - 466 ms java.io.IOException.
>   1.2% - 460 ms java.lang.Exception.
>   1.2% - 460 ms java.lang.Throwable.
>   1.2% - 460 ms java.lang.Throwable.fillInStackTrace
> {noformat}
> These exceptions are produced by the ByteBufferInputStream (which modifies 
> InputStream's contract: return -1 at eof), but are catched higher up by the 
> tryReadRaw method.
> What happens is this:
> The message in question has an (empty) array at the end of its message, thus 
> the reader tries to read the size of this array in BinaryDecoder.readLong. 
> This calls ensureBounds(10), whose contract is that it should read 10 bytes 
> if they are available, and otherwise be quiet. ensureBounds calls via 
> compactAndFill the tryReadRaw method. It is this method which catches the 
> EOFException, because it only 'tries' to read so many bytes.
> Note that InputStreamByteSource.readRaw (without the 'try' part) does itself 
> check if read < 0 in order to throw EOFException, making the throwing of 
> EOFException in ByteBufferInputStream unnecessary (for this particular usage).
> There was some talk about EOFException in AVRO-392 too, though it seems this 
> particular common case was not mentioned there. When using Avro RPC, or more 
> in general, when using Avro to read small messages rather than large files, 
> it seems like one can very easily run into this EOFException situation, which 
> hurts performance.
> I'll attach a patch which simply removes the throwing of EOFException in 
> ByteBufferInputStream, but this will likely break other cases which rely on 
> the EOFException being thrown (haven't researched this to the bottom).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807179#comment-16807179
 ] 

Hudson commented on AVRO-2364:
--

FAILURE: Integrated in Jenkins build AvroJava #633 (See 
[https://builds.apache.org/job/AvroJava/633/])
AVRO-2364 Use Jackson TokenBuffer in JsonDecoder (dan: 
[https://github.com/apache/avro/commit/dbf977989105217b37a798d30793d90e747eceb1])
* (edit) lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java


> Use Jackson TokenBuffer in JsonDecoder
> --
>
> Key: AVRO-2364
> URL: https://issues.apache.org/jira/browse/AVRO-2364
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Anthony Miyaguchi
>Assignee: Daniel Kulp
>Priority: Trivial
> Fix For: 1.9.0
>
>
> The JsonDecoder implements a class for replaying parts of the JSON stream 
> when accessing fields out of order.
> This module can take advantage of 
> [TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html]
>  to reduce code size. This buffer can be read back as a parser.
>  
> [https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2365) Enhancements to induce maven plugin goal

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807178#comment-16807178
 ] 

Hudson commented on AVRO-2365:
--

FAILURE: Integrated in Jenkins build AvroJava #633 (See 
[https://builds.apache.org/job/AvroJava/633/])
AVRO-2365 Enhancements to maven-plugin induce goal (dan: 
[https://github.com/apache/avro/commit/451deec474935db54824069dcf5cfe7ae49a1b11])
* (edit) 
lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/InduceMojo.java
* (edit) lang/java/maven-plugin/src/test/resources/unit/protocol/induce-pom.xml
* (edit) lang/java/maven-plugin/src/test/resources/unit/schema/induce-pom.xml
* (edit) 
lang/java/maven-plugin/src/test/java/org/apache/avro/mojo/TestInduceMojo.java
AVRO-2365: Allow using custom ReflectData impl (dan: 
[https://github.com/apache/avro/commit/22c4685e1711235e7435a2acbdb4df48f39e3f92])
* (edit) 
lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/InduceMojo.java


> Enhancements to induce maven plugin goal
> 
>
> Key: AVRO-2365
> URL: https://issues.apache.org/jira/browse/AVRO-2365
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' 
> maven plugin goal (see AVRO-1749), there are a few issues with it which in my 
> mind are critical to enable real-world use of the goal:
>  * Allow excluding classes based on a pattern, and/or the ability to specify 
> multiple sourceDirectories
>  * Option to specify whether to use ReflectData.AllowNull or ReflectData
>  * Throw MojoExecutionException with an informative message when appropriate, 
> instead of just printing stacktraces. This is very important, we definitely 
> need a build to fail when we hit a problem.
>  * Use the current project defined encoding rather than hard coded UTF-8.
>  * Use try-with-resources to ensure the PrintWriter gets closed in the event 
> of exception
> Nice to haves:
>  * Specify custom ReflectData implementation (i.e user declared extensions) 
> to be used
>  * Specify separate output directories for schemata and protocols (these 
> could both default to the single output directory if not specified 
> individually)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: AvroJava #633

[Apache Jenkins Server]

See 

Changes:

[dan] AVRO-2365 Enhancements to maven-plugin induce goal

[dan] AVRO-2365: Allow using custom ReflectData impl

[dan] AVRO-2364 Use Jackson TokenBuffer in JsonDecoder

[dan] Wrap tokenBuffer in try-with-resource

[dan] AVRO-2354: Add CombineAvroKeyValueFileInputFormat in avro-mapred to

[dan] reorder test expectation

[dkulp] Update to latest jetty

[dkulp] Fix spotify errors after merge

[dkulp] New test may have different order on different systems

--
[...truncated 324.20 KB...]
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at 
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[ERROR] 
testCompileSchemaSingleFile(org.apache.avro.tool.TestSpecificCompilerTool)  
Time elapsed: 0.027 s  <<< FAILURE!
org.junit.ComparisonFailure: Found file: 
target/compiler/output/avro/examples/baseball/Position.java does not match 
expected file: src/test/compiler/output/Position.java expected:<...ublic enum 
Position [implements org.apache.avro.generic.GenericEnumSymbol {  P, 
C, B1, B2, B3, SS, LF, CF, RF, DH  ;  public static final 
org.apache.avro.Schema SCHEMA$ = new 
org.apache.avro.Schema.Parser().parse("{\"type\":\"enum\",\"name\":\"Position\",\"namespace\":\"avro.examples.baseball\",\"symbols\":[\"P\",\"C\",\"B1\",\"B2\",\"B3\",\"SS\",\"LF\",\"CF\",\"RF\",\"DH\"]}");
  public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }  
public org.apache.avro.Schema get]Schema() { return SC...> but was:<...ublic 
enum Position [{  P, C, B1, B2, B3, SS, LF, CF, RF, DH  ;  public static final 
org.apache.avro.Schema SCHEMA$ = new 
org.apache.avro.Schema.Parser().parse("{\"type\":\"enum\",\"name\":\"Position\",\"namespace\":\"avro.examples.baseball\",\"symbols\":[\"P\",\"C\",\"B1\",\"B2\",\"B3\",\"SS\",\"LF\",\"CF\",\"RF\",\"DH\"]}");
  public static org.apache.avro.Schema getClass]Schema() { return SC...>
at org.junit.Assert.assertEquals(Assert.java:115)
at 
org.apache.avro.tool.TestSpecificCompilerTool.assertFileMatch(TestSpecificCompilerTool.java:133)
at 
org.apache.avro.tool.TestSpecificCompilerTool.testCompileSchemaSingleFile(TestSpecificCompilerTool.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 

[jira] [Commented] (AVRO-2354) Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro keyvalue files into combineSplit

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807180#comment-16807180
 ] 

Hudson commented on AVRO-2354:
--

FAILURE: Integrated in Jenkins build AvroJava #633 (See 
[https://builds.apache.org/job/AvroJava/633/])
AVRO-2354: Add CombineAvroKeyValueFileInputFormat in avro-mapred to (dan: 
[https://github.com/apache/avro/commit/7cd66387b7a249eabcfbd57fb651613d9eb1b8c4])
* (add) 
lang/java/mapred/src/test/java/org/apache/avro/mapreduce/TestCombineAvroKeyValueFileInputFormat.java
* (add) 
lang/java/mapred/src/main/java/org/apache/avro/mapreduce/CombineAvroKeyValueFileInputFormat.java


> Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro 
> keyvalue files into combineSplit
> 
>
> Key: AVRO-2354
> URL: https://issues.apache.org/jira/browse/AVRO-2354
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Wang, Xinglong
>Assignee: Daniel Kulp
>Priority: Minor
> Fix For: 1.9.0
>
>
> In our production env, we generate avro files to track some user behavior 
> events. Every hour, we will have several avro files created. And daily, we 
> will run MR to do analysis, when using AvroKeyValueInputFormat, a lot of 
> small mappers started due to we have small avro files. 
> A combine file inputformat will be very helpful for such case. 
> Hadoop already provided some implementation for sequencefile and text file. 
> This Jira is propose a CombineAvroKeyValueFileInputFormat class to implement 
> the same for avro keyvalue files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-1547) AvroApp Schema Tool

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp updated AVRO-1547:
--
Fix Version/s: (was: 1.9.0)

> AvroApp Schema Tool
> ---
>
> Key: AVRO-1547
> URL: https://issues.apache.org/jira/browse/AVRO-1547
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: java
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>
> Over in Gora, I have been thinking for a while that the process of writing 
> JSON data beans is rather time consuming when beans are LARGE.
> I wanted to open this ticket for a while and now only get around to. I 
> proposed to have the following
> A simple HTML webpage that defines a form of sorts, the form will enable 
> users to create JSON schemas and will be driven by enabling users to enter 
> Object values based on the current Avro specification document e.g. ti will 
> be restrictive in scope.
> On top of this I propose to then use simple JQuery to send a request to the 
> JSONBlob API [0], obtain a JSON representation of the data and then pretty 
> print write this information to a file within the browser. The users can then 
> save this file focally and do with it what they wish.
> I think that this page can easily be hosted alongside the current static Avro 
> website and that there is no need to write a web application for this yet.
> I'll try to work on it sooner rather than later as this would also lower the 
> barrier for users of Gora (as I am sure it would Users of other technologies 
> requiring definition of Objects via JSOn schemas).
> I've not assigned this against any component as there is none which I feel 
> appropriate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-1922) Fixed dimension for array

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp updated AVRO-1922:
--
Fix Version/s: (was: 1.9.0)

> Fixed dimension for array
> -
>
> Key: AVRO-1922
> URL: https://issues.apache.org/jira/browse/AVRO-1922
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: spec
>Affects Versions: 1.8.1
>Reporter: Jim Pivarski
>Priority: Major
>
> This is a feature request for future versions of the Avro specification.
> We have found one kind of data structure that is hard to express in Avro: 
> tensors. Although we can (and do) build matrices as {"type": "array", 
> "items": {"type": "array", "items": "double"}}, this type does not specify 
> that the grid of numbers is rectangular. We believe that rectangular arrays 
> of numbers (or other nested types) would be a strong addition to Avro, both 
> as a type system and as a serialization format. With the total size of all 
> dimensions fixed in the schema, they would not need to be repeated in each 
> serialized datum.
> For instance, suppose there was an extension of type "array" to specify 
> dimensions:
> {"type": "array", "dimensions": [3, 3, 3, 3], "items": "double"}
> This 3-by-3-by-3-by-3 tensor (representing, for instance, the Riemann 
> curvature tensor in 3-space) specifies that 81 double-precision numbers 
> (3*3*3*3) are expected for each datum. With nested arrays, the size, "3," 
> would have to be separately encoded 40 times (1 + 3*(1 + 3*(1 + 3))) for each 
> datum, even though they would never change in a dataset of Riemann tensors. 
> With a "dimensions" attribute in the schema, only the content needs to be 
> serialized. Moreover, this extension can clearly be used with any other 
> "items" type, to make dense tables of strings, for instance.
> Avro has been extended in a similar way in the past. The "fixed" type is a 
> "bytes" without the need to specify the number of bytes for each datum. Our 
> proposal provides a similar packing for structured objects that can be 
> significant for large numbers of dimensions, as shown above. The advantage to 
> consumers of Avro data is that we can write functions that do not need to 
> check all array sizes at runtime (for operations like tensor contractions and 
> products).
> We have searched the web and the Avro JIRA site for similar proposals and 
> found none, so we're adding this proposal to JIRA in addition to this e-mail. 
> Please let us know if you have any comments, or if we can provide any more 
> information.
> Thank you!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2182) avro-maven-plugin: IDLProtocolMojo should regard property "project.build.sourceEncoding" for file generation

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2182.
---
Resolution: Fixed

> avro-maven-plugin: IDLProtocolMojo should regard property 
> "project.build.sourceEncoding" for file generation
> 
>
> Key: AVRO-2182
> URL: https://issues.apache.org/jira/browse/AVRO-2182
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.2
> Environment: Tested on windows with Java 8.
>Reporter: Hans-Peter Werner
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: JIRA-AVRO-2182.patch
>
>
> Currently it's immpossible to use e.g. german Umlauts in neither documention 
> nor fieldnames.
> Linked to [AVRO-1471|http://issues.apache.org/jira/browse/AVRO-1471]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-1559) Drop support for Ruby 1.8

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-1559.
---
Resolution: Fixed

1.9 will be Ruby 2.3.3+

> Drop support for Ruby 1.8
> -
>
> Key: AVRO-1559
> URL: https://issues.apache.org/jira/browse/AVRO-1559
> Project: Apache Avro
>  Issue Type: Wish
>  Components: ruby
>Affects Versions: 1.7.7
>Reporter: Willem van Bergen
>Assignee: Willem van Bergen
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: AVRO-1559.patch
>
>
> - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. 
> - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a 
> recent OSX, it won't compile without manual fiddling).
> - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. 
> Supporting both at the same time adds a lot of overhead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2020) Update BUILD instructions with correct versions for language-specific libraries

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2020.
---
   Resolution: Fixed
Fix Version/s: (was: 1.7.9)
   (was: 1.8.3)

> Update BUILD instructions with correct versions for language-specific 
> libraries
> ---
>
> Key: AVRO-2020
> URL: https://issues.apache.org/jira/browse/AVRO-2020
> Project: Apache Avro
>  Issue Type: Task
>  Components: community, doc
>Affects Versions: 1.7.7, 1.8.1, 1.9.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 1.9.0
>
>
> our BUILD instructions current list a few things I know are wrong (Ruby 1.8 
> support) and some things I'm pretty sure are wrong (Java 1.6 support).
> we should do a pass checking all the languages.
> {code}
> REQUIREMENTS
> The following packages must be installed before Avro can be built:
>  - Java: JDK 1.6, Maven 2 or better, protobuf-compile
>  - PHP: php5, phpunit, php5-gmp
>  - Python: 2.5 or greater, python-setuptools for dist target
>  - C: gcc, cmake, asciidoc, source-highlight
>  - C++: cmake 2.8.4 or greater, g++, flex, bison, libboost-dev
>  - C#: mono-devel mono-gmcs nunit
>  - JavaScript: nodejs, npm
>  - Ruby: ruby 1.86 or greater, ruby-dev, gem, rake, echoe, yajl-ruby
>  - Perl: perl 5.8.1 or greater, gmake, Module::Install,
>Module::Install::ReadmeFromPod, Module::Install::Repository,
>Math::BigInt, JSON::XS, Try::Tiny, Regexp::Common, Encode,
>IO::String, Object::Tiny, Compress::ZLib, Test::More,
>Test::Exception, Test::Pod
>  - Apache Ant 1.7
>  - Apache Forrest 0.8 (for documentation)
>  - md5sum, sha1sum, used by top-level dist target
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-1626) Missing lang/csharp/src/apache/perf/app.config

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-1626.
---
Resolution: Fixed

> Missing lang/csharp/src/apache/perf/app.config
> --
>
> Key: AVRO-1626
> URL: https://issues.apache.org/jira/browse/AVRO-1626
> Project: Apache Avro
>  Issue Type: Bug
>  Components: csharp
>Reporter: Niels Basjes
>Priority: Major
> Fix For: 1.9.0
>
>
> This error is output during the build 
> {code}
> Target _CopyAppConfigFile:
> /usr/lib/mono/4.5/Microsoft.Common.targets: error : Cannot copy 
> /home/nbasjes/avro/lang/csharp/src/apache/perf/app.config to 
> /home/nbasjes/avro/lang/csharp/build/perf/Release/Avro.perf.exe.config, as 
> the source file doesn't exist.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2251) Modify Perf.java to better support automation scripts

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2251.
---
Resolution: Fixed

> Modify Perf.java to better support automation scripts
> -
>
> Key: AVRO-2251
> URL: https://issues.apache.org/jira/browse/AVRO-2251
> Project: Apache Avro
>  Issue Type: Test
>  Components: java
>Reporter: Raymie Stata
>Assignee: Raymie Stata
>Priority: Major
> Fix For: 1.9.0
>
>
> To better support automated performance-test suites, this patch proposes two 
> new arguments to the 'Perf.java' command-line tool:
> The `-o' argument gives 'Perf.java' the name of a file that should get the 
> results of the run.  Currently, Perf.java sends output to System.out – but if 
> 'Perf.java' is invoked using Maven, which is the easiest way to invoke it, 
> then System.out is polluted with a bunch of other output.  Redirecting 
> 'Perf.java' output metrics to a file makes it easier for automation scripts 
> to process those metrics.
> The `-c [spec]` argument tells 'Perf.java' to generate a comma-separated 
> output.  By default, all benchmark metrics are output, but the optional 
> `spec` argument can be used to indicate exactly which metrics should be 
> included in the CSV output.  The default output of 'Perf.java' is optimized 
> for human inspection – for example, it includes the text "ms" in the 
> running-time column so humans will understand the units of the running-time 
> metric.  The `-c` option will tell 'Perf.java' to generate machine-optimized 
> output that is easier to consume by automation scripts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2357) (ReflectData) Support for generic types in protocol definitions

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2357.
---
Resolution: Fixed

> (ReflectData) Support for generic types in protocol definitions
> ---
>
> Key: AVRO-2357
> URL: https://issues.apache.org/jira/browse/AVRO-2357
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ivan Greene
>Priority: Minor
> Fix For: 1.9.0
>
>
> For a Java interface extending another interface with type parameters, we may 
> resolve actual type parameters and build the protocol definition based upon 
> those.
> For example, let's say we have a generic protocol defined by a Java interface:
> {code:java}
> public interface CrudProto {
>   void persist(T record);
>   T fetchById(I id);
> }{code}
> It would be natural to define a set of interfaces that extend this, such as:
> {code:java}
> public interface FooBarRecordProto extends CrudProto {}
> public interface OtherRecordProto extends CrudProto {}
> {code}
> Calling ReflectData.get().getProtocol(FooBarRecordProto.class) should be able 
> to resolve that this protocol deals in FooBarRecords and Strings, and build a 
> protocol accordingly.
> Currently, this call will produce an exception stating that a schema for 'T' 
> cannot be resolved.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2354) Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro keyvalue files into combineSplit

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2354.
---
   Resolution: Fixed
 Assignee: Daniel Kulp
Fix Version/s: 1.9.0

> Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro 
> keyvalue files into combineSplit
> 
>
> Key: AVRO-2354
> URL: https://issues.apache.org/jira/browse/AVRO-2354
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Wang, Xinglong
>Assignee: Daniel Kulp
>Priority: Minor
> Fix For: 1.9.0
>
>
> In our production env, we generate avro files to track some user behavior 
> events. Every hour, we will have several avro files created. And daily, we 
> will run MR to do analysis, when using AvroKeyValueInputFormat, a lot of 
> small mappers started due to we have small avro files. 
> A combine file inputformat will be very helpful for such case. 
> Hadoop already provided some implementation for sequencefile and text file. 
> This Jira is propose a CombineAvroKeyValueFileInputFormat class to implement 
> the same for avro keyvalue files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2364.
---
   Resolution: Fixed
 Assignee: Daniel Kulp
Fix Version/s: 1.9.0

> Use Jackson TokenBuffer in JsonDecoder
> --
>
> Key: AVRO-2364
> URL: https://issues.apache.org/jira/browse/AVRO-2364
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Anthony Miyaguchi
>Assignee: Daniel Kulp
>Priority: Trivial
> Fix For: 1.9.0
>
>
> The JsonDecoder implements a class for replaying parts of the JSON stream 
> when accessing fields out of order.
> This module can take advantage of 
> [TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html]
>  to reduce code size. This buffer can be read back as a parser.
>  
> [https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2354) Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro keyvalue files into combineSplit

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807134#comment-16807134
 ] 

ASF subversion and git services commented on AVRO-2354:
---

Commit 7cd66387b7a249eabcfbd57fb651613d9eb1b8c4 in avro's branch 
refs/heads/master from suxingfate
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=7cd6638 ]

AVRO-2354: Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine 
small avro keyvalue files into combineSplit


> Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro 
> keyvalue files into combineSplit
> 
>
> Key: AVRO-2354
> URL: https://issues.apache.org/jira/browse/AVRO-2354
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Wang, Xinglong
>Priority: Minor
>
> In our production env, we generate avro files to track some user behavior 
> events. Every hour, we will have several avro files created. And daily, we 
> will run MR to do analysis, when using AvroKeyValueInputFormat, a lot of 
> small mappers started due to we have small avro files. 
> A combine file inputformat will be very helpful for such case. 
> Hadoop already provided some implementation for sequencefile and text file. 
> This Jira is propose a CombineAvroKeyValueFileInputFormat class to implement 
> the same for avro keyvalue files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807130#comment-16807130
 ] 

ASF subversion and git services commented on AVRO-2364:
---

Commit dbf977989105217b37a798d30793d90e747eceb1 in avro's branch 
refs/heads/master from Anthony Miyaguchi
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=dbf9779 ]

AVRO-2364 Use Jackson TokenBuffer in JsonDecoder


> Use Jackson TokenBuffer in JsonDecoder
> --
>
> Key: AVRO-2364
> URL: https://issues.apache.org/jira/browse/AVRO-2364
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Anthony Miyaguchi
>Priority: Trivial
>
> The JsonDecoder implements a class for replaying parts of the JSON stream 
> when accessing fields out of order.
> This module can take advantage of 
> [TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html]
>  to reduce code size. This buffer can be read back as a parser.
>  
> [https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2365) Enhancements to induce maven plugin goal

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807129#comment-16807129
 ] 

ASF subversion and git services commented on AVRO-2365:
---

Commit 22c4685e1711235e7435a2acbdb4df48f39e3f92 in avro's branch 
refs/heads/master from Ivan Greene
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=22c4685 ]

AVRO-2365: Allow using custom ReflectData impl


> Enhancements to induce maven plugin goal
> 
>
> Key: AVRO-2365
> URL: https://issues.apache.org/jira/browse/AVRO-2365
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ivan Greene
>Priority: Major
> Fix For: 1.9.0, 1.8.3
>
>
> Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' 
> maven plugin goal (see AVRO-1749), there are a few issues with it which in my 
> mind are critical to enable real-world use of the goal:
>  * Allow excluding classes based on a pattern, and/or the ability to specify 
> multiple sourceDirectories
>  * Option to specify whether to use ReflectData.AllowNull or ReflectData
>  * Throw MojoExecutionException with an informative message when appropriate, 
> instead of just printing stacktraces. This is very important, we definitely 
> need a build to fail when we hit a problem.
>  * Use the current project defined encoding rather than hard coded UTF-8.
>  * Use try-with-resources to ensure the PrintWriter gets closed in the event 
> of exception
> Nice to haves:
>  * Specify custom ReflectData implementation (i.e user declared extensions) 
> to be used
>  * Specify separate output directories for schemata and protocols (these 
> could both default to the single output directory if not specified 
> individually)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2365) Enhancements to induce maven plugin goal

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807128#comment-16807128
 ] 

ASF subversion and git services commented on AVRO-2365:
---

Commit 451deec474935db54824069dcf5cfe7ae49a1b11 in avro's branch 
refs/heads/master from Ivan Greene
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=451deec ]

AVRO-2365 Enhancements to maven-plugin induce goal


> Enhancements to induce maven plugin goal
> 
>
> Key: AVRO-2365
> URL: https://issues.apache.org/jira/browse/AVRO-2365
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ivan Greene
>Priority: Major
> Fix For: 1.9.0, 1.8.3
>
>
> Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' 
> maven plugin goal (see AVRO-1749), there are a few issues with it which in my 
> mind are critical to enable real-world use of the goal:
>  * Allow excluding classes based on a pattern, and/or the ability to specify 
> multiple sourceDirectories
>  * Option to specify whether to use ReflectData.AllowNull or ReflectData
>  * Throw MojoExecutionException with an informative message when appropriate, 
> instead of just printing stacktraces. This is very important, we definitely 
> need a build to fail when we hit a problem.
>  * Use the current project defined encoding rather than hard coded UTF-8.
>  * Use try-with-resources to ensure the PrintWriter gets closed in the event 
> of exception
> Nice to haves:
>  * Specify custom ReflectData implementation (i.e user declared extensions) 
> to be used
>  * Specify separate output directories for schemata and protocols (these 
> could both default to the single output directory if not specified 
> individually)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2365) Enhancements to induce maven plugin goal

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp updated AVRO-2365:
--
Fix Version/s: (was: 1.8.3)

> Enhancements to induce maven plugin goal
> 
>
> Key: AVRO-2365
> URL: https://issues.apache.org/jira/browse/AVRO-2365
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ivan Greene
>Priority: Major
> Fix For: 1.9.0
>
>
> Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' 
> maven plugin goal (see AVRO-1749), there are a few issues with it which in my 
> mind are critical to enable real-world use of the goal:
>  * Allow excluding classes based on a pattern, and/or the ability to specify 
> multiple sourceDirectories
>  * Option to specify whether to use ReflectData.AllowNull or ReflectData
>  * Throw MojoExecutionException with an informative message when appropriate, 
> instead of just printing stacktraces. This is very important, we definitely 
> need a build to fail when we hit a problem.
>  * Use the current project defined encoding rather than hard coded UTF-8.
>  * Use try-with-resources to ensure the PrintWriter gets closed in the event 
> of exception
> Nice to haves:
>  * Specify custom ReflectData implementation (i.e user declared extensions) 
> to be used
>  * Specify separate output directories for schemata and protocols (these 
> could both default to the single output directory if not specified 
> individually)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2365) Enhancements to induce maven plugin goal

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved AVRO-2365.
---
Resolution: Fixed
  Assignee: Daniel Kulp

> Enhancements to induce maven plugin goal
> 
>
> Key: AVRO-2365
> URL: https://issues.apache.org/jira/browse/AVRO-2365
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' 
> maven plugin goal (see AVRO-1749), there are a few issues with it which in my 
> mind are critical to enable real-world use of the goal:
>  * Allow excluding classes based on a pattern, and/or the ability to specify 
> multiple sourceDirectories
>  * Option to specify whether to use ReflectData.AllowNull or ReflectData
>  * Throw MojoExecutionException with an informative message when appropriate, 
> instead of just printing stacktraces. This is very important, we definitely 
> need a build to fail when we hit a problem.
>  * Use the current project defined encoding rather than hard coded UTF-8.
>  * Use try-with-resources to ensure the PrintWriter gets closed in the event 
> of exception
> Nice to haves:
>  * Specify custom ReflectData implementation (i.e user declared extensions) 
> to be used
>  * Specify separate output directories for schemata and protocols (these 
> could both default to the single output directory if not specified 
> individually)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2250) Release 1.9.0

[Mariano (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806984#comment-16806984
 ] 

Mariano commented on AVRO-2250:
---

Yes would love to have any ETA, we are currently blocked by 
https://issues.apache.org/jira/browse/AVRO-2217 as we are not allowed to 
release with vulnerabilities. Do we have an ETA

> Release 1.9.0
> -
>
> Key: AVRO-2250
> URL: https://issues.apache.org/jira/browse/AVRO-2250
> Project: Apache Avro
>  Issue Type: Task
>  Components: release
>Reporter: Nandor Kollar
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AVRO-2363) avro.schema.Schema in py3 prevents same record type from being used multiple times in same schema

[Gabriel Andersson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806713#comment-16806713
 ] 

Gabriel Andersson edited comment on AVRO-2363 at 4/1/19 12:33 PM:
--

Pull request was closed.

The reason was that when importing the schemas in correct order there was no 
problem.

solved by doing this.
{code:java}
[
  {
"fields": [
  {
"name": "value",
"type":  "string"
  }
],
"name": "C",
"namespace": "namespace"
"type": "record"
  },
  {
"fields": [
  {
"name": "c",
"type": "namespace.C"
  }
],
"name": "B",
"namespace": "namespace",
"type": "record"
  },
  {
"doc": "",
"fields": [
  {
"name": "b",
"type": "namespace.B"
  },
  {
"name": "c",
"type": "namespace.C"
  }
],
"name": "A",
"namespace": "namespace",
"type": "record"
  }
]

{code}
 


was (Author: krabban91):
Pull request was closed.

The reason was that when importing the schemas in correct order there was no 
problem.

solved by doing this.
{code:java}
[
  {
"fields": [
  {
"name": "value",
"type":  "string"
  }
],
"name": "C",
"namespace": "namespace"
"type": "record"
  },
  {
"fields": [
  {
"name": "c",
"type": namespace.C
  }
],
"name": "B",
"namespace": "namespace",
"type": "record"
  },
  {
"doc": "",
"fields": [
  {
"name": "b",
"type": namespace.B
  },
  {
"name": "c",
"type": namespace.C
  }
],
"name": "A",
"namespace": "namespace",
"type": "record"
  }
]

{code}
 

> avro.schema.Schema in py3 prevents same record type from being used multiple 
> times in same schema
> -
>
> Key: AVRO-2363
> URL: https://issues.apache.org/jira/browse/AVRO-2363
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.9.0
>Reporter: Gabriel Andersson
>Priority: Major
>
> After mostly worked in Java with avro, schemas have been used in an 
> hierachial way. I.e. Schema A imports Schema B and Schema C, Schema B imports 
> schema C. 
>  
> This works in a confluent setting. But when loading the schema, the 
> application crashes due to reusing record types. 
> The following relaxes this relation.
> avro.schema.py
>  
> {code:java}
> diff --git a/lang/py3/avro/schema.py b/lang/py3/avro/schema.py
> index 7ce29731..63af3581 100644
> --- a/lang/py3/avro/schema.py
> +++ b/lang/py3/avro/schema.py
> @@ -397,9 +397,6 @@ class Names(object):
>  if schema.fullname in VALID_TYPES:
>raise SchemaParseException(
>'%s is a reserved type name.' % schema.fullname)
> -if schema.fullname in self.names:
> -  raise SchemaParseException(
> -  'Avro name %r already exists.' % schema.fullname)
>  
>  logger.log(DEBUG_VERBOSE, 'Register new name for %r', schema.fullname)
>  self._names[schema.fullname] = schema
> {code}
>  
>  
> Example schema:
> {code:java}
> {
>   "doc": "",
>   "fields": [
> {
>   "default": null,
>   "doc": "Optional. The context used for tracing of flow.",
>   "name": "b",
>   "type": [
> "null",
> {
>   "doc": "",
>   "fields": [
> {
>   "doc": "",
>   "name": "c",
>   "type": {
> "doc": "",
> "fields": [
>   {
> "doc": "",
> "name": "value",
> "type":  "string"
>   
>   }
> ],
> "name": "C",
> "type": "record"
>   }
> }
>   ],
>   "name": "B",
>   "type": "record"
> }
>   ]
> },
> {
>   "doc": "",
>   "name": "c",
>   "type": {
> "doc": "",
> "fields": [
>   {
> "doc": "",
> "name": "value",
> "type":  "string"
>   }
> ],
> "name": "C",
> "type": "record"
>   }
> }
>   ],
>   "name": "A",
>   "namespace": "namespace",
>   "type": "record"
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2363) avro.schema.Schema in py3 prevents same record type from being used multiple times in same schema

[Gabriel Andersson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806713#comment-16806713
 ] 

Gabriel Andersson commented on AVRO-2363:
-

Pull request was closed.

The reason was that when importing the schemas in correct order there was no 
problem.

solved by doing this.
{code:java}
[
  {
"fields": [
  {
"name": "value",
"type":  "string"
  }
],
"name": "C",
"namespace": "namespace"
"type": "record"
  },
  {
"fields": [
  {
"name": "c",
"type": namespace.C
  }
],
"name": "B",
"namespace": "namespace",
"type": "record"
  },
  {
"doc": "",
"fields": [
  {
"name": "b",
"type": namespace.B
  },
  {
"name": "c",
"type": namespace.C
  }
],
"name": "A",
"namespace": "namespace",
"type": "record"
  }
]

{code}
 

> avro.schema.Schema in py3 prevents same record type from being used multiple 
> times in same schema
> -
>
> Key: AVRO-2363
> URL: https://issues.apache.org/jira/browse/AVRO-2363
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.9.0
>Reporter: Gabriel Andersson
>Priority: Major
>
> After mostly worked in Java with avro, schemas have been used in an 
> hierachial way. I.e. Schema A imports Schema B and Schema C, Schema B imports 
> schema C. 
>  
> This works in a confluent setting. But when loading the schema, the 
> application crashes due to reusing record types. 
> The following relaxes this relation.
> avro.schema.py
>  
> {code:java}
> diff --git a/lang/py3/avro/schema.py b/lang/py3/avro/schema.py
> index 7ce29731..63af3581 100644
> --- a/lang/py3/avro/schema.py
> +++ b/lang/py3/avro/schema.py
> @@ -397,9 +397,6 @@ class Names(object):
>  if schema.fullname in VALID_TYPES:
>raise SchemaParseException(
>'%s is a reserved type name.' % schema.fullname)
> -if schema.fullname in self.names:
> -  raise SchemaParseException(
> -  'Avro name %r already exists.' % schema.fullname)
>  
>  logger.log(DEBUG_VERBOSE, 'Register new name for %r', schema.fullname)
>  self._names[schema.fullname] = schema
> {code}
>  
>  
> Example schema:
> {code:java}
> {
>   "doc": "",
>   "fields": [
> {
>   "default": null,
>   "doc": "Optional. The context used for tracing of flow.",
>   "name": "b",
>   "type": [
> "null",
> {
>   "doc": "",
>   "fields": [
> {
>   "doc": "",
>   "name": "c",
>   "type": {
> "doc": "",
> "fields": [
>   {
> "doc": "",
> "name": "value",
> "type":  "string"
>   
>   }
> ],
> "name": "C",
> "type": "record"
>   }
> }
>   ],
>   "name": "B",
>   "type": "record"
> }
>   ]
> },
> {
>   "doc": "",
>   "name": "c",
>   "type": {
> "doc": "",
> "fields": [
>   {
> "doc": "",
> "name": "value",
> "type":  "string"
>   }
> ],
> "name": "C",
> "type": "record"
>   }
> }
>   ],
>   "name": "A",
>   "namespace": "namespace",
>   "type": "record"
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2363) avro.schema.Schema in py3 prevents same record type from being used multiple times in same schema

[Gabriel Andersson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806625#comment-16806625
 ] 

Gabriel Andersson commented on AVRO-2363:
-

Pull request created:
https://github.com/apache/avro/pull/499

> avro.schema.Schema in py3 prevents same record type from being used multiple 
> times in same schema
> -
>
> Key: AVRO-2363
> URL: https://issues.apache.org/jira/browse/AVRO-2363
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python
>Affects Versions: 1.9.0
>Reporter: Gabriel Andersson
>Priority: Major
>
> After mostly worked in Java with avro, schemas have been used in an 
> hierachial way. I.e. Schema A imports Schema B and Schema C, Schema B imports 
> schema C. 
>  
> This works in a confluent setting. But when loading the schema, the 
> application crashes due to reusing record types. 
> The following relaxes this relation.
> avro.schema.py
>  
> {code:java}
> diff --git a/lang/py3/avro/schema.py b/lang/py3/avro/schema.py
> index 7ce29731..63af3581 100644
> --- a/lang/py3/avro/schema.py
> +++ b/lang/py3/avro/schema.py
> @@ -397,9 +397,6 @@ class Names(object):
>  if schema.fullname in VALID_TYPES:
>raise SchemaParseException(
>'%s is a reserved type name.' % schema.fullname)
> -if schema.fullname in self.names:
> -  raise SchemaParseException(
> -  'Avro name %r already exists.' % schema.fullname)
>  
>  logger.log(DEBUG_VERBOSE, 'Register new name for %r', schema.fullname)
>  self._names[schema.fullname] = schema
> {code}
>  
>  
> Example schema:
> {code:java}
> {
>   "doc": "",
>   "fields": [
> {
>   "default": null,
>   "doc": "Optional. The context used for tracing of flow.",
>   "name": "b",
>   "type": [
> "null",
> {
>   "doc": "",
>   "fields": [
> {
>   "doc": "",
>   "name": "c",
>   "type": {
> "doc": "",
> "fields": [
>   {
> "doc": "",
> "name": "value",
> "type":  "string"
>   
>   }
> ],
> "name": "C",
> "type": "record"
>   }
> }
>   ],
>   "name": "B",
>   "type": "record"
> }
>   ]
> },
> {
>   "doc": "",
>   "name": "c",
>   "type": {
> "doc": "",
> "fields": [
>   {
> "doc": "",
> "name": "value",
> "type":  "string"
>   }
> ],
> "name": "C",
> "type": "record"
>   }
> }
>   ],
>   "name": "A",
>   "namespace": "namespace",
>   "type": "record"
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)