[jira] [Work started] (AVRO-1833) Release 1.8.1
[ https://issues.apache.org/jira/browse/AVRO-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AVRO-1833 started by Ryan Blue. --- > Release 1.8.1 > - > > Key: AVRO-1833 > URL: https://issues.apache.org/jira/browse/AVRO-1833 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.8.1 >Reporter: Ryan Blue >Assignee: Ryan Blue > Fix For: 1.8.1 > > > Please link issues that should be included in the 1.8.1 release as blockers > of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
1.8.1 released! (Almost)
I've pushed the changes for the 1.8.1 release. I don't have permission to push to npm, rubygems, or pypi yet so those are still pending. I'll send out the formal announcement tomorrow when mirrors are updated. Thanks, everyone! rb -- Ryan Blue
[jira] [Updated] (AVRO-1810) GenericDatumWriter broken with Enum
[
https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blue updated AVRO-1810:
Fix Version/s: (was: 1.8.1)
> GenericDatumWriter broken with Enum
> ---
>
> Key: AVRO-1810
> URL: https://issues.apache.org/jira/browse/AVRO-1810
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Ryon Day
>Priority: Blocker
>
> {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD}
> Using the GenericDatumWriter with either Generic OR SpecificRecord will break
> if an Enum is present.
> {panel}
> {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD}
> I have been tracking Avro decoding oddities for a while.
> The tests for this issue can be found
> [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java]
> {panel}
> {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD}
> Due to the debacle that is the Avro "UTF8" object, we have been avoiding it
> by using the following scheme:
> * Write incoming records to a byte array using the GenericDatumWriter
> * Read back the byte array to our compiled Java domain objects using a
> SpecificDatumWriter
> This worked great with Avro 1.7.7, and this is a binary-incompatable breaking
> change with 1.8.0.
> This would appear to be caused by an addition in the
> {{GenericDatumWriter:163-164}}:
> {code}
> if (!data.isEnum(datum))
> throw new AvroTypeException("Not an enum: "+datum);
> {code}
> {panel}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1849) C++ printJson fails on record with no fields
[
https://issues.apache.org/jira/browse/AVRO-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blue updated AVRO-1849:
Fix Version/s: (was: 1.8.1)
> C++ printJson fails on record with no fields
>
>
> Key: AVRO-1849
> URL: https://issues.apache.org/jira/browse/AVRO-1849
> Project: Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.7.7, 1.8.0
> Environment: Not relevant
>Reporter: Simon Woodford
>Priority: Minor
>
> The function NodeRecord::printJson in lang/c++/impl/NodeImpl.cc does not
> correctly handle a record with no fields. It injects an extra closing curly
> brace, and the result is invalid JSON.
> Starting with a schema
> {
> "type": "record",
> "name": "Example",
> "fields" : [
> ]
> }
> and parsing it to create a ValidSchema, then calling toJson on the
> ValidSchema generates
> {
> "type": "record",
> "name": "Example",
> "fields" : [
> }
> ]
> }
> A record with no fields is unusual but we have encountered use cases for it,
> avro does not invalidate a record with no fields, and I've confirmed that
> Java, C and C# handle this case correctly. (I have not checked the other
> supported languages.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1833) Release 1.8.1
[ https://issues.apache.org/jira/browse/AVRO-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295286#comment-15295286 ] Ryan Blue commented on AVRO-1833: - [~tomwhite], can you add me as a maintainer for Avro on RubyGems and PyPI? My usernames are blue (RubyGems) and rdblue (PyPI). [~mtth], can you add me as a maintainer for Avro on NPM? My username is rdblue. Thanks! > Release 1.8.1 > - > > Key: AVRO-1833 > URL: https://issues.apache.org/jira/browse/AVRO-1833 > Project: Avro > Issue Type: New Feature >Affects Versions: 1.8.1 >Reporter: Ryan Blue >Assignee: Ryan Blue > Fix For: 1.8.1 > > > Please link issues that should be included in the 1.8.1 release as blockers > of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1850) Align JSON and binary record serialization
[
https://issues.apache.org/jira/browse/AVRO-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Kay updated AVRO-1850:
Description:
The documentation describes the encoding of Avro records as:
bq.Binary: A record is encoded by encoding the values of its fields in the
order that they are declared. In other words, a record is encoded as just the
concatenation of the encodings of its fields. Field values are encoded per
their schema.
bq.JSON: Except for unions, the JSON encoding is the same as is used to encode
field default values.
The _field default values_ table says that records and maps are both encoded as
JSON type _object_.
*Enhancement:*
There is currently no way to write an Avro schema describing a JSON array of
positional parameters (i.e. an array containing variables of possibly different
type). An Avro record is the datatype representing an ordered collection of
values. For consistency with the binary encoding, and to allow Avro to
represent a schema for JSON tuples, encoding should say:
bq.JSON: Except for unions and records, the JSON encoding is the same as is
used to encode field default values. A record is encoded as an array by
encoding the values of its fields in the order that they are declared.
For the example schema:
{noformat}
{"namespace": "example.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
{noformat}
the JSON encoding currently converts an Avro record to an Avro map (JSON
object):
{noformat}
{ "name": "Joe",
"favorite_number": 42,
"favorite_color": null }
{noformat}
Instead Avro records should be encoded in JSON in the same manner as they are
encoded in binary, as a JSON array containing the fields in the order they are
defined:
{noformat}
["Joe", 42, null]
{noformat}
The set of JSON texts validated by the example Avro schema and by the
corresponding JSON schema should be equal:
{noformat}
{
"$schema": "http://json-schema.org/draft-04/schema#;,
"type": "array",
"name": "User",
"items": [
{"name":"name", "type": "string"},
{"name":"favorite_number", "type":["integer","null"]},
{"name":"favorite_color", "type":["string","null"]}
]
}
{noformat}
was:
The documentation describes the encoding of Avro records as:
bq.Binary: A record is encoded by encoding the values of its fields in the
order that they are declared. In other words, a record is encoded as just the
concatenation of the encodings of its fields. Field values are encoded per
their schema.
bq.JSON: Except for unions, the JSON encoding is the same as is used to encode
field default values.
The _field default values_ table says that records and maps are both encoded as
JSON type _object_.
*Enhancement:*
There is currently no way to write an Avro schema describing a JSON array of
positional parameters (i.e. an array containing variables of possibly different
type). An Avro record is the datatype representing an ordered collection of
values. For consistency with the binary encoding, and to allow Avro to
represent a schema for JSON tuples, encoding should say:
bq.JSON: Except for unions and records, the JSON encoding is the same as is
used to encode field default values. A record is encoded as an array by
encoding the values of its fields in the order that they are declared.
For the example schema:
{noformat}
{"namespace": "example.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
{noformat}
the JSON encoding currently converts an Avro record to an Avro map (JSON
object):
{noformat}
{ "name": "Joe",
"favorite_number": 42,
"favorite_color": null }
{noformat}
Instead Avro records should be encoded in JSON in the same manner as they are
encoded in binary, as a JSON array containing the fields in the order they are
defined:
{noformat}
["Joe", 42, null]
{noformat}
> Align JSON and binary record serialization
> --
>
> Key: AVRO-1850
> URL: https://issues.apache.org/jira/browse/AVRO-1850
> Project: Avro
> Issue Type: Improvement
>Reporter: David Kay
> Labels: Encoding, Record
>
> The documentation describes the encoding of Avro records as:
> bq.Binary: A record is encoded by encoding the values of its fields in the
> order that they are declared. In other words, a record is encoded as just the
> concatenation of the encodings of its fields. Field values are encoded per
> their schema.
> bq.JSON: Except for unions, the JSON encoding is the same as is used to
> encode field default values.
> The _field default
[jira] [Created] (AVRO-1850) Align JSON and binary record serialization
David Kay created AVRO-1850:
---
Summary: Align JSON and binary record serialization
Key: AVRO-1850
URL: https://issues.apache.org/jira/browse/AVRO-1850
Project: Avro
Issue Type: Improvement
Reporter: David Kay
The documentation describes the encoding of Avro records as:
bq.Binary: A record is encoded by encoding the values of its fields in the
order that they are declared. In other words, a record is encoded as just the
concatenation of the encodings of its fields. Field values are encoded per
their schema.
bq.JSON: Except for unions, the JSON encoding is the same as is used to encode
field default values.
The _field default values_ table says that records and maps are both encoded as
JSON type _object_.
*Enhancement:*
There is currently no way to write an Avro schema describing a JSON array of
positional parameters (i.e. an array containing variables of possibly different
type). An Avro record is the datatype representing an ordered collection of
values. For consistency with the binary encoding, and to allow Avro to
represent a schema for JSON tuples, encoding should say:
bq.JSON: Except for unions and records, the JSON encoding is the same as is
used to encode field default values. A record is encoded as an array by
encoding the values of its fields in the order that they are declared.
For the example schema:
{noformat}
{"namespace": "example.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
{noformat}
the JSON encoding currently converts an Avro record to an Avro map (JSON
object):
{noformat}
{ "name": "Joe",
"favorite_number": 42,
"favorite_color": null }
{noformat}
Instead Avro records should be encoded in JSON in the same manner as they are
encoded in binary, as a JSON array containing the fields in the order they are
defined:
{noformat}
["Joe", 42, null]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1847) IDL compiler uses ByteBuffer for decimal type even if logical type is supported
[ https://issues.apache.org/jira/browse/AVRO-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yibing Shi updated AVRO-1847: - Attachment: AVRO-1847.2.patch [~rdblue], I have attached another patch that adds the missing parts mentioned in your previous comment. Could you please help have another look? > IDL compiler uses ByteBuffer for decimal type even if logical type is > supported > > > Key: AVRO-1847 > URL: https://issues.apache.org/jira/browse/AVRO-1847 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Yibing Shi > Attachments: AVRO-1847.1.patch, AVRO-1847.2.patch > > > Version 1.8.0 has added the support of logical types. A conversion class > (Conversions.DecimalConversion) has also been added for decimal type. > However, the IDL compiler still uses ByteBuffer for decimal types, which is > not the same behaviour as data, time or timestamp type (added in AVRO-1684). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
