[
https://issues.apache.org/jira/browse/PARQUET-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17179278#comment-17179278
]
Sean Colgan commented on PARQUET-1681:
--------------------------------------
Thanks to all of you for producing great software! :D
I'd just like to comment on this ticket that this appears to be the issue
forcing us to exclude some integration test cases from our parquet writer. We
have a system that consumes Avro, transforms it, and uses this library to emit
Parquet. It has some tests that read in that Parquet back to Avro, to validate
the transformations. We currently exclude some test cases since the parquet ->
avro reader blows up with this error.
> Avro's isElementType() change breaks the reading of some parquet(1.8.1) files
> -----------------------------------------------------------------------------
>
> Key: PARQUET-1681
> URL: https://issues.apache.org/jira/browse/PARQUET-1681
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-avro
> Affects Versions: 1.10.0, 1.9.1, 1.11.0
> Reporter: Xinli Shang
> Assignee: Xinli Shang
> Priority: Critical
>
> When using the Avro schema below to write a parquet(1.8.1) file and then read
> back by using parquet 1.10.1 without passing any schema, the reading throws
> an exception "XXX is not a group" . Reading through parquet 1.8.1 is fine.
> {
> "name": "phones",
> "type": [
> "null",
> {
> "type": "array",
> "items": {
> "type": "record",
> "name": "phones_items",
> "fields": [
>
> { "name": "phone_number",
> "type": [ "null",
> "string" ], "default": null
> }
> ]
> }
> }
> ],
> "default": null
> }
> The code to read is as below
> val reader =
> AvroParquetReader._builder_[SomeRecordType](parquetPath).withConf(*new*
> Configuration).build()
> reader.read()
> PARQUET-651 changed the method isElementType() by relying on Avro's
> checkReaderWriterCompatibility() to check the compatibility. However,
> checkReaderWriterCompatibility() consider the ParquetSchema and the
> AvroSchema(converted from File schema) as not compatible(the name in avro
> schema is ‘phones_items’, but the name is ‘array’ in Parquet schema, hence
> not compatible) . Hence return false and caused the “phone_number” field in
> the above schema to be considered as group type which is not true. Then the
> exception throws as .asGroupType().
> I didn’t try writing via parquet 1.10.1 would reproduce the same problem or
> not. But it could because the translation of Avro schema to Parquet schema is
> not changed(didn’t verify yet).
> I hesitate to revert PARQUET-651 because it solved several problems. I would
> like to hear the community's thoughts on it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
