clairemcginty opened a new pull request, #3753: URL: https://github.com/apache/parquet-java/pull/3753
<!-- Thanks for opening a pull request! If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose and format pull request title as below: GH-${GITHUB_ISSUE_ID}: ${SUMMARY} or simply use the title below if it is a minor issue: MINOR: ${SUMMARY} --> ### Rationale for this change parquet-avro supports writing both "old" and "new" list encodings via the [parquet.avro.write-old-list-structure](https://github.com/apache/parquet-java/blob/apache-parquet-1.18.0/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L72-L73) config. "old" encodings (aka "2-level"), which wrap the list in a `repeated group array` schema, are the default; "new" encodings (aka "3-level") are opt-in. On the reader side, if you're using `ParquetAvroReader` to read data that was written using `ParquetAvroWriter`, and don't specify a projection, both type sof list encoding get parsed automatically from a combination of the file schema + the `parquet.avro.schema` metadata key. There's no need to set `parquet.avro.write-old-list-structure` key in your Configuration. However, if you're either: - specifying a projection (`AvroReadSupport.setRequestedProjection(...)`), or - reading data _not_ written using ParquetAvroWriter (and thus not containing the `parquet.avro.schema` metadata key), 3-levle list encodings will not be parsed correctly - the reader will inject an extra nested record, named `element`, into the list item type. As a reader this introduces some pain, since you have to look up the underlying file metadata of the upstream Parquet file, and risk reading incorrect data. This PR attempts to automatically detect new list encodings based on the writer file schema. lmk what you think of this change. Automatic inference is always a bit risky, but I tried to be conservative with the approach (only set the list structure property if _all_ list fields in the schema use 3-level encoding; don't override `parquet.avro.write-old-list-structure` if the user is already setting it). any ideas for a better approach here are welcome - this is becoming more of a pain point as 3-level lists become a more popular option among other writer sdks. ### What changes are included in this PR? A new read configuration property `parquet.avro.read.autoDetectListStructure` (defaulting to true) that will instruct AvroReadSupport to automatically set List configuration properties based on parsing the writer file schema. ### Are these changes tested? Yes, unit tests + locally on real data. ### Are there any user-facing changes? Yes, since the new property defaults to `true` - it would impact anyone who's reading 3-level list data without setting the `parquet.avro.write-old-list-structure` key and who's relying on/working around the incorrectly formatted data (e.g. `{"locations": [{"element": {"latitude": 0.0, "longitude": 180.0}}, ...]}` instead of `{"locations": [{"latitude": 0.0, "longitude": 180.0}, ...]}` . additionally, this change also modifies the underlying Configuration object to add the properties. <!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. --> <!-- Closes #${GITHUB_ISSUE_ID} --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
