DsThakurRawat opened a new pull request, #26588:
URL: https://github.com/apache/datafusion/pull/26588
## Which issue does this PR close?
- Closes #26393.
## Rationale for this change
In `datafusion/datasource-parquet/src/schema_coercion.rs`, `Int96Coercer`
matches leaf column paths against `int96_fields` by traversing the schema with
a DFS stack. When encountering a list field (`DataType::List`), the previous
implementation hard-coded:
```rust
child_path.push(".list.");
child_path.push(unprocessed_child.name());
```
However, Parquet files written by Spark with legacy formatting
(`spark.sql.parquet.writeLegacyFormat=true`), Hive, or older Parquet writers
produce alternative list structures:
- Spark legacy 3-level list: repeated group named `bag`, child element named
`array` (e.g. `at.bag.array`).
- Hive legacy 3-level list: repeated group named `bag`, child element named
`array_element` (e.g. `at.bag.array_element`).
- 2-level list: repeated primitive or group (e.g. `at.array`).
- Repeated primitive at root or struct level without a synthetic list group.
Because `child_path` assumed `.list.<child_name>`, these legacy schemas
resulted in non-matching paths such as `at.list.array` vs `at.bag.array`,
leaving INT96 timestamps as `Timestamp(Nanosecond, None)` instead of coercing
them to the target resolution (e.g., `Timestamp(Microsecond, None)`).
In addition, container types like `DataType::LargeList` and
`DataType::FixedSizeList` were previously ignored by `Int96Coercer`.
## What changes are included in this PR?
- Pass `Option<&'a ParquetType>` in the `StackContext` tuple during schema
DFS traversal to inspect the matching Parquet schema group/primitive types.
- Introduce `get_parquet_list_element_info`:
- For primitive repeated fields (2-level lists of primitives or repeated
root fields), uses the actual field name without superfluous `.list.` segments.
- For 3-level lists, dynamically extracts the repeated group name (`list`,
`bag`, etc.) and element field name (`element`, `array`, `array_element`, etc.).
- For 2-level lists of structs with multiple fields, resolves the repeated
group directly as the struct element.
- Gracefully falls back to standard conventions if Parquet type info is
missing or unannotated.
- Expand list pattern matching to support `DataType::LargeList` and
`DataType::FixedSizeList`.
- Check both `int96_fields.contains(path)` and
`current_parquet_type.is_some_and(|pt| pt.is_primitive() &&
pt.get_physical_type() == Type::INT96)`.
## What is the testing strategy for this PR?
- Added `coerce_int96_to_resolution_with_legacy_list_layouts` unit test
covering:
- `standard` 3-level (`list/element`)
- `spark_legacy` 3-level (`bag/array`)
- `hive` 3-level (`bag/array_element`)
- `two_level` (`array`)
- `two_level_struct` (`array` with multiple struct fields)
- `repeated_primitive_root` (`repeated int96 at`)
- `nested_lists_legacy` (multi-dimensional list of lists with legacy
layouts)
- Added `coerce_int96_to_resolution_with_large_and_fixed_lists` unit test
testing `LargeList` and `FixedSizeList` containers.
- Verified all 295 unit tests in `datafusion-datasource-parquet` pass
without regressions.
- Ran `cargo fmt --check` and `cargo clippy -p datafusion-datasource-parquet
--all-targets -- -D warnings`.
## Are there any user-facing changes?
No breaking changes. Queries reading Parquet files with legacy list
structures containing INT96 timestamps will now correctly coerce to the
configured timestamp resolution instead of retaining nanosecond precision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
