kazuyukitanimura opened a new pull request, #7148:
URL: https://github.com/apache/datafusion-comet/pull/7148
## Which issue does this PR close?
Closes #7029.
## Rationale for this change
The native Parquet scan returns wrong values, with no error, for INT96
timestamps outside
1677-09-21 to 2262-04-11 when they sit inside a list that is not in the
standard `list` / `element`
layout. That covers Spark's `writeLegacyFormat=true` (`bag` / `array`),
Hive's `bag` /
`array_element`, and the two-level `repeated int96` form. `9999-12-31`
came back as
`1816-03-30 06:03:09.066278`.
The scan sets `coerce_int96` so DataFusion's opener widens INT96 leaves to
microseconds before
arrow-rs decodes them. `Int96Coercer` finds a leaf by building its Parquet
path with a hard-coded
`.list.` between a list and its child, so a legacy layout never matches
and the leaf stays
`Timestamp(Nanosecond)`, which wraps. This is apache/datafusion#26393,
still open upstream.
## What changes are included in this PR?
- New `native/core/src/parquet/int96_coercion.rs`.
`coerce_missed_int96_leaves` pairs Arrow leaves
with Parquet columns by ordinal, which holds for any layout, and finds
the INT96 leaves that
DataFusion's coercion leaves as `Timestamp(Nanosecond)`. It rebuilds the
footer with an
`ARROW:schema` hint that makes those leaves `Timestamp(Microsecond,
UTC)`. arrow-rs applies the
hint when it derives the file schema, so the opener sees them already
coerced and its own pass
skips them. Files with no INT96 column inside a list return without
building a schema.
- `EagerPageIndexReader::get_metadata` applies it last, per open, so the
conversion check, the
Variant rewrite and the INT96 leaf stamp see the footer as read. The
rebuilt copy is never
written to the shared metadata cache.
- `coerce_int96` / `coerce_int96_tz` values are now shared constants so
the scan options and this
module cannot drift.
- User guide: the #7029 entry now describes only the remaining gap.
Known gap: an open with decryption properties is left as read, because the
parquet API cannot
carry a file decryptor across the footer rebuild. Encryption is configured
per scan, so every file
of an encryption-configured scan still reads these values wrong, plaintext
files included. The user
guide says so.
Once DataFusion fixes apache/datafusion#26393,
`coerces_the_leaf_of_every_legacy_list_layout` fails
with "DataFusion already coerces this layout", which is the signal to
delete this module.
## How are these changes tested?
- Rust unit tests in `int96_coercion.rs`: the four legacy layouts read the
issue's four timestamps
correctly, the standard layout is left to DataFusion, a mixed schema
(standard list, legacy list,
top-level INT96, legacy list in a struct), idempotence, and preservation
of other footer entries,
row groups and any existing `ARROW:schema`.
- Scala: `ParquetReadSuite` "INT96 timestamps outside the nanosecond range
inside lists in either
layout" writes `array<timestamp>`, `array<struct<ts>>` and
`map<string,array<timestamp>>` with
`writeLegacyFormat` on and off, and checks Comet against Spark and
against the pinned rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
