LuciferYang opened a new issue, #68118:
URL: https://github.com/apache/doris/issues/68118
### Search before asking
- [X] I had searched in the issues and found no similar issues.
### Version
master, dea1b99e528.
### What's Wrong?
The v1 Parquet reader builds a column zone map from a TIMESTAMP column's
min/max statistics without the monotonicity fence the v2 reader applies, and a
validation guard for legacy INT96 stats is dead code. Both let non-conservative
bounds drive predicate pruning, which drops matching rows.
**1. No DST-rollback fence on the v1 timestamp range.**
`ParquetPredicate::parse_min_max_value`
(`be/src/format/parquet/parquet_predicate.h`) converts the physical min/max of
an adjusted-to-UTC TIMESTAMP into local civil time using the session time zone,
and the v1 slot-zone-map builder in `be/src/format/parquet/vparquet_reader.cpp`
records the result. Conversion to local civil time is not monotonic across a
DST fall-back: in `America/New_York`, UTC `[05:30, 06:30]` maps to local
`[01:30, 01:30]`, while an interior row at UTC `05:59` maps to `01:59`, which
is outside the converted range. The converted `[min, max]` is therefore not a
valid bound, and a predicate such as `dt > 01:45` prunes the row group even
though the `01:59` row satisfies it.
The v2 reader already guards this. `timestamp_min_max_is_safe`
(`be/src/format_v2/parquet/parquet_statistics.cpp:294-309`) calls
`utc_timestamp_range_is_monotonic`
(`be/src/format_v2/timestamp_statistics.h:40`) and treats a non-monotonic range
as unusable. The v1 path has no equivalent.
**2. The INT96 min == max validation never runs.**
The INT96 / DATETIMEV2 branch of `parse_min_max_value` reads both local
values from `min_field`:
```cpp
} else if (col_schema->parquet_schema.type == tparquet::Type::type::INT96 ||
logical_prim_type == TYPE_DATETIMEV2) {
auto min_value = min_field->get<TYPE_DATETIMEV2>();
auto max_value = min_field->get<TYPE_DATETIMEV2>(); // reads min_field
...
if (min_value != max_value) {
return Status::DataQualityError("invalid min/max value");
}
}
```
`be/src/format/parquet/parquet_predicate.h:344-345`. These are local copies;
the output fields `*min_field` / `*max_field` are set correctly earlier at
`:309-310`, so the recorded bounds are not affected. The effect is that the
`min_value != max_value` check compares a value with itself and can never fire,
so the PARQUET-1065 rejection of legacy INT96 stats whose min and max are
unreliable (the comment right below it explains why only min == max INT96 stats
are trustworthy) is dead. A legacy INT96 file with min != max is used for
pruning instead of being rejected.
Reading `max_field` in that second line is not a complete fix on its own:
the branch condition also matches INT64-backed DATETIMEV2, so `min_value !=
max_value` would then reject every multi-value DATETIMEV2 file, disabling
pruning for the common case. The INT96 min == max rule and the general
DATETIMEV2 path need to be separated.
Both problems are on the v1 path only (`enable_file_scanner_v2` defaults
true, so v2 is the default reader), and both are pre-existing. They surface for
any predicate pushed to a v1 Parquet TIMESTAMP column, including the
single-slot expression path and the column-vs-column path proposed in #67774.
### What You Expected?
A converted timestamp range that is not monotonic over a clock change, and a
legacy INT96 range that PARQUET-1065 marks unreliable, are treated as unusable
statistics rather than driving pruning. Pruning and row-level evaluation agree.
### How to Reproduce?
Query a v1 Parquet table (`set enable_file_scanner_v2 = false`) whose
TIMESTAMP column is adjusted to UTC, under a session time zone with DST, with a
row group whose UTC min/max straddle a fall-back transition, and a predicate
selecting a civil time inside the collapsed range. Compare results with and
without predicate pushdown. Not reproduced end to end here; this is from the
read path and the contrast with the v2 fence.
### Anything Else?
The v2 helpers `timestamp_min_max_is_safe` /
`utc_timestamp_range_is_monotonic` are the reference for the fence to add to
v1. Leaving the slot's zone map unset when the range is unusable lets every
consumer (single-slot expression, column-vs-column, and any other zone-map
reader) fall back conservatively without a per-consumer change.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
