Zoltan Borok-Nagy has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24988 )
Change subject: IMPALA-15443: Fix Parquet stats of widened INT32/INT64/FLOAT columns ...................................................................... IMPALA-15443: Fix Parquet stats of widened INT32/INT64/FLOAT columns Parquet min/max statistics and page indexes were decoded at the width of the table's type instead of the Parquet physical type. For widened columns (ALTER TABLE CHANGE COLUMN, Iceberg type promotion, INT/BIGINT read as DECIMAL since IMPALA-13625) the decoded values were garbage, so row groups and pages were skipped wrongly and queries silently returned fewer rows. Affected: - DECIMAL stored as INT32/INT64 and read into a wider decimal slot: the generic ParquetPlainEncoder::DecodeNoBoundsCheck() copied 8 or 16 bytes out of a 4 or 8 byte value. This affected stats and page index conjuncts and min/max runtime filters. - INT32 read as BIGINT/DOUBLE and FLOAT read as DOUBLE in page-level min/max filters: SkipPagesBatch() binary searched the raw page index values of sorted columns, and DecodeBatchOneBoundsCheckFastTrack() memcpy-ed them. This patch adds DecodeNoBoundsCheck() specializations that sign-extend INT32/INT64 values into Decimal8Value/Decimal16Value. Page-level min/max filters decoded the page index with ColumnStatsReader::ReadFromStringsBatch(), an unrolled variant of ReadFromString(). It had more bugs that did not need widening: after an all-NULL page, e.g. in files sorted NULLS FIRST, it left up to 7 values at the end of the span of pages undecoded, which could skip pages wrongly or crash on STRING columns (likely IMPALA-11665). It also ignored decoding errors, checked only the first value for NaN and did not decode INT64 or convert INT96 timestamps (IMPALA-10793). This patch removes ReadFromStringsBatch(), its helpers and the binary search on raw values. SkipPagesBatch() decodes the min/max of each page with ReadStatFromIndex(), like page index conjuncts do, binary searches only if all pages were decoded, and keeps the pages it cannot decode instead of disabling page filtering for the row group. ParquetPageIndex::DeserializeColumnIndex() now returns an error if the column index does not have a min and a max value for each page. The verification mode of the binary search no longer drops the skip ranges collected before it. Testing: - Added BE tests for the decoders and ColumnStatsReader. - Added EE tests for stats, page index and min/max filtering on widened columns and on pages after a NULL page, with a new Parquet test file. Assisted-by: Claude Opus 5.5 <[email protected]> Change-Id: Ia39297e906802ed698178943b9c60b79eb905cdb --- M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h A be/src/exec/parquet/parquet-column-stats-test.cc M be/src/exec/parquet/parquet-column-stats.cc M be/src/exec/parquet/parquet-column-stats.h M be/src/exec/parquet/parquet-column-stats.inline.h M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-page-index.cc M be/src/exec/parquet/parquet-page-index.h M be/src/exec/parquet/parquet-plain-test.cc M testdata/data/README A testdata/data/decimal_int32_int64_page_index.parquet A testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_widened_columns.test M testdata/workloads/functional-query/queries/QueryTest/parquet-decimal-precision-and-scale-widening.test M testdata/workloads/functional-query/queries/QueryTest/parquet-type-widening.test M tests/query_test/test_runtime_filters.py M tests/query_test/test_scanners.py 18 files changed, 1,014 insertions(+), 411 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/24988/2 -- To view, visit http://gerrit.cloudera.org:8080/24988 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia39297e906802ed698178943b9c60b79eb905cdb Gerrit-Change-Number: 24988 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
