Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24988
Change subject: IMPALA-15443: Fix Parquet stats of widened INT32/INT64/FLOAT columns ...................................................................... IMPALA-15443: Fix Parquet stats of widened INT32/INT64/FLOAT columns Parquet min/max statistics and page indexes were decoded at the width of the table's type instead of the Parquet physical type. For widened columns (ALTER TABLE CHANGE COLUMN, Iceberg type promotion, INT/BIGINT read as DECIMAL since IMPALA-13625) the decoded values were garbage, so row groups and pages were skipped wrongly and queries silently returned fewer rows. Affected: - DECIMAL stored as INT32/INT64 and read into a wider decimal slot: the generic ParquetPlainEncoder::DecodeNoBoundsCheck() copied 8 or 16 bytes out of a 4 or 8 byte value. This affected stats and page index conjuncts and min/max runtime filters. - INT32 read as BIGINT/DOUBLE and FLOAT read as DOUBLE in page-level min/max filters, both in DecodeBatchOneBoundsCheckFastTrack() and in the sorted-column fast code path of SkipPagesBatch(). This patch adds DecodeNoBoundsCheck() specializations that sign-extend INT32/INT64 values into Decimal8Value/Decimal16Value, and takes the fast paths only if the table's type is identical to the physical type (ColumnStatsReader::HasIdenticalStorageType()). It also fixes the batch decoders, which did not decode the last values of a batch that started after the first page, e.g. after a NULL page. Widened sorted columns now use them, and the regular path of page-level min/max filters had this bug for all types. Testing: - Added BE tests for the decoders and ColumnStatsReader. - Added EE tests for stats, page index and min/max filtering on widened columns, with a new Parquet test file. Assisted-by: Claude Opus 5.5 <[email protected]> Change-Id: Ia39297e906802ed698178943b9c60b79eb905cdb --- M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc A be/src/exec/parquet/parquet-column-stats-test.cc M be/src/exec/parquet/parquet-column-stats.cc M be/src/exec/parquet/parquet-column-stats.h M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-plain-test.cc M testdata/data/README A testdata/data/decimal_int32_int64_page_index.parquet A testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_widened_columns.test M testdata/workloads/functional-query/queries/QueryTest/parquet-decimal-precision-and-scale-widening.test M testdata/workloads/functional-query/queries/QueryTest/parquet-type-widening.test M tests/query_test/test_runtime_filters.py M tests/query_test/test_scanners.py 14 files changed, 1,196 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/24988/1 -- To view, visit http://gerrit.cloudera.org:8080/24988 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia39297e906802ed698178943b9c60b79eb905cdb Gerrit-Change-Number: 24988 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
