Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/24887 )
Change subject: IMPALA-15375: Materialize the file position slot before
evaluating conjuncts
......................................................................
IMPALA-15375: Materialize the file position slot before evaluating conjuncts
With late materialization only the readers whose slots a conjunct or runtime
filter reads are materialized before the conjuncts are evaluated; the rest
are materialized afterwards, for the surviving rows only.
The file position slot has no reader of its own. CreateColumnReaders()
attaches it to (*column_readers)[0], whichever reader that is. The split
only looked at each reader's own slot, so a conjunct on FILE__POSITION did
not move that reader into the filter group. When it was a non-filter reader
the conjunct was evaluated against a slot that had not been written yet,
reading whatever InitTupleBuffer() left there.
Rows went missing silently:
set parquet_late_materialization_threshold=1;
select string_col, file__position from alltypes_tiny_pages
where id = 6285 and file__position = 7000;
returned no rows, against one with late materialization disabled. The
Iceberg _row_id virtual column is affected the same way on V3 tables, as it
is rewritten to an expression over FILE__POSITION.
Treat the reader that writes the slot as a filter reader whenever a conjunct
reads it. That is always safe, since filter readers are simply materialized
for every row. If the reader is a collection it becomes filter_readers_[0]
and the existing max_rep_level() check falls back to
AssembleRowsWithoutLateMaterialization.
Testing:
- Added two queries to virtual-column-file-position-parquet.test, which
runs with parquet_late_materialization_threshold in {-1, 1, 17}. The
second keeps another non-filter column so late materialization stays
enabled. Both fail without the fix.
- Ran test_scanners.py -k "virtual_column or mixing" and
test_parquet_late_materialization.py.
Change-Id: I13fc71b5f81088e7efd1d64b8828b30268c40fb8
Assisted-by: Claude Opus 5 (1M context) <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/24887
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M
testdata/workloads/functional-query/queries/QueryTest/virtual-column-file-position-parquet.test
2 files changed, 30 insertions(+), 5 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I13fc71b5f81088e7efd1d64b8828b30268c40fb8
Gerrit-Change-Number: 24887
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Xuebin Su <[email protected]>
