Gabriel39 commented on code in PR #68780:
URL: https://github.com/apache/doris/pull/68780#discussion_r4218527609
##########
be/src/format_v2/parquet/reader/native/column_reader.cpp:
##########
@@ -1470,10 +1470,12 @@ Status ScalarColumnReader<IN_COLLECTION,
OFFSET_INDEX>::read_dictionary_filter(
int64_t right_row = 0;
if constexpr (OFFSET_INDEX == false) {
RETURN_IF_ERROR(_chunk_reader->parse_page_header());
- right_row = _chunk_reader->page_end_row();
} else {
- right_row = _chunk_reader->page_end_row();
+ // An indexed reader can fall back at runtime; dictionary filtering
then needs the current
+ // page header to establish row bounds just like ordinary and
fixed-width reads.
+ RETURN_IF_ERROR(_chunk_reader->ensure_first_data_page_parsed());
Review Comment:
Fixed in 2b5591cbba. Ordinary, fixed-width predicate, and dictionary
predicate reads now share page-range reconciliation: parse a selected page
before using the final range, and recompute both the range and page end if its
header discards the index. Pages outside the requested ranges still use indexed
skipping. The new regression reproduced values {20, 21, 30} instead of {20, 21,
22} before the fix. It now passes across all three paths, batch sizes 1/2/3,
cache hits/misses, and both expanding and shrinking ranges (including an empty
selection after reconciliation). All 733 related ASAN tests pass.
##########
be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp:
##########
@@ -1105,6 +1105,11 @@ Status ColumnChunkReader<IN_COLLECTION,
OFFSET_INDEX>::parse_page_header() {
}
}
const bool active_offset_index = _page_reader->has_active_offset_index();
Review Comment:
Fixed in 2b5591cbba, entirely within format_v2. A later V1 data page in a
mixed-version nested chunk retains its valid OffsetIndex; loading checks that
its first repetition level starts a row, and exhausting its levels checks the
indexed row count. Actual index fallback after unverified skips still returns
corruption. The new regression reproduced the reported failure before the fix
and now reads the expected value with and without the index;
continuation-at-page-start and inconsistent row-span cases are rejected. All
733 related ASAN tests pass. The legacy reader under be/src/format is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
