Gabriel39 opened a new pull request, #65548:
URL: https://github.com/apache/doris/pull/65548
### What problem does this PR solve?
Issue Number: None
Related PR: None
This PR fixes two correctness areas in external file scans.
#### 1. Preserve external `COUNT(*)` / `COUNT(column)` semantics
External `COUNT(*)` retained an arbitrary scan slot after column pruning. BE
then inferred aggregate semantics from that post-pruning scan shape and could
treat the retained nullable placeholder as the argument of `COUNT(column)`.
This produced wrong results, including 9015 instead of 10000 rows for special
ORC data and 116 instead of 219 rows for Hive basic types.
The fix transports semantic COUNT arguments from Nereids through
`TPlanNode.push_down_count_slot_ids` to both file-scanner implementations and
their table/hybrid readers. The thrift field has an explicit compatibility
contract:
- absent: an old FE plan with unknown COUNT semantics; use the conservative
normal-scan path;
- present and empty: explicit `COUNT(*)` / row-count semantics;
- present and non-empty: explicit `COUNT(column)` arguments, which may use
metadata only when every mapping is proven safe.
Nereids no longer applies storage COUNT pushdown to cast arguments whose
null/error behavior would be lost. FE split planning also restricts
Iceberg/Paimon metadata-count split reduction and Hive/TVF no-split shortcuts
to explicit row-count semantics, so a `COUNT(column)` fallback retains all data
splits and normal scan parallelism. BE V1, V2, Hudi, Paimon, and table-reader
paths preserve the same argument state, reject unsafe metadata shortcuts, and
keep adaptive batching enabled for real-row fallbacks.
#### 2. Harden FileScannerV2 schema and reader edge cases
FileScannerV2 treated valid empty splits as failures, could count stopped
EOF or malformed Native input as an empty file, localized evolved nested
predicates without fully preserving type/nullability contracts, and rejected
whole Parquet schemas when only unprojected leaves used unsupported logical
types. Parquet page-cache range metadata also lost warm non-exact reuse while
lacking a bound.
The fix distinguishes stopped reads, valid empty input, and
malformed/truncated input; keeps unsafe nested predicates above `TableReader`;
validates only semantically required Parquet projections while preserving
strict checks for real predicate and `COUNT(column)` inputs; uses safe
row-position/default carriers for placeholder-only paths; and shares bounded
per-file page-range indexes without adding a process-wide lock to the `ReadAt`
hot path.
### Release note
Fix wrong external `COUNT(*)` / `COUNT(column)` results and unsafe metadata
pushdown across Nereids, thrift, FE split planning, and BE file readers. Also
harden FileScannerV2 handling of empty or interrupted input, evolved nested
predicates, unsupported unprojected Parquet logical types, COUNT placeholders,
hybrid readers, and bounded cross-reader page-cache range reuse.
### Check List (For Author)
- Test: Regression test / Unit Test
- BE ASAN unit coverage for V1/V2 COUNT semantics, table-level metadata
gates, schema-evolution fallbacks, COUNT placeholders, adaptive batching, and
Hudi/Paimon/Iceberg hybrid paths
- FE unit coverage for Nereids COUNT argument capture and
Hive/Iceberg/Paimon/TVF split-planning behavior
- `ColumnMapper*.*` BE ASAN suite (111 tests passed)
- Existing external regression cases `test_special_orc_formats` and
`test_hive_basic_type` reproduce the fixed COUNT(*) wrong results
- `test_file_scanner_v2_review_fixes` regression suite covers empty
input, evolved nested types, unsupported Parquet placeholders, and COUNT
fallback behavior
- Behavior changed: Yes (COUNT semantics are transported explicitly; unsafe
or ambiguous metadata COUNT plans fall back to normal scanning; valid empty
splits are skipped; stopped or malformed input is not counted as empty;
unsupported unprojected Parquet leaves no longer fail unrelated scans)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
