englefly opened a new pull request, #68314: URL: https://github.com/apache/doris/pull/68314
### What problem does this PR solve? Cherry-pick of #65805 (`[fix](nereids) Disambiguate NULL/OFFSET metadata from same-named nested fields`) to `branch-4.2`. The port also carries the two prerequisite commits that #65805 is written on top of, because branch-4.2 stopped at the state just before them: | commit | PR | why it is needed | | --- | --- | --- | | `branch-4.2 [fix](NestedColumnPruning) Rewrite the NestedColumnPruning meta-path handling ...` | #64535 | branch-4.2's FE pruning code was exactly master-before-#64535; #65805's FE hunks are written against the rewritten meta-path handling. | | `branch-4.2 [feat](olap) Support lazy reading mode for pruned complex columns` | #59263 | on master #59263 replaced #64535's `MetaPathStriper` approach again (predicate metadata paths are kept, `normalizePredicateMetaPathForAllAccessPath` / `addPredicatePathsToFinalAllAccessPaths` / `expandMapStarPaths`), and that is the code #65805 patches. | | `branch-4.2 [fix](nereids) Disambiguate NULL/OFFSET metadata from same-named nested fields` | #65805 | the requested pick. | Master order for reference: `#61888 → #62205 → #62315 → #62631 → #62304 → #63229 → #63736 → #64486 → #64535 → #59263 → … → #65805`; branch-4.2 already had everything up to #64486 (ported by #68214). ### What #65805 fixes * **Versioned, type-selected access paths.** `TColumnAccessPath` gets an optional `version` (`TCOLUMN_ACCESS_PATH_VERSION_LEGACY = 0`, `TCOLUMN_ACCESS_PATH_VERSION_TYPED = 1`) that is carried through thrift, protobuf and both FE/BE descriptor conversions. In the typed format the path type is authoritative: `DATA` selects `data_access_path`, `META` selects `meta_access_path`, and `NULL`/`OFFSET` are only ever emitted as typed `META` paths. New BEs still decode the legacy all-`DATA` encoding, so the supported rolling upgrade is: upgrade all BEs first, then let FEs send typed paths. * **FE.** Marks `NULL`/`OFFSET` collector contexts as `META`; checks the physical column's nullability instead of slot nullability, so an outer-join-nullable `NOT NULL` column no longer gets a `[col, NULL]` path; falls back to plain data reads for variant sub-columns; keeps the exact meta path when a sibling data path is also read; and keys pruning on the path type instead of string suffixes. * **BE.** Validates path versions and type-selected payloads; partitions current data / current metadata / descendant routing with explicit per-container ownership; derives `NULL_MAP_ONLY` / `OFFSET_ONLY` only when no current or predicate data path requires payload; keeps sibling `META` paths explicit; routes Map children by the logical `KEYS`/`VALUES` selectors instead of physical child column names; and rejects unrecognized Map selectors instead of silently pruning everything. ### branch-4.2 adaptations * branch-4.2 keeps the FE access-path model as thrift types instead of master's `ColumnAccessPath` / `DescriptorToThriftConverter`, so the typed version is stamped in `NestedColumnPruning.buildColumnAccessPaths` (the branch's single producer) and preserved by `SlotTypeReplacer.replaceIcebergAccessPathToId`. * The master-only `DescriptorToThriftConverter` / `DescriptorToThriftConverterTest` files do not exist on this branch and were not added; the equivalent coverage lives in `PruneNestedColumnTest`, whose `path()` / `metaPath()` helpers (and `VariantPruningLogicTest`'s) now carry the typed version. * Ported FE code was mapped onto this branch's names (`TAccessPathType`, `TColumnAccessPath`, `isFunctionNullCheckPath`) and keeps the segment-wise access-path comparator introduced by the branch's nested-column-pruning port. * `gensrc/thrift/PaloInternalService.thrift`: master uses field id `226` for `enable_prune_nested_column`, but branch-4.2 already uses `226`-`228`, so the option was moved to the next free id (`229`). * `segment_iterator.cpp` keeps branch-4.2's `_non_predicate_column_ids` naming and drops the master-only `_update_lsn_col_if_needed` / `_update_tso_col_if_needed` calls from the ported hunk; branch-4.2 keeps its own `common/compile_check_begin.h` include in front of the new `ScopedColumnIteratorReadPhase` helper. `_read_lazy_pruned_columns` uses the branch's `_schema_block_id_map` because master's `Schema::column_index()` does not exist here, and `FileColumnIterator::get_reader()` was added as in master. * Regression suites `lambda_null_pruning` and `map_contains_arg_pruning`, and the unit tests for lambda access paths / Map lazy-read-by-rowids, only exist on master (they come from PRs outside this chain) and were not added. ### Test * FE: `./build.sh --fe` OK. `./run-fe-ut.sh --run 'org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest'` → 63 passed; `VariantPruningLogicTest` 14, `IcebergScanNodeTest` 95, `MaterializeProbeVisitorTest` 8, `StringEmptyToLengthRuleTest`, `PullUpProjectExprUnderTopNTest` — all green. * BE: `./build.sh --be` OK (the two ported BE test sources were additionally checked with the unit-test flags, `-DBE_TEST -fno-access-control`). * Regression: `./run-regression-test.sh --run -d nereids_rules_p0/column_pruning` → 8 suites, 0 failed, including the new `left_join_not_null_column` suite that guards the outer-join null-map crash. ### Behavior changed Yes. Typed `META` access paths with an explicit version replace the legacy all-`DATA` encoding between new FEs and BEs (upgrade BEs before FEs); struct fields literally named `NULL`/`OFFSET` are now read correctly instead of being pruned as metadata; `IS NULL` on an outer-join-nullable `NOT NULL` column no longer crashes the BE; invalid Map descendant selectors now return an internal error instead of being silently ignored. The port also brings #59263, which defers reading non-predicate pruned complex columns until after predicate evaluation, reducing nested-column I/O. ### Release note None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
