Aleksandr Efimov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24723
Change subject: IMPALA-15280: Take referenced partitions from the final plan ...................................................................... IMPALA-15280: Take referenced partitions from the final plan HdfsScanNode registers its partitions with the descriptor table from computeScanRangeLocations(), which runs while the scan's subtree is being built. The descriptor table therefore describes every scan the planner ever constructed, not the scans that ended up in the plan. A subtree that is replaced afterwards leaves its partitions behind, and toThrift() ships partition descriptors that nothing reads. The TODO above the tuple descriptors in DescriptorTable.toThrift() already names this situation. Have each HdfsScanNode remember the ids it registered, and once the plan is final, keep the union over the scans still reachable from the plan fragments. Taking what the surviving scans registered, rather than deriving the partitions again, is what makes this safe: HdfsScanNode is the only caller of addReferencedPartition(), so the new set is by construction a subset of the old one, it is the same set whenever every scan that was built survives, and no scan can lose a partition it still reads. The walk covers every fragment, not the getFragmentsInPlanPreorder() subset that createPlanExecInfo() uses: that one skips fragments reached through a join build, and a scan below one of those still runs. The EmptySetNode replacements on master - createScanNode() on conjuncts implied false, the empty SPJ result set, dropped union operands - all happen before the discarded side is planned, so no plan is expected to change here. This is a prerequisite for IMPALA-7996, which replaces the pruned input of a constant-false outer join after that input has been built and trips PlannerTestBase.testHdfsPartitionsReferenced without it. Testing: - PlannerTest, which asserts on every query that each partition in the descriptor table is covered by a scan range. Change-Id: I553281e7c77e3297585360f37160bc3f9d69a208 Assisted-by: claude-opus-5 (Claude Code) --- M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java 4 files changed, 74 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/24723/1 -- To view, visit http://gerrit.cloudera.org:8080/24723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I553281e7c77e3297585360f37160bc3f9d69a208 Gerrit-Change-Number: 24723 Gerrit-PatchSet: 1 Gerrit-Owner: Aleksandr Efimov <[email protected]>
