[
https://issues.apache.org/jira/browse/IMPALA-15280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15280:
--------------------------------------
Component/s: Frontend
> Descriptor table lists partitions of scans that are not in the final plan
> -------------------------------------------------------------------------
>
> Key: IMPALA-15280
> URL: https://issues.apache.org/jira/browse/IMPALA-15280
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> HdfsScanNode registers the partitions it will read with the descriptor table
> from computeScanRangeLocations(), which init() calls while the scan's subtree
> is being built:
>
> {code:java}
> analyzer.getDescTbl().addReferencedPartition(tbl_, partition.getId());
> {code}
>
> DescriptorTable.toThrift() then sends the backend exactly those partitions
> for every table except a table sink's target, which keeps all of them. So the
> set describes every scan the planner ever constructed rather than the scans
> that ended up in the plan: anything built and later replaced leaves its
> partitions behind, and the backend gets partition metadata that no scan
> reads. The TODO above the tuple descriptors in toThrift() already names the
> situation:
>
> {code:java}
> // TODO: Ideally, we should call tupleDesc.checkIsExecutable() here, but there
> // currently are several situations in which we send materialized tuples
> without
> // a mem layout to the BE, e.g., when unnesting unions or when replacing plan
> // trees with an EmptySetNode.
> {code}
>
> I did not find a query on master that reproduces this. The subtree
> replacements I looked at - createScanNode() when the conjuncts are implied
> false, the empty SPJ result set in createSelectPlan(), and union operands
> dropped for the same reason - all decide before the discarded side is
> planned, so nothing is registered and then thrown away. I did not try to
> establish that no other path exists.
>
> IMPALA-7996 does hit it. It replaces the pruned input of a constant-false
> outer join after that input has been built, and
> PlannerTestBase.testHdfsPartitionsReferenced, which asserts that every
> partition in the descriptor table is covered by a scan range, fails on the
> PlannerTest#testEmpty queries it adds.
>
> Proposal: have each HdfsScanNode remember the ids it registered, and once the
> plan is final - after Planner.createPlans() in Frontend.createExecRequest(),
> before the descriptor table is serialized - keep the union over the scans
> still reachable from the plan fragments.
>
> Keeping what the surviving scans registered, rather than deriving the
> partitions again, is what bounds the risk. HdfsScanNode is the only caller of
> addReferencedPartition(), so the resulting set is by construction a subset of
> today's, and it is the same set whenever every scan that was built survives -
> which, per the paragraph above, is every plan on master. No plan should
> change here; the change is a prerequisite rather than a fix with its own
> visible effect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
