Hello Quanlong Huang, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24784
to look at the new patch set (#3).
Change subject: IMPALA-15324: Size exchange memory on input rows
......................................................................
IMPALA-15324: Size exchange memory on input rows
ExchangeNode sized both parts of its memory estimate, the row batch queues
and the deferred RPC payloads, from its own cardinality, which
computeStats() has already reduced by the offset. The senders never apply
that offset: DistributedPlanner clears it on the sender-side sort and
raises that sort's limit to limit + offset. So every row before the offset
still reaches the receiver's queues, and both parts were short by that many
rows.
Size them from the child's cardinality instead, which is what the senders
put on the wire. Not the filtered cardinality: this estimate bounds what
the receiver can buffer, and a runtime filter that arrives late or does
not select still leaves the full stream in the queues. When the child's
estimate is unknown the senders still stop after limit + offset rows, and
that is the bound to fall back to.
card-limit-offset.test now runs with VALIDATE_RESOURCES, which compares
the resource lines its queries already printed. Its three merging
exchanges move: 37.76KB to 54.75KB, 16.00KB to 27.56KB, and 16.00KB to
55.01KB for OFFSET without a LIMIT, where the sender-side sort keeps no
limit at all and the estimate has to cover the whole 7.30K-row input.
Two older gaps in the same estimate are left alone and tracked in
IMPALA-15326: the queue term divides by getNumNodes(), which on an
exchange is the sender count rather than the number of receivers, and a
limit that applies per instance is counted once for the whole fragment.
Testing:
- PlannerTest#testOffsetCardinality, and the same file with the fix
reverted, which fails on all three merging exchanges
- TpcdsPlannerTest, TpcdsCpuCostPlannerTest and TpcdsTupleCachePlannerTest:
their 322 expected files come out byte-identical to the ones master
produces, including the two ddl.test files that carry an offset
Change-Id: I4ab880549e63267f97b2c193cf76aa92bbfc581c
Assisted-by: Claude Opus 5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M
testdata/workloads/functional-planner/queries/PlannerTest/card-limit-offset.test
3 files changed, 73 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/24784/3
--
To view, visit http://gerrit.cloudera.org:8080/24784
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4ab880549e63267f97b2c193cf76aa92bbfc581c
Gerrit-Change-Number: 24784
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
