jrt1899 opened a new issue, #8858:
URL: https://github.com/apache/texera/issues/8858
### Feature Summary
On links with no partitioning requirement, each sender keeps one buffer per
receiver and fills them round robin. A receiver gets nothing until some buffer
reaches the batch size, so the link can hold up to senders × receivers ×
batchSize tuples before the first batch ships. With 8 Python UDF workers per
stage and a batch size of 5,000, that is about 320,000 rows. JVM workers flush
on a timer (adaptive-buffering-timeout-ms), but Python workers don't, so in
Python UDF workflows the downstream stage sits idle for most of the run.
Discussion: #8519
### Proposed Solution or Design
An opt-in LeastLoadedPartitioning
(network-buffering.enable-least-loaded-routing, default off). Each sender fills
one batch at a time and sends it to the receiver with the smallest backlog
(received − processed tuples). That backlog comes from the existing statistics
poll, and the coordinator delivers the choice through a lightweight
UpdateRoutingPreference RPC. Round robin stays the default and
RoundRobinPartitioner is not modified.
Round robin (today) Least-loaded (proposed)
sender ─┬─[buf]─> r1 sender ─[buf]─> least-backlogged r
├─[buf]─> r2
└─[buf]─> rm
first ship: n × m × batchSize first ship: batchSize per sender
Planned as three PRs: (1) partitioner + backlog statistic, (2) coordinator
ranking + RPC, (3) opt-in switch + benchmark.
### Affected Area
Workflow Engine (Amber)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
