jrt1899 opened a new pull request, #8863: URL: https://github.com/apache/texera/pull/8863
### What changes were proposed in this PR? The second of three PRs. Still inert: no link uses the least-loaded partitioner until PR 3, and the flag defaults to off. > **Stacked on #8862.** Until that merges, this PR also shows its commit. Only the last commit, [`55f451f`](https://github.com/jrt1899/texera/commit/55f451f52720f198eacf90b0e299e7d4cbe38401), is new here. - `LeastLoadedRoutingService`: on each statistics poll the coordinator ranks a link's receivers by backlog (received − processed) and gives each sender one, spreading senders across receivers rather than sending them all to the single least-loaded one. - `UpdateRoutingPreference` RPC (Scala and Python workers): tells each sender which receiver to fill next. It only moves an index on the existing partitioner. It is kept separate from `AddPartitioning`, which rebuilds the partitioner and would throw away a partly filled batch on every poll. - Config flags: `network-buffering.enable-least-loaded-routing` (default `false`) and `least-loaded-routing-verbose`. ```mermaid flowchart LR W["UDF2 workers<br/>received, processed"] -->|"stats poll (~500 ms)"| C["Coordinator<br/>rank by backlog"] C -->|"UpdateRoutingPreference(link, idx)"| S["UDF1 senders"] S -->|"next full batch"| W ``` ### Any related issues, documentation, discussions? Part of: #8858 Depends on: #8862 ### How was this PR tested? - `sbt "testOnly *LeastLoadedRoutingServiceSpec"`: the least-backlogged receiver goes to the first sender, senders with equal counts get different receivers, a receiver with no reading counts as idle, an empty link yields no assignments, and every sender is assigned when senders outnumber receivers. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Sonnet 5.5, Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
