GitHub user jrt1899 added a comment to the discussion: Parallelism bottleneck in multi-UDF workflows
Thanks for the detailed ordering, this is very helpful. Going through each step: **1. Timer-based flushing.** JVM workers already do this: `WorkerTimerService` flushes output buffers every `adaptive-buffering-timeout-ms` (500 ms by default). Python workers have no equivalent, which is why the stall shows up specifically in Python UDF workflows. Rather than the stats-poll approach from Design 1, I think the cleaner fix is to give the Python worker the same timer, so both engines behave the same and the existing config applies. I can open an issue and PR for that next. **2. Static partitioning.** Agreed, and not covered yet. For row-independent UDFs with equal parallelism, one-to-one forwarding takes the 64 buffers down to 8 without any runtime logic. I'll run that as a baseline on the same workload, and look at a rescale-style fixed mapping for unequal parallelism. **3. Dynamic routing.** I went ahead and put up the least-loaded version (#8862 → #8863 → #8865, summarised in my comment above) so the design can be discussed on real code. To keep the risk low, it's opt-in (`enable-least-loaded-routing`, default off), it's a new partitioning rather than a change to round robin, and the first two PRs are inert. I'm happy for these to wait until 1 and 2 are done, and I'll read FLIP-339 and compare against it before asking for a full review. GitHub link: https://github.com/apache/texera/discussions/8519#discussioncomment-18762654 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
