ulysses-you commented on PR #56957: URL: https://github.com/apache/spark/pull/56957#issuecomment-5087877708
thank you @dongjoon-hyun and @starcatmeow for your concerns: **1. Once-per-round lifecycle (default-path behavior).** The strategy is now created and `prepare`d exactly once per `resourceOffers` round; `resourceOfferSingleTaskSet` only calls a new `reset()` to restart the iteration cursor. `roundrobin` does its shuffle once in `prepare()`, so the default path is back to a single shuffle per round — no more per-TaskSet-×-locality re-shuffle/re-sort on the hot path. The trait scaladoc now matches the code (prepare-once, reset-per-call), with the call flow spelled out. **2. Live cpus + deterministic ordering.** Both `binpack` and `balance` now order off the live `availableCpus` array (recomputed in `reset()`) rather than the static `offer.cores` snapshot. `binpack` visits the executor with the **fewest live free cores first**, with `executorId` as a deterministic tie-breaker. That gives exactly the convergence you're after: executors already carrying work have fewer free cores, so they keep sorting to the front and packing targets the same executors across rounds, while the id tie-break keeps equal-core cases stable. `balance` uses a priority queue on the live counts (most free cores first), id-tie-broken. **3. Tests.** Added for both `binpack` and `balance`: barrier task sets (including the partial-launch path that reverts `availableCpus`), multiple TaskSets in a single round (asserting a later TaskSet observes the earlier one's assignments), and custom (GPU) resources. **About locality**: The strategy only decides the order in which offers are visited within a single locality level; the locality level itself is advanced by the outer loop and enforced independently by `TaskSetManager.resourceOffer`. The two are orthogonal — a strategy can never let a task violate its locality constraint; it only ranks a set of offers that are already equally legal at that level. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
