andygrove commented on PR #2670: URL: https://github.com/apache/datafusion-ballista/pull/2670#issuecomment-6453441266
Addressed in 0d53dfcaf. Fixed: unclaimed queues are closed on cleanup (the writer keeps the outputs alive, as you said), a second open of a claimed queue fails at once, only a failure before the server answers is retried, and `is_transport_failure` uses the source chain alone (a reset shows as `Cancelled` and a refused connection as `Unavailable`, both with the transport error as source). The simplifications are in too: `TaskKey`, one `Outputs` struct, `with_retry` and a shared `do_get` helper, the hash partitioning encoder, single failure store, metric names, dev-dependency, pool fast path, test helpers and listeners kept open. Filed as follow-ups rather than grow this PR: memory accounting #2672, shared slot bounding #2673, writer send path and the `RepartitionExec` question #2674, location size #2675. Task loss as a non-goal is already in #2652's comparison table. `with_partitions` matches how `task_builder.rs` restricts `ShuffleReaderExec` today and gets its caller in the scheduler PR. The two overlapping test pairs cover different layers, so I kept both. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
