Yicong-Huang opened a new pull request, #59451: URL: https://github.com/apache/spark/pull/59451
### What changes were proposed in this pull request? In distributed mode, `TorchDistributor` picks `MASTER_PORT` in partition 0 by binding a socket to port 0 and releasing it right away. This PR adds `TorchDistributor._reserve_free_port`, which binds the socket with `SO_REUSEADDR` (without listening), and keeps it open until training finishes. The socket is closed in the existing `finally` block of the barrier task. While the socket is held, no other socket can bind the port, and the kernel does not hand it out for outgoing connections. torchrun's `TCPStore` also sets `SO_REUSEADDR`, so it can still listen on the port. ### Why are the changes needed? `pyspark.ml.tests.connect.test_connect_pipeline` `PipelineTestsOnConnect.test_pipeline` is flaky in CI: ``` torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 39055, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use ``` The port comes from the ephemeral range, and between releasing it and torchrun binding it, another socket (e.g. an outgoing connection on the shared CI runner) can take it. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added `test_reserve_free_port`, which checks that a competing bind fails while the port is reserved and that a `TCPStore` can still listen on it. Ran `pyspark.ml.torch.tests.test_distributor`, `pyspark.ml.tests.connect.test_parity_torch_distributor`, `pyspark.ml.tests.connect.test_legacy_mode_pipeline` and `pyspark.ml.tests.connect.test_connect_pipeline` locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
