Avinash M. created SPARK-61369:
----------------------------------
Summary: TorchDistributor: CPU (gloo) training across nodes fails
when the hostname resolves to a loopback address
Key: SPARK-61369
URL: https://issues.apache.org/jira/browse/SPARK-61369
Project: Spark
Issue Type: Improvement
Components: ML, PySpark
Affects Versions: 4.2.1
Reporter: Avinash M.
h2. Description
*Problem*
{{TorchDistributor(local_mode=False, use_gpu=False)}} across two or more nodes
fails in {{torch.distributed.init_process_group("gloo")}} when each worker's
hostname resolves to a loopback address. That is the default on Debian and
Ubuntu, which put {{127.0.1.1 <hostname>}} in {{{}/etc/hosts{}}}:
RuntimeError: Gloo connectFullMesh failed with
[.../gloo/transport/tcp/pair.cc:152] timed out connecting:
SO_ERROR: Connection refused, remote=[127.0.1.1]:<port>
The {{torchrun}} rendezvous works, because {{TorchDistributor}} sets
{{MASTER_ADDR}} to the routable executor address. The failure is in the next
step, when gloo connects the ranks to each other.
*Cause*
Spark already handles this host setup for its own traffic. Each worker logs:
WARN Utils: Your hostname, caseA-w1, resolves to a loopback address: 127.0.1.1;
using 172.28.0.3 instead (on interface eth0)
Gloo does not get that correction. PyTorch's default gloo device takes its
address from the hostname. PyTorch accepts {{127.0.1.1}} because it can bind to
it, so it prints no warning. (It warns only when the hostname does not resolve
at all: "Unable to resolve hostname to a (local) address. Using the loopback
address as fallback. Manually set the network interface to bind to with
GLOO_SOCKET_IFNAME.") Each rank then tells the others to connect to
{{{}127.0.1.1{}}}, so a rank on another node connects to itself.
*Proposal*
{{TorchDistributor}} knows the routable executor address (it uses it for
{{{}MASTER_ADDR{}}}), and Spark already chose the right interface for it. In
the barrier task, before it starts {{{}torchrun{}}}:
# If {{GLOO_SOCKET_IFNAME}} is set, do nothing.
# If the hostname resolves to a loopback address, set {{GLOO_SOCKET_IFNAME}}
for the {{torchrun}} process to the interface that holds the executor's address.
# If no interface matches, log a warning that names {{{}GLOO_SOCKET_IFNAME{}}}.
A smaller first step is a clear warning (step 3) plus a note in the
{{TorchDistributor}} docs. NCCL is not affected in our tests: a 2-node GPU run
chose the right interface by itself.
*Reproduction (Apache Spark 4.2.1, standalone)*
Ubuntu 24.04 containers, OpenJDK 17, {{pyspark==4.2.1}} and CPU {{torch}}
2.15.0 from PyPI, Python 3.12.3. One master and 2 workers with 1 core each, so
the 2 ranks run on 2 workers.
```
from pyspark.ml.torch.distributor import TorchDistributor
def train():
import datetime, torch, torch.distributed as dist
dist.init_process_group("gloo", timeout=datetime.timedelta(seconds=60))
t = torch.tensor([float(dist.get_rank())])
dist.all_reduce(t)
dist.destroy_process_group()
return t.item()
print(TorchDistributor(num_processes=2, local_mode=False,
use_gpu=False).run(train))
```
||Case||Worker hostname resolves to||{{GLOO_SOCKET_IFNAME}}||Result||
|A|{{127.0.1.1}} (Ubuntu default)|not set|*Fails:* {{connectFullMesh …
remote=[127.0.1.1]}}|
|B|{{127.0.1.1}}|{{eth0}} (set in {{{}train{}}})|Passes: all-reduce sum 1.0|
|C|the container's own address|not set|Passes: all-reduce sum 1.0|
We first saw case A on a 4-node cluster with a vendor Spark 4.0 runtime and
PyTorch 2.7. The {{GLOO_SOCKET_IFNAME}} workaround fixed it there too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]