Avinash M. created SPARK-61369:
----------------------------------

             Summary: TorchDistributor: CPU (gloo) training across nodes fails 
when the hostname resolves to a loopback address
                 Key: SPARK-61369
                 URL: https://issues.apache.org/jira/browse/SPARK-61369
             Project: Spark
          Issue Type: Improvement
          Components: ML, PySpark
    Affects Versions: 4.2.1
            Reporter: Avinash M.


h2. Description

*Problem*

{{TorchDistributor(local_mode=False, use_gpu=False)}} across two or more nodes 
fails in {{torch.distributed.init_process_group("gloo")}} when each worker's 
hostname resolves to a loopback address. That is the default on Debian and 
Ubuntu, which put {{127.0.1.1 <hostname>}} in {{{}/etc/hosts{}}}:
 
 
RuntimeError: Gloo connectFullMesh failed with 
[.../gloo/transport/tcp/pair.cc:152] timed out connecting:
SO_ERROR: Connection refused, remote=[127.0.1.1]:<port>
 
The {{torchrun}} rendezvous works, because {{TorchDistributor}} sets 
{{MASTER_ADDR}} to the routable executor address. The failure is in the next 
step, when gloo connects the ranks to each other.

*Cause*

Spark already handles this host setup for its own traffic. Each worker logs:
 
WARN Utils: Your hostname, caseA-w1, resolves to a loopback address: 127.0.1.1; 
using 172.28.0.3 instead (on interface eth0)
 
Gloo does not get that correction. PyTorch's default gloo device takes its 
address from the hostname. PyTorch accepts {{127.0.1.1}} because it can bind to 
it, so it prints no warning. (It warns only when the hostname does not resolve 
at all: "Unable to resolve hostname to a (local) address. Using the loopback 
address as fallback. Manually set the network interface to bind to with 
GLOO_SOCKET_IFNAME.") Each rank then tells the others to connect to 
{{{}127.0.1.1{}}}, so a rank on another node connects to itself.

*Proposal*

{{TorchDistributor}} knows the routable executor address (it uses it for 
{{{}MASTER_ADDR{}}}), and Spark already chose the right interface for it. In 
the barrier task, before it starts {{{}torchrun{}}}:
 # If {{GLOO_SOCKET_IFNAME}} is set, do nothing.
 # If the hostname resolves to a loopback address, set {{GLOO_SOCKET_IFNAME}} 
for the {{torchrun}} process to the interface that holds the executor's address.
 # If no interface matches, log a warning that names {{{}GLOO_SOCKET_IFNAME{}}}.

A smaller first step is a clear warning (step 3) plus a note in the 
{{TorchDistributor}} docs. NCCL is not affected in our tests: a 2-node GPU run 
chose the right interface by itself.

*Reproduction (Apache Spark 4.2.1, standalone)*

Ubuntu 24.04 containers, OpenJDK 17, {{pyspark==4.2.1}} and CPU {{torch}} 
2.15.0 from PyPI, Python 3.12.3. One master and 2 workers with 1 core each, so 
the 2 ranks run on 2 workers.
 
```
from pyspark.ml.torch.distributor import TorchDistributor
 
def train():
import datetime, torch, torch.distributed as dist
dist.init_process_group("gloo", timeout=datetime.timedelta(seconds=60))
t = torch.tensor([float(dist.get_rank())])
dist.all_reduce(t)
dist.destroy_process_group()
return t.item()
 
print(TorchDistributor(num_processes=2, local_mode=False, 
use_gpu=False).run(train))
```
 
||Case||Worker hostname resolves to||{{GLOO_SOCKET_IFNAME}}||Result||
|A|{{127.0.1.1}} (Ubuntu default)|not set|*Fails:* {{connectFullMesh … 
remote=[127.0.1.1]}}|
|B|{{127.0.1.1}}|{{eth0}} (set in {{{}train{}}})|Passes: all-reduce sum 1.0|
|C|the container's own address|not set|Passes: all-reduce sum 1.0|

We first saw case A on a 4-node cluster with a vendor Spark 4.0 runtime and 
PyTorch 2.7. The {{GLOO_SOCKET_IFNAME}} workaround fixed it there too.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to