[
https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15147434#comment-15147434
]
Paulo Villegas commented on SPARK-4563:
---------------------------------------
Hi. I would have a use case for this functionality: when the driver is running
behind a NAT barrier. In that case the driver can send messages with a cluster
master & executors, but they cannot reach the driver, hence it fails (it
launches executors, but since they cannot register back with the driver, the
driver decides they are not working and terminates them).
Given that we can do NAT, we could fix the ports the driver is listening at
(spark.driver.port, spark.fileserver.port, etc) and forward them through NAT,
and the outside world could reach the driver easily. Except that it actually
can't, because the driver advertises itself with the IP address of the internal
network it binds to, not with the outside reachable IP. And both aspects (bind
address & broadcast address) cannot be disentangled, currently.
Two real setups I've faced in which this would have been useful are:
* machine in which the driver is running is in a home/private network, and the
Spark processing cluster is reachable through the router, but not back due to
the private addresses
* driver is running in a virtual machine having a private address and trying
to connect to a cluster in the host network. The VM cannot be put in bridge
mode (which would solve the private address problem) due to corporate
restrictions that preclude any non-authorized computers obtain a valid IP
address in the host network.
I don't know if this is at all possible. I tried to look at the code, but it
seems to be hardcoded in the akka stack. SPARK-11638 SPARK-4389 seem to be
related (or the same). The latter suggests that it is indeed not possible.
> Allow spark driver to bind to different ip then advertise ip
> ------------------------------------------------------------
>
> Key: SPARK-4563
> URL: https://issues.apache.org/jira/browse/SPARK-4563
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Reporter: Long Nguyen
> Priority: Minor
>
> Spark driver bind ip and advertise is not configurable. spark.driver.host is
> only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option
> to set advertised ip/hostname
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
