[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15636802#comment-15636802 ]
György Süveges edited comment on SPARK-4563 at 11/4/16 4:13 PM: ---------------------------------------------------------------- +1 However, it's related to SPARK_LOCAL_IP and Spark UI, I think it's similar. We use AWS EMR5 (Spark 2.0 over YARN) While developing we use spark from intellij in yarn-client mode. To connect to AWS, OpenVPN is applied with biderectional routing so the executors in amazon can access the driver on the local workstation. With OpenVPN there are multiple network interfaces then, so SPARK_LOCAL_IP should be set. But it's really inconvenient to set SPARK_LOCAL_IP each time we connect to OpenVPN and get different IP. So we decided to set it automatically from code. Since SPARK_LOCAL_IP is env var and env vars are inmutable in java, we change the spark.driver.host corresponding java property. But as noticed spark.driver.host is only second order citizen in spark, for some features only the SPARK_LOCAL_IP is taken into account. For example Spark UI listens on the host set in SPARK_LOCAL_IP, but ignores the spark.driver.host totally. I hope this improvement will fix this issue, as I saw in https://github.com/apache/spark/commit/2cd1bfa4f0c6625b0ab1dbeba2b9586b9a6a9f42 However in [WebUI|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/WebUI.scala#L139] on master I can still see places where only SPARK_LOCAL_IP is checked only I think SPARK_LOCAL_IP and spark.driver.host should be equivalent counterparts, shouldn't they? was (Author: gyorgy_suve...@epam.com): +1 However, it's related to SPARK_LOCAL_IP and Spark UI, I think it's similar. We use AWS EMR5 (Spark 2.0 over YARN) While developing we use spark from intellij in yarn-client mode. To connect to AWS, OpenVPN is applied with biderectional routing so the executors in amazon can access the driver on the local workstation. With OpenVPN there are multiple network interfaces then, so SPARK_LOCAL_IP should be set. But it's really inconvenient to set SPARK_LOCAL_IP each time we connect to OpenVPN and get different IP. So we decided to set it automatically from code. Since SPARK_LOCAL_IP is env var and env vars are inmutable in java, we change the spark.driver.host corresponding java property. But as noticed spark.driver.host is only second order citizen in spark, for some features only the SPARK_LOCAL_IP is taken into account. For example Spark UI listens on the host set in SPARK_LOCAL_IP, but ignores the spark.driver.host totally. I hope this improvement will fix this issue, as I saw in https://github.com/apache/spark/commit/2cd1bfa4f0c6625b0ab1dbeba2b9586b9a6a9f42 I think SPARK_LOCAL_IP and spark.driver.host should be equivalent counterparts, shouldn't they? > Allow spark driver to bind to different ip then advertise ip > ------------------------------------------------------------ > > Key: SPARK-4563 > URL: https://issues.apache.org/jira/browse/SPARK-4563 > Project: Spark > Issue Type: Improvement > Components: Deploy > Reporter: Long Nguyen > Assignee: Marcelo Vanzin > Priority: Minor > Fix For: 2.1.0 > > > Spark driver bind ip and advertise is not configurable. spark.driver.host is > only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option > to set advertised ip/hostname -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org