[jira] [Commented] (SPARK-18483) spark on yarn always connect to yarn resourcemanager at 0.0.0.0:8032
[ https://issues.apache.org/jira/browse/SPARK-18483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15676265#comment-15676265 ] Sean Owen commented on SPARK-18483: --- I don't think that path is valid, even on Windows. Try a Linux-style path. To be honest I'm not sure all of this works on Windows to begin with. > spark on yarn always connect to yarn resourcemanager at 0.0.0.0:8032 > -- > > Key: SPARK-18483 > URL: https://issues.apache.org/jira/browse/SPARK-18483 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.1 > Environment: java8 > SBT0.13 > scala2.11.8 > spark-2.0.1-bin-hadoop2.6 >Reporter: inred > > I have installed the yarn resource manager at 192.168.13.159:8032 and have > set YARN_CONF_DIR environment var and have the yarn-site.xml > configured as the following, but it always connects to 0.0.0.0:8032 instead > of > 192.168.13.159:8032 > set environment > E:\app>set yarn > YARN_CONF_DIR=D:\Documents\download\hadoop > E:\app>set had > HADOOP_HOME=D:\Documents\download\hadoop > E:\app>cat D:\Documents\download\hadoop\yarn-site.xml > > yarn.resourcemanager.address > 192.168.13.159:8032 > > "C:\Program Files\Java\jdk1.8.0_92\bin\java" -Didea.launcher.port=7532 > "-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA > Community Edition 2016.2.5\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program > Files\Java\jdk1.8.0_92\jre\lib\charsets.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\deploy.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\access-bridge-64.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\cldrdata.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\dnsns.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\jaccess.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\jfxrt.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\localedata.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\nashorn.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunec.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunjce_provider.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunmscapi.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunpkcs11.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\zipfs.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\javaws.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jce.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jfr.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jfxswt.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jsse.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\management-agent.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\plugin.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\resources.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\rt.jar;E:\app\adam-ano\target\scala-2.11\classes;C:\Users\feiwu\.ivy2\cache\org.scala-lang\scala-library\jars\scala-library-2.11.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\xz-1.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jta-1.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jpam-1.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\guice-3.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\ivy-2.4.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\lz4-1.3.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\oro-2.0.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\ST4-4.0.4.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\avro-1.7.7.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\core-1.1.2.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\gson-2.2.4.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\mail-1.4.7.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\mx4j-3.0.2.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\snappy-0.2.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\antlr-2.7.7.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\opencsv-2.3.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\py4j-0.10.3.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\xmlenc-0.52.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\base64-2.3.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\guava-14.0.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\janino-2.7.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jets3t-0.9.3.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jetty-6.1.26.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jline-2.12.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jsr305-1.3.9.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\log4j-1.2.17.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\minlog-1.3.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\
[jira] [Commented] (SPARK-18483) spark on yarn always connect to yarn resourcemanager at 0.0.0.0:8032
[ https://issues.apache.org/jira/browse/SPARK-18483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675388#comment-15675388 ] inred commented on SPARK-18483: --- it failed even when i set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop, i submit it from windows development node to remote Linux yarn cluster > spark on yarn always connect to yarn resourcemanager at 0.0.0.0:8032 > -- > > Key: SPARK-18483 > URL: https://issues.apache.org/jira/browse/SPARK-18483 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.1 > Environment: java8 > SBT0.13 > scala2.11.8 > spark-2.0.1-bin-hadoop2.6 >Reporter: inred > > I have installed the yarn resource manager at 192.168.13.159:8032 and have > set YARN_CONF_DIR environment var and have the yarn-site.xml > configured as the following, but it always connects to 0.0.0.0:8032 instead > of > 192.168.13.159:8032 > set environment > E:\app>set yarn > YARN_CONF_DIR=D:\Documents\download\hadoop > E:\app>set had > HADOOP_HOME=D:\Documents\download\hadoop > E:\app>cat D:\Documents\download\hadoop\yarn-site.xml > > yarn.resourcemanager.address > 192.168.13.159:8032 > > "C:\Program Files\Java\jdk1.8.0_92\bin\java" -Didea.launcher.port=7532 > "-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA > Community Edition 2016.2.5\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program > Files\Java\jdk1.8.0_92\jre\lib\charsets.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\deploy.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\access-bridge-64.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\cldrdata.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\dnsns.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\jaccess.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\jfxrt.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\localedata.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\nashorn.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunec.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunjce_provider.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunmscapi.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\sunpkcs11.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\ext\zipfs.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\javaws.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jce.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jfr.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jfxswt.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\jsse.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\management-agent.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\plugin.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\resources.jar;C:\Program > Files\Java\jdk1.8.0_92\jre\lib\rt.jar;E:\app\adam-ano\target\scala-2.11\classes;C:\Users\feiwu\.ivy2\cache\org.scala-lang\scala-library\jars\scala-library-2.11.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\xz-1.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jta-1.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jpam-1.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\guice-3.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\ivy-2.4.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\lz4-1.3.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\oro-2.0.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\ST4-4.0.4.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\avro-1.7.7.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\core-1.1.2.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\gson-2.2.4.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\mail-1.4.7.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\mx4j-3.0.2.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\snappy-0.2.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\antlr-2.7.7.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\opencsv-2.3.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\py4j-0.10.3.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\xmlenc-0.52.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\base64-2.3.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\guava-14.0.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\janino-2.7.8.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jets3t-0.9.3.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jetty-6.1.26.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jline-2.12.1.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\jsr305-1.3.9.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\log4j-1.2.17.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\minlog-1.3.0.jar;D:\Documents\download\spark-2.0.1-bin-hadoop2.6\jars\pyrolite-4.9.j
