Hi, So far I have been using Spark "embedded" in my app. Now, I'd like to run it on a dedicated server.
I am that far: - fresh ubuntu 16, server name is mocha / ip 10.0.100.120, installed scala 2.10, installed Spark 1.6.2, recompiled - Pi test works - UI on port 8080 works Log says: Spark Command: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp /opt/apache-spark-1.6.2/conf/:/opt/apache-spark-1.6.2/assembly/target/scala-2.10/spark-assembly-1.6.2-hadoop2.2.0.jar:/opt/apache-spark-1.6.2/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/opt/apache-spark-1.6.2/lib_managed/jars/datanucleus-core-3.2.10.jar:/opt/apache-spark-1.6.2/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip micha --port 7077 --webui-port 8080 ======================================== Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 16/07/10 13:03:55 INFO Master: Registered signal handlers for [TERM, HUP, INT] 16/07/10 13:03:55 WARN Utils: Your hostname, micha resolves to a loopback address: 127.0.1.1; using 10.0.100.120 instead (on interface eno1) 16/07/10 13:03:55 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 16/07/10 13:03:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/07/10 13:03:55 INFO SecurityManager: Changing view acls to: root 16/07/10 13:03:55 INFO SecurityManager: Changing modify acls to: root 16/07/10 13:03:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 16/07/10 13:03:56 INFO Utils: Successfully started service 'sparkMaster' on port 7077. 16/07/10 13:03:56 INFO Master: Starting Spark master at spark://micha:7077 16/07/10 13:03:56 INFO Master: Running Spark version 1.6.2 16/07/10 13:03:56 INFO Server: jetty-8.y.z-SNAPSHOT 16/07/10 13:03:56 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080 16/07/10 13:03:56 INFO Utils: Successfully started service 'MasterUI' on port 8080. 16/07/10 13:03:56 INFO MasterWebUI: Started MasterWebUI at http://10.0.100.120:8080 16/07/10 13:03:56 INFO Server: jetty-8.y.z-SNAPSHOT 16/07/10 13:03:56 INFO AbstractConnector: Started SelectChannelConnector@micha:6066 16/07/10 13:03:56 INFO Utils: Successfully started service on port 6066. 16/07/10 13:03:56 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066 16/07/10 13:03:56 INFO Master: I have been elected leader! New state: ALIVE In my app, i changed the config to: SparkConf conf = new SparkConf().setAppName("myapp").setMaster("spark://10.0.100.120:6066"); (also tried 7077) On the client: 16-07-10 13:22:58:300 INFO org.spark-project.jetty.server.AbstractConnector - Started SelectChannelConnector@0.0.0.0:4040 16-07-10 13:22:58:300 DEBUG org.spark-project.jetty.util.component.AbstractLifeCycle - STARTED SelectChannelConnector@0.0.0.0:4040 16-07-10 13:22:58:300 DEBUG org.spark-project.jetty.util.component.AbstractLifeCycle - STARTED org.spark-project.jetty.server.Server@3eb292cd 16-07-10 13:22:58:301 INFO org.apache.spark.util.Utils - Successfully started service 'SparkUI' on port 4040. 16-07-10 13:22:58:306 INFO org.apache.spark.ui.SparkUI - Started SparkUI at http://10.0.100.100:4040 16-07-10 13:22:58:621 INFO org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to master spark://10.0.100.120:6066... 16-07-10 13:22:58:648 DEBUG org.apache.spark.network.client.TransportClientFactory - Creating new connection to /10.0.100.120:6066 16-07-10 13:22:58:689 DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetectionLevel: simple 16-07-10 13:22:58:714 WARN org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect to master 10.0.100.120:6066 java.io.IOException: Failed to connect to /10.0.100.120:6066 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216) and if I try to telnet: $ telnet 10.0.100.120 6066 Trying 10.0.100.120... telnet: connect to address 10.0.100.120: Connection refused telnet: Unable to connect to remote host $ telnet 10.0.100.120 7077 Trying 10.0.100.120... telnet: connect to address 10.0.100.120: Connection refused telnet: Unable to connect to remote host On the server, I checked with netstat: jgp@micha:/opt/apache-spark$ netstat -a | grep 6066 tcp6 0 0 micha.nc.rr.com:6066 [::]:* LISTEN jgp@micha:/opt/apache-spark$ netstat -a | grep 7077 tcp6 0 0 micha.nc.rr.com:7077 [::]:* LISTEN If I interpret this, it looks like it listens in IP v6 and not 4... Any clue would be very helpful. I do not think I am that far, but... Thanks jg
