[jira] [Commented] (SPARK-6388) Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40
[ https://issues.apache.org/jira/browse/SPARK-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366982#comment-14366982 ] Sean Owen commented on SPARK-6388: -- I am just using JDK8 to compile not targeting Java 8 in javac, and running on Java 8. I don't think there is a Java 8 problem in how Spark uses Hadoop or how Spark runs. You can see Spark + Hadoop even has Java 8-specific tests you can enable if you want. Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40 -- Key: SPARK-6388 URL: https://issues.apache.org/jira/browse/SPARK-6388 Project: Spark Issue Type: Bug Components: Block Manager, Spark Submit, YARN Affects Versions: 1.3.0 Environment: 1. Linux version 3.16.0-30-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #40-Ubuntu SMP Mon Jan 12 22:06:37 UTC 2015 2. Oracle Java 8 update 40 for Linux X64 3. Scala 2.10.5 4. Hadoop 2.6 (pre-build version) Reporter: John Original Estimate: 24h Remaining Estimate: 24h I build Apache Spark 1.3 munally. --- JAVA_HOME=PATH_TO_JAVA8 mvn clean package -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests --- Something goes wrong, akka always tell me --- 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. --- I build another version of Spark 1.3 + Hadoop 2.6 under Java 7. Everything goes well. Logs --- 15/03/17 21:27:06 INFO spark.SparkContext: Running Spark version 1.3.0 15/03/17 21:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/17 21:27:08 INFO spark.SecurityManager: Changing view Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: Changing modify Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser) 15/03/17 21:27:08 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/03/17 21:27:08 INFO Remoting: Starting remoting 15/03/17 21:27:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@Server3:37951] 15/03/17 21:27:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 37951. 15/03/17 21:27:09 INFO spark.SparkEnv: Registering MapOutputTracker 15/03/17 21:27:09 INFO spark.SparkEnv: Registering BlockManagerMaster 15/03/17 21:27:09 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-0db692bb-cd02-40c8-a8f0-3813c6da18e2/blockmgr-a1d0ad23-ab76-4177-80a0-a6f982a64d80 15/03/17 21:27:09 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB 15/03/17 21:27:09 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-502ef3f8-b8cd-45cf-b1df-97df297cdb35/httpd-6303e24d-4b2b-4614-bb1d-74e8d331189b 15/03/17 21:27:09 INFO spark.HttpServer: Starting HTTP Server 15/03/17 21:27:09 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48000 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'HTTP file server' on port 48000. 15/03/17 21:27:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/03/17 21:27:10 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/03/17 21:27:10 INFO ui.SparkUI: Started SparkUI at http://Server3:4040 15/03/17 21:27:10 INFO spark.SparkContext: Added JAR file:/home/hduser/spark-java2.jar at http://192.168.11.42:48000/jars/spark-java2.jar with timestamp 1426598830307 15/03/17 21:27:10 INFO client.RMProxy: Connecting to ResourceManager at Server3/192.168.11.42:8050 15/03/17 21:27:11 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 15/03/17 21:27:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 15/03/17 21:27:11 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/03/17 21:27:11 INFO yarn.Client: Setting up container launch context for our AM 15/03/17 21:27:11 INFO yarn.Client: Preparing resources for our AM container 15/03/17 21:27:12 INFO yarn.Client: Uploading resource file:/home/hduser/spark-1.3.0/assembly/target/scala-2.10/spark-assembly-1.3.0-hadoop2.6.0.jar -
[jira] [Commented] (SPARK-6388) Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40
[ https://issues.apache.org/jira/browse/SPARK-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365758#comment-14365758 ] Michael Malak commented on SPARK-6388: -- Isn't it Hadoop 2.7 that is supposed to provide Java 8 compatibility? Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40 -- Key: SPARK-6388 URL: https://issues.apache.org/jira/browse/SPARK-6388 Project: Spark Issue Type: Bug Components: Block Manager, Spark Submit, YARN Affects Versions: 1.3.0 Environment: 1. Linux version 3.16.0-30-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #40-Ubuntu SMP Mon Jan 12 22:06:37 UTC 2015 2. Oracle Java 8 update 40 for Linux X64 3. Scala 2.10.5 4. Hadoop 2.6 (pre-build version) Reporter: John Original Estimate: 24h Remaining Estimate: 24h I build Apache Spark 1.3 munally. --- JAVA_HOME=PATH_TO_JAVA8 mvn clean package -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests --- Something goes wrong, akka always tell me --- 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. --- I build another version of Spark 1.3 + Hadoop 2.6 under Java 7. Everything goes well. Logs --- 15/03/17 21:27:06 INFO spark.SparkContext: Running Spark version 1.3.0 15/03/17 21:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/17 21:27:08 INFO spark.SecurityManager: Changing view Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: Changing modify Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser) 15/03/17 21:27:08 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/03/17 21:27:08 INFO Remoting: Starting remoting 15/03/17 21:27:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@Server3:37951] 15/03/17 21:27:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 37951. 15/03/17 21:27:09 INFO spark.SparkEnv: Registering MapOutputTracker 15/03/17 21:27:09 INFO spark.SparkEnv: Registering BlockManagerMaster 15/03/17 21:27:09 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-0db692bb-cd02-40c8-a8f0-3813c6da18e2/blockmgr-a1d0ad23-ab76-4177-80a0-a6f982a64d80 15/03/17 21:27:09 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB 15/03/17 21:27:09 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-502ef3f8-b8cd-45cf-b1df-97df297cdb35/httpd-6303e24d-4b2b-4614-bb1d-74e8d331189b 15/03/17 21:27:09 INFO spark.HttpServer: Starting HTTP Server 15/03/17 21:27:09 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48000 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'HTTP file server' on port 48000. 15/03/17 21:27:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/03/17 21:27:10 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/03/17 21:27:10 INFO ui.SparkUI: Started SparkUI at http://Server3:4040 15/03/17 21:27:10 INFO spark.SparkContext: Added JAR file:/home/hduser/spark-java2.jar at http://192.168.11.42:48000/jars/spark-java2.jar with timestamp 1426598830307 15/03/17 21:27:10 INFO client.RMProxy: Connecting to ResourceManager at Server3/192.168.11.42:8050 15/03/17 21:27:11 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 15/03/17 21:27:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 15/03/17 21:27:11 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/03/17 21:27:11 INFO yarn.Client: Setting up container launch context for our AM 15/03/17 21:27:11 INFO yarn.Client: Preparing resources for our AM container 15/03/17 21:27:12 INFO yarn.Client: Uploading resource file:/home/hduser/spark-1.3.0/assembly/target/scala-2.10/spark-assembly-1.3.0-hadoop2.6.0.jar - hdfs://Server3:9000/user/hduser/.sparkStaging/application_1426595477608_0002/spark-assembly-1.3.0-hadoop2.6.0.jar 15/03/17 21:27:21 INFO yarn.Client: Setting up the launch environment for our
[jira] [Commented] (SPARK-6388) Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40
[ https://issues.apache.org/jira/browse/SPARK-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365142#comment-14365142 ] Sean Owen commented on SPARK-6388: -- Hm, it's not clear what you mean here. You are just showing that you are building Spark. That can't fail with an Akka error. Can you report the output of the build and of tests? What is this log output you are attaching -- what are you running? I am running Java 8_40 on Mac OS X and have no problems. Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40 -- Key: SPARK-6388 URL: https://issues.apache.org/jira/browse/SPARK-6388 Project: Spark Issue Type: Bug Components: Block Manager, Spark Submit, YARN Affects Versions: 1.3.0 Environment: 1. Linux version 3.16.0-30-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #40-Ubuntu SMP Mon Jan 12 22:06:37 UTC 2015 2. Oracle Java 8 update 40 for Linux X64 3. Scala 2.10.5 4. Hadoop 2.6 (pre-build version) Reporter: John Original Estimate: 24h Remaining Estimate: 24h I build Apache Spark 1.3 munally. --- JAVA_HOME=PATH_TO_JAVA8 mvn clean package -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests --- Something goes wrong, akka always tell me --- 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. --- I build another version of Spark 1.3 + Hadoop 2.6 under Java 7. Everything goes well. Logs --- 15/03/17 21:27:06 INFO spark.SparkContext: Running Spark version 1.3.0 15/03/17 21:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/17 21:27:08 INFO spark.SecurityManager: Changing view Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: Changing modify Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser) 15/03/17 21:27:08 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/03/17 21:27:08 INFO Remoting: Starting remoting 15/03/17 21:27:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@Server3:37951] 15/03/17 21:27:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 37951. 15/03/17 21:27:09 INFO spark.SparkEnv: Registering MapOutputTracker 15/03/17 21:27:09 INFO spark.SparkEnv: Registering BlockManagerMaster 15/03/17 21:27:09 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-0db692bb-cd02-40c8-a8f0-3813c6da18e2/blockmgr-a1d0ad23-ab76-4177-80a0-a6f982a64d80 15/03/17 21:27:09 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB 15/03/17 21:27:09 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-502ef3f8-b8cd-45cf-b1df-97df297cdb35/httpd-6303e24d-4b2b-4614-bb1d-74e8d331189b 15/03/17 21:27:09 INFO spark.HttpServer: Starting HTTP Server 15/03/17 21:27:09 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48000 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'HTTP file server' on port 48000. 15/03/17 21:27:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/03/17 21:27:10 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/03/17 21:27:10 INFO ui.SparkUI: Started SparkUI at http://Server3:4040 15/03/17 21:27:10 INFO spark.SparkContext: Added JAR file:/home/hduser/spark-java2.jar at http://192.168.11.42:48000/jars/spark-java2.jar with timestamp 1426598830307 15/03/17 21:27:10 INFO client.RMProxy: Connecting to ResourceManager at Server3/192.168.11.42:8050 15/03/17 21:27:11 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 15/03/17 21:27:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 15/03/17 21:27:11 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/03/17 21:27:11 INFO yarn.Client: Setting up container launch context for our AM 15/03/17 21:27:11 INFO yarn.Client: Preparing resources for our AM container 15/03/17 21:27:12 INFO yarn.Client: Uploading resource
[jira] [Commented] (SPARK-6388) Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40
[ https://issues.apache.org/jira/browse/SPARK-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365148#comment-14365148 ] John commented on SPARK-6388: - Ok, it looks like my problem. I'll try it later. Sorry for opening a issue. Let me close it. Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40 -- Key: SPARK-6388 URL: https://issues.apache.org/jira/browse/SPARK-6388 Project: Spark Issue Type: Bug Components: Block Manager, Spark Submit, YARN Affects Versions: 1.3.0 Environment: 1. Linux version 3.16.0-30-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #40-Ubuntu SMP Mon Jan 12 22:06:37 UTC 2015 2. Oracle Java 8 update 40 for Linux X64 3. Scala 2.10.5 4. Hadoop 2.6 (pre-build version) Reporter: John Original Estimate: 24h Remaining Estimate: 24h I build Apache Spark 1.3 munally. --- JAVA_HOME=PATH_TO_JAVA8 mvn clean package -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests --- Something goes wrong, akka always tell me --- 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. --- I build another version of Spark 1.3 + Hadoop 2.6 under Java 7. Everything goes well. Logs --- 15/03/17 21:27:06 INFO spark.SparkContext: Running Spark version 1.3.0 15/03/17 21:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/17 21:27:08 INFO spark.SecurityManager: Changing view Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: Changing modify Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser) 15/03/17 21:27:08 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/03/17 21:27:08 INFO Remoting: Starting remoting 15/03/17 21:27:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@Server3:37951] 15/03/17 21:27:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 37951. 15/03/17 21:27:09 INFO spark.SparkEnv: Registering MapOutputTracker 15/03/17 21:27:09 INFO spark.SparkEnv: Registering BlockManagerMaster 15/03/17 21:27:09 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-0db692bb-cd02-40c8-a8f0-3813c6da18e2/blockmgr-a1d0ad23-ab76-4177-80a0-a6f982a64d80 15/03/17 21:27:09 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB 15/03/17 21:27:09 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-502ef3f8-b8cd-45cf-b1df-97df297cdb35/httpd-6303e24d-4b2b-4614-bb1d-74e8d331189b 15/03/17 21:27:09 INFO spark.HttpServer: Starting HTTP Server 15/03/17 21:27:09 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48000 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'HTTP file server' on port 48000. 15/03/17 21:27:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/03/17 21:27:10 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/03/17 21:27:10 INFO ui.SparkUI: Started SparkUI at http://Server3:4040 15/03/17 21:27:10 INFO spark.SparkContext: Added JAR file:/home/hduser/spark-java2.jar at http://192.168.11.42:48000/jars/spark-java2.jar with timestamp 1426598830307 15/03/17 21:27:10 INFO client.RMProxy: Connecting to ResourceManager at Server3/192.168.11.42:8050 15/03/17 21:27:11 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 15/03/17 21:27:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 15/03/17 21:27:11 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/03/17 21:27:11 INFO yarn.Client: Setting up container launch context for our AM 15/03/17 21:27:11 INFO yarn.Client: Preparing resources for our AM container 15/03/17 21:27:12 INFO yarn.Client: Uploading resource file:/home/hduser/spark-1.3.0/assembly/target/scala-2.10/spark-assembly-1.3.0-hadoop2.6.0.jar - hdfs://Server3:9000/user/hduser/.sparkStaging/application_1426595477608_0002/spark-assembly-1.3.0-hadoop2.6.0.jar 15/03/17 21:27:21 INFO yarn.Client: Setting up the launch environment for
[jira] [Commented] (SPARK-6388) Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40
[ https://issues.apache.org/jira/browse/SPARK-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365152#comment-14365152 ] Sean Owen commented on SPARK-6388: -- OK. Although I don't see a problem building and running tests with 8u40, that doesn't mean there's no problem at all. I was mostly asking for more info about what you are reporting and how you are running things. Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40 -- Key: SPARK-6388 URL: https://issues.apache.org/jira/browse/SPARK-6388 Project: Spark Issue Type: Bug Components: Block Manager, Spark Submit, YARN Affects Versions: 1.3.0 Environment: 1. Linux version 3.16.0-30-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #40-Ubuntu SMP Mon Jan 12 22:06:37 UTC 2015 2. Oracle Java 8 update 40 for Linux X64 3. Scala 2.10.5 4. Hadoop 2.6 (pre-build version) Reporter: John Original Estimate: 24h Remaining Estimate: 24h I build Apache Spark 1.3 munally. --- JAVA_HOME=PATH_TO_JAVA8 mvn clean package -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests --- Something goes wrong, akka always tell me --- 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. --- I build another version of Spark 1.3 + Hadoop 2.6 under Java 7. Everything goes well. Logs --- 15/03/17 21:27:06 INFO spark.SparkContext: Running Spark version 1.3.0 15/03/17 21:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/17 21:27:08 INFO spark.SecurityManager: Changing view Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: Changing modify Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser) 15/03/17 21:27:08 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/03/17 21:27:08 INFO Remoting: Starting remoting 15/03/17 21:27:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@Server3:37951] 15/03/17 21:27:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 37951. 15/03/17 21:27:09 INFO spark.SparkEnv: Registering MapOutputTracker 15/03/17 21:27:09 INFO spark.SparkEnv: Registering BlockManagerMaster 15/03/17 21:27:09 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-0db692bb-cd02-40c8-a8f0-3813c6da18e2/blockmgr-a1d0ad23-ab76-4177-80a0-a6f982a64d80 15/03/17 21:27:09 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB 15/03/17 21:27:09 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-502ef3f8-b8cd-45cf-b1df-97df297cdb35/httpd-6303e24d-4b2b-4614-bb1d-74e8d331189b 15/03/17 21:27:09 INFO spark.HttpServer: Starting HTTP Server 15/03/17 21:27:09 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48000 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'HTTP file server' on port 48000. 15/03/17 21:27:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/03/17 21:27:10 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/03/17 21:27:10 INFO ui.SparkUI: Started SparkUI at http://Server3:4040 15/03/17 21:27:10 INFO spark.SparkContext: Added JAR file:/home/hduser/spark-java2.jar at http://192.168.11.42:48000/jars/spark-java2.jar with timestamp 1426598830307 15/03/17 21:27:10 INFO client.RMProxy: Connecting to ResourceManager at Server3/192.168.11.42:8050 15/03/17 21:27:11 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 15/03/17 21:27:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 15/03/17 21:27:11 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/03/17 21:27:11 INFO yarn.Client: Setting up container launch context for our AM 15/03/17 21:27:11 INFO yarn.Client: Preparing resources for our AM container 15/03/17 21:27:12 INFO yarn.Client: Uploading resource file:/home/hduser/spark-1.3.0/assembly/target/scala-2.10/spark-assembly-1.3.0-hadoop2.6.0.jar -
[jira] [Commented] (SPARK-6388) Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40
[ https://issues.apache.org/jira/browse/SPARK-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365211#comment-14365211 ] John commented on SPARK-6388: - Thanks, I will try it later Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40 -- Key: SPARK-6388 URL: https://issues.apache.org/jira/browse/SPARK-6388 Project: Spark Issue Type: Bug Components: Block Manager, Spark Submit, YARN Affects Versions: 1.3.0 Environment: 1. Linux version 3.16.0-30-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #40-Ubuntu SMP Mon Jan 12 22:06:37 UTC 2015 2. Oracle Java 8 update 40 for Linux X64 3. Scala 2.10.5 4. Hadoop 2.6 (pre-build version) Reporter: John Original Estimate: 24h Remaining Estimate: 24h I build Apache Spark 1.3 munally. --- JAVA_HOME=PATH_TO_JAVA8 mvn clean package -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests --- Something goes wrong, akka always tell me --- 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. --- I build another version of Spark 1.3 + Hadoop 2.6 under Java 7. Everything goes well. Logs --- 15/03/17 21:27:06 INFO spark.SparkContext: Running Spark version 1.3.0 15/03/17 21:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/17 21:27:08 INFO spark.SecurityManager: Changing view Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: Changing modify Servers to: hduser 15/03/17 21:27:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser) 15/03/17 21:27:08 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/03/17 21:27:08 INFO Remoting: Starting remoting 15/03/17 21:27:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@Server3:37951] 15/03/17 21:27:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 37951. 15/03/17 21:27:09 INFO spark.SparkEnv: Registering MapOutputTracker 15/03/17 21:27:09 INFO spark.SparkEnv: Registering BlockManagerMaster 15/03/17 21:27:09 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-0db692bb-cd02-40c8-a8f0-3813c6da18e2/blockmgr-a1d0ad23-ab76-4177-80a0-a6f982a64d80 15/03/17 21:27:09 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB 15/03/17 21:27:09 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-502ef3f8-b8cd-45cf-b1df-97df297cdb35/httpd-6303e24d-4b2b-4614-bb1d-74e8d331189b 15/03/17 21:27:09 INFO spark.HttpServer: Starting HTTP Server 15/03/17 21:27:09 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48000 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'HTTP file server' on port 48000. 15/03/17 21:27:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/03/17 21:27:10 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/03/17 21:27:10 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/03/17 21:27:10 INFO ui.SparkUI: Started SparkUI at http://Server3:4040 15/03/17 21:27:10 INFO spark.SparkContext: Added JAR file:/home/hduser/spark-java2.jar at http://192.168.11.42:48000/jars/spark-java2.jar with timestamp 1426598830307 15/03/17 21:27:10 INFO client.RMProxy: Connecting to ResourceManager at Server3/192.168.11.42:8050 15/03/17 21:27:11 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 15/03/17 21:27:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 15/03/17 21:27:11 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 15/03/17 21:27:11 INFO yarn.Client: Setting up container launch context for our AM 15/03/17 21:27:11 INFO yarn.Client: Preparing resources for our AM container 15/03/17 21:27:12 INFO yarn.Client: Uploading resource file:/home/hduser/spark-1.3.0/assembly/target/scala-2.10/spark-assembly-1.3.0-hadoop2.6.0.jar - hdfs://Server3:9000/user/hduser/.sparkStaging/application_1426595477608_0002/spark-assembly-1.3.0-hadoop2.6.0.jar 15/03/17 21:27:21 INFO yarn.Client: Setting up the launch environment for our AM container 15/03/17 21:27:21 INFO spark.SecurityManager:
