Hello folks,
This is my first msg to the list. New to Spark, and trying to run the SparkPi
example shown in the Cloudera documentation. We have Cloudera 5.5.1 running on
a small cluster at our lab, with Spark 1.5.
My trial invocation is given below. The output that I get *says* that I
"SUCCEEDED" at the end. But - I don't get any screen output on the value of pi.
I also tried a SecondarySort Spark program that I compiled and jarred from Dr.
Parsian's Data Algorithms book. That program failed. So - I am focusing on
getting SparkPi to work properly, to get started. Can somebody look at the
screen output that I cut-and-pasted below and infer what I might be doing wrong?
Am I forgetting to set one or more environment variables? Or not setting such
properly?
Here is the CLASSPATH value that I set:
CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:/people/rtaylor/SparkWork/DataAlgUtils
Here is the settings of other environment variables:
HADOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
HADOOP_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar'
SPARK_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar:':'/people/rtaylor/SparkWork/DataAlgUtils'
I am not sure that those env vars are properly set (or if even all of them are
needed). But that's what I'm currently using.
As I said, the invocation below appears to terminate with final status set to
"SUCCEEDED". But - there is no screen output on the value of pi, which I
understood would be shown. So - something appears to be going wrong. I went to
the tracking URL given at the end, but could not access it.
I would very much appreciate some guidance!
- Ron Taylor
%%%%%
INVOCATION:
[rtaylor@bigdatann]$ spark-submit --class org.apache.spark.examples.SparkPi
--master yarn --deploy-mode cluster --name RT_SparkPi
/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
10
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/pig-0.12.0-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/02/12 18:16:59 INFO client.RMProxy: Connecting to ResourceManager at
bigdatann.ib/172.17.115.18:8032
16/02/12 18:16:59 INFO yarn.Client: Requesting a new application from cluster
with 15 NodeManagers
16/02/12 18:16:59 INFO yarn.Client: Verifying our application has not requested
more than the maximum memory capability of the cluster (65536 MB per container)
16/02/12 18:16:59 INFO yarn.Client: Will allocate AM container, with 1408 MB
memory including 384 MB overhead
16/02/12 18:16:59 INFO yarn.Client: Setting up container launch context for our
AM
16/02/12 18:16:59 INFO yarn.Client: Setting up the launch environment for our
AM container
16/02/12 18:16:59 INFO yarn.Client: Preparing resources for our AM container
16/02/12 18:17:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
16/02/12 18:17:00 INFO yarn.Client: Uploading resource
file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
->
hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
16/02/12 18:17:21 INFO yarn.Client: Uploading resource
file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
->
hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
16/02/12 18:17:23 INFO yarn.Client: Uploading resource
file:/tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de/__spark_conf__8357851336386157291.zip
->
hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/__spark_conf__8357851336386157291.zip
16/02/12 18:17:23 INFO spark.SecurityManager: Changing view acls to: rtaylor
16/02/12 18:17:23 INFO spark.SecurityManager: Changing modify acls to: rtaylor
16/02/12 18:17:23 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(rtaylor); users
with modify permissions: Set(rtaylor)
16/02/12 18:17:24 INFO yarn.Client: Submitting application 70 to ResourceManager
16/02/12 18:17:24 INFO impl.YarnClientImpl: Submitted application
application_1454115464826_0070
16/02/12 18:17:25 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:25 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.rtaylor
start time: 1455329844451
final status: UNDEFINED
tracking URL:
http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
user: rtaylor
16/02/12 18:17:26 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:27 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:28 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:29 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:30 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:31 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:32 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:33 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:33 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.17.115.8
ApplicationMaster RPC port: 0
queue: root.rtaylor
start time: 1455329844451
final status: UNDEFINED
tracking URL:
http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
user: rtaylor
16/02/12 18:17:34 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:35 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:36 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:37 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:38 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:39 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:40 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:41 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:42 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:43 INFO yarn.Client: Application report for
application_1454115464826_0070 (state: FINISHED)
16/02/12 18:17:43 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.17.115.8
ApplicationMaster RPC port: 0
queue: root.rtaylor
start time: 1455329844451
final status: SUCCEEDED
tracking URL:
http://bigdatann.ib:8088/proxy/application_1454115464826_0070/A
user: rtaylor
16/02/12 18:17:43 INFO util.ShutdownHookManager: Shutdown hook called
16/02/12 18:17:43 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de
[rtaylor@bigdatann SecondarySort]$
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Ronald C. Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle)
Richland, WA 99352
phone: (509) 372-6568, email: ronald.tay...@pnnl.gov
web page: http://www.pnnl.gov/science/staff/staff_info.asp?staff_num=7048
