Try looking at stdout logs. I ran the exactly same job as you and did not see anything on the console as well but found it in stdout.
[csingh@<> ~]$ spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --name RT_SparkPi /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar 10 Log Type: stdout Log Upload Time: Sat Feb 13 11:00:08 +0000 2016 Log Length: 23 Pi is roughly 3.140224 Hope that helps! On Sat, Feb 13, 2016 at 3:14 AM, Taylor, Ronald C <ronald.tay...@pnnl.gov> wrote: > Hello folks, > > This is my first msg to the list. New to Spark, and trying to run the > SparkPi example shown in the Cloudera documentation. We have Cloudera > 5.5.1 running on a small cluster at our lab, with Spark 1.5. > > My trial invocation is given below. The output that I get **says** that I > “SUCCEEDED” at the end. But – I don’t get any screen output on the value of > pi. I also tried a SecondarySort Spark program that I compiled and jarred > from Dr. Parsian’s Data Algorithms book. That program failed. So – I am > focusing on getting SparkPi to work properly, to get started. Can somebody > look at the screen output that I cut-and-pasted below and infer what I > might be doing wrong? > > Am I forgetting to set one or more environment variables? Or not setting > such properly? > > Here is the CLASSPATH value that I set: > > > CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:/people/rtaylor/SparkWork/DataAlgUtils > > Here is the settings of other environment variables: > > HADOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11 > SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11 > > HADOOP_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar' > > SPARK_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar:':'/people/rtaylor/SparkWork/DataAlgUtils' > > I am not sure that those env vars are properly set (or if even all of them > are needed). But that’s what I’m currently using. > > As I said, the invocation below appears to terminate with final status set > to “SUCCEEDED”. But – there is no screen output on the value of pi, which I > understood would be shown. So – something appears to be going wrong. I went > to the tracking URL given at the end, but could not access it. > > I would very much appreciate some guidance! > > > - Ron Taylor > > > %%%%% > > INVOCATION: > > [rtaylor@bigdatann]$ spark-submit --class > org.apache.spark.examples.SparkPi --master yarn --deploy-mode > cluster --name RT_SparkPi > /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar > 10 > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/pig-0.12.0-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 16/02/12 18:16:59 INFO client.RMProxy: Connecting to ResourceManager at > bigdatann.ib/172.17.115.18:8032 > 16/02/12 18:16:59 INFO yarn.Client: Requesting a new application from > cluster with 15 NodeManagers > 16/02/12 18:16:59 INFO yarn.Client: Verifying our application has not > requested more than the maximum memory capability of the cluster (65536 MB > per container) > 16/02/12 18:16:59 INFO yarn.Client: Will allocate AM container, with 1408 > MB memory including 384 MB overhead > 16/02/12 18:16:59 INFO yarn.Client: Setting up container launch context > for our AM > 16/02/12 18:16:59 INFO yarn.Client: Setting up the launch environment for > our AM container > 16/02/12 18:16:59 INFO yarn.Client: Preparing resources for our AM > container > 16/02/12 18:17:00 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 16/02/12 18:17:00 INFO yarn.Client: Uploading resource > file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar > -> > hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar > 16/02/12 18:17:21 INFO yarn.Client: Uploading resource > file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar > -> > hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar > 16/02/12 18:17:23 INFO yarn.Client: Uploading resource > file:/tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de/__spark_conf__8357851336386157291.zip > -> > hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/__spark_conf__8357851336386157291.zip > 16/02/12 18:17:23 INFO spark.SecurityManager: Changing view acls to: > rtaylor > 16/02/12 18:17:23 INFO spark.SecurityManager: Changing modify acls to: > rtaylor > 16/02/12 18:17:23 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(rtaylor); users with modify permissions: Set(rtaylor) > 16/02/12 18:17:24 INFO yarn.Client: Submitting application 70 to > ResourceManager > 16/02/12 18:17:24 INFO impl.YarnClientImpl: Submitted application > application_1454115464826_0070 > 16/02/12 18:17:25 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:25 INFO yarn.Client: > client token: N/A > diagnostics: N/A > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: root.rtaylor > start time: 1455329844451 > final status: UNDEFINED > tracking URL: > http://bigdatann.ib:8088/proxy/application_1454115464826_0070/ > user: rtaylor > 16/02/12 18:17:26 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:27 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:28 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:29 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:30 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:31 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:32 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: ACCEPTED) > 16/02/12 18:17:33 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:33 INFO yarn.Client: > client token: N/A > diagnostics: N/A > ApplicationMaster host: 172.17.115.8 > ApplicationMaster RPC port: 0 > queue: root.rtaylor > start time: 1455329844451 > final status: UNDEFINED > tracking URL: > http://bigdatann.ib:8088/proxy/application_1454115464826_0070/ > user: rtaylor > 16/02/12 18:17:34 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:35 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:36 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:37 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:38 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:39 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:40 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:41 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:42 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: RUNNING) > 16/02/12 18:17:43 INFO yarn.Client: Application report for > application_1454115464826_0070 (state: FINISHED) > 16/02/12 18:17:43 INFO yarn.Client: > client token: N/A > diagnostics: N/A > ApplicationMaster host: 172.17.115.8 > ApplicationMaster RPC port: 0 > queue: root.rtaylor > start time: 1455329844451 > final status: SUCCEEDED > tracking URL: > http://bigdatann.ib:8088/proxy/application_1454115464826_0070/A > user: rtaylor > 16/02/12 18:17:43 INFO util.ShutdownHookManager: Shutdown hook called > 16/02/12 18:17:43 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de > [rtaylor@bigdatann SecondarySort]$ > > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > > Ronald C. Taylor, Ph.D. > Computational Biology & Bioinformatics Group > Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle) > Richland, WA 99352 > phone: (509) 372-6568, email: ronald.tay...@pnnl.gov > web page: http://www.pnnl.gov/science/staff/staff_info.asp?staff_num=7048 > > > >
