Evans, we couldn't use BigTop Provisioner since ppc artifacts are not in the repository yet. That is why we had to made the script. Thanks
On Thu, Feb 18, 2016 at 1:40 PM, Konstantin Boudnik <[email protected]> wrote: > On Fri, Feb 19, 2016 at 02:07AM, Evans Ye wrote: > > Have you tried Bigtop Provisioner? > > > https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Provisioner+User+Guide > > > > ----- > > $ cd bigtop/bigtop-deploy/vm/vagrant-puppet-vm > > $ cat vagrantconfig.yaml # You just need to update repo and add spark > into > > component list > > memory_size: 4096 > > number_cpus: 1 > > box: "puppetlabs/centos-7.0-64-nocm" > > repo: " > http://bigtop-repos.s3.amazonaws.com/releases/1.1.0/centos/7/x86_64"; > > num_instances: 1 > > distro: centos > > components: [hadoop, yarn, spark] > > enable_local_repo: false > > run_smoke_tests: false > > smoke_test_components: [mapreduce, pig] > > jdk: "java-1.7.0-openjdk-devel.x86_64" > > > > $ ./docker-hadoop.sh -c 1 # wait for 5mins > > $ vagrant ssh bigtop1 > > $ cd /usr/lib/spark > > $ spark-submit --class org.apache.spark.examples.SparkPi --master > > yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar 10 > > ... > > Pi is roughly 3.144728 > > Good enough for 1.1.0 release ;) > > But seriously: we have a real good tool to provision clusters quickly and > painlessly. And with Puppet to guarantee the consistency. > > Cos > > > ----- > > > > Is this what you want? > > I can run this either on Docker or on a CentOS 7 VM. > > Sorry I don't have PPC machine to test. > > > > > > 2016-02-18 11:46 GMT+08:00 MrAsanjar . <[email protected]>: > > > > > I have build a single node hadoop/spark sandbox based on the latest > > > Apachue Bigtop 1.1.0 build. Spark in standalone mode + HDFS functions > > > perfectly, however, fails if yarn-client/yarn-master mode is used as > > > follow: > > > > > > *>>spark-submit --class org.apache.spark.examples.SparkPi --master > > > yarn-client /usr/lib/spark/lib/spark-examples-1.5.1-hadoop2.7.1.jar > 10* > > > 16/02/17 05:19:52 ERROR YarnClientSchedulerBackend: Yarn application > has > > > already exited with state FINISHED! > > > Exception in thread "main" java.lang.IllegalStateException: Cannot call > > > methods on a stopped SparkContext > > > at org.apache.spark.SparkContext.org > > > $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) > > > ...... > > > > > > > > > Looking at yarn Application log file, there is a *RECEIVED SIGNAL 15: > > > SIGTERM *termination signal from the the yarn container. > > > >>*yarn logs -applicationId application_1455683261278_0001* > > > > > > YARN executor launch context: > > > env: > > > CLASSPATH -> > > > > > > > {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* > > > SPARK_LOG_URL_STDERR -> > http://2657cd5dc2f6:8042/node/containerlogs > > > > > > > =============================================================================== > > > > > > 16/02/17 04:29:18 INFO impl.ContainerManagementProtocolProxy: Opening > proxy > > > : 2657cd5dc2f6:33785 > > > 16/02/17 04:29:18 ERROR yarn.ApplicationMaster: *RECEIVED SIGNAL 15: > > > SIGTERM* > > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Final app status: > UNDEFINED, > > > exitCode: 0, (reason: Shutdown hook called before final status was > > > reported.) > > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Unregistering > > > ApplicationMaster with UNDEFINED (diag message: Shutdown hook called > before > > > final status was reported.) > > > 16/02/17 04:29:18 INFO impl.AMRMClientImpl: Waiting for application to > be > > > successfully unregistered. > > > 16/02/17 04:29:18 INFO yarn.ApplicationMaster: Deleting staging > directory > > > .sparkStaging/application_1455683261278_0001 > > > 16/02/17 04:29:18 INFO util.ShutdownHookManager: Shutdown hook called > > > End of LogType:stderr > > > > > > > > > BTW, I have successfully tested hadoop yarn by running Teragen/Terasort > > > mapreduce job. > > > Before i start debugging, has anyone tested spark in yarn-client mode? > > > >
