Hi, some of you could help me?
thanks a lot! ---------- Forwarded message ---------- From: Gianluigi Barbarano <gianluigi.barbar...@gmail.com> Date: 2015-08-21 12:35 GMT+02:00 Subject: Use Zeppelin in CDH cluster, using Yarn To: users@zeppelin.incubator.apache.org Hi all, I want to use Zeppelin in my CDH cluster, to run spark and pyspark code, through Yarn. My environment is: CDH 5.4.2 (spark and yarn installed) Zeppelin 0.5.0 I've built Zeppelin in this way: "mvn clean package -Pspark-1.3 -Ppyspark -Dhadoop.version=2.6.0-cdh5.4.2 -Phadoop-2.6 -Pyarn -DskipTests" and installed it on a node of my CDH cluster. I've set the following env. variables: export ZEPPELIN_HOME=/opt/incubator-zeppelin export ZEPPELIN_PORT=7979 export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark export SPARK_DIST_CLASSPATH=$(hadoop classpath) export HIVE_CONF_DIR="/etc/hive/conf" export HIVECLASSPATH=$(find /opt/cloudera/parcels/CDH/lib/hive/lib/ -name '*.jar' -print0 | sed 's/\x0/:/g') My zeppelin-env.sh is: export MASTER=yarn export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=512m -Dspark.cores.max=1" export HADOOP_CONF_DIR="/etc/hadoop/conf.cloudera.yarn" export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH Spark-Jar: /opt/cloudera/parcels/CDH/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.2-hadoop2.6.0-cdh5.4.2.jar When I execute a simple PYSPARK code in my notebook: sc.textFile("/user/example/test.txt").count() I see the new application in my Yarn console, Zeppelin tells me that it's RUNNING but nothing happens: I don't receive any results and I can't execute any other code. No errors in my Yarn log. In Zeppelin log: INFO [2015-08-21 12:15:13,740] ({pool-1-thread-2} SchedulerFactory.java[jobStarted]:132) - Job paragraph_1440152108501_1888952717 started by scheduler remoteinterpreter_695055854 INFO [2015-08-21 12:15:13,745] ({pool-1-thread-2} Paragraph.java[jobRun]:189) - run paragraph 20150821-121508_181152709 using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@91f60ce INFO [2015-08-21 12:15:13,776] ({pool-1-thread-2} RemoteInterpreterProcess.java[reference]:108) - Run interpreter process /opt/incubator-zeppelin/bin/interpreter.sh -d /opt/incubator-zeppelin/interpreter/spark$ INFO [2015-08-21 12:15:15,346] ({pool-1-thread-2} RemoteInterpreter.java[init]:144) - Create remote interpreter org.apache.zeppelin.spark.PySparkInterpreter INFO [2015-08-21 12:15:15,416] ({pool-1-thread-2} RemoteInterpreter.java[init]:144) - Create remote interpreter org.apache.zeppelin.spark.SparkInterpreter INFO [2015-08-21 12:15:15,423] ({pool-1-thread-2} RemoteInterpreter.java[init]:144) - Create remote interpreter org.apache.zeppelin.spark.SparkSqlInterpreter INFO [2015-08-21 12:15:15,425] ({pool-1-thread-2} RemoteInterpreter.java[init]:144) - Create remote interpreter org.apache.zeppelin.spark.DepInterpreter INFO [2015-08-21 12:15:15,440] ({pool-1-thread-2} Paragraph.java[jobRun]:206) - RUN : sc.textFile("/user/admin/1.txt").count() INFO [2015-08-21 12:15:25,146] ({qtp1379078592-48} NotebookServer.java[onMessage]:112) - RECEIVE << PING INFO [2015-08-21 12:15:58,657] ({Thread-33} NotebookServer.java[broadcast]:264) - SEND >> NOTE INFO [2015-08-21 12:15:58,800] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS INFO [2015-08-21 12:15:59,314] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS INFO [2015-08-21 12:15:59,829] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS INFO [2015-08-21 12:16:00,344] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS INFO [2015-08-21 12:16:00,857] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS INFO [2015-08-21 12:16:01,370] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS INFO [2015-08-21 12:16:01,883] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS INFO [2015-08-21 12:16:02,397] ({Thread-34} NotebookServer.java[broadcast]:264) - SEND >> PROGRESS If I execute the similar code in %spark, I see the application running in Yarn console but after some seconds Zeppelin gives me this error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, cdhlva03.gcio.unicredit.eu): ExecutorLostFailure (executor 4 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) Guys, do you have some ideas? Thanks a lot in advance