Hi Guys, As part of debugging this "native library" error in our environment, it would be great if somebody can help me with this question. What kind of temp, scratch, and staging directories does Spark need and use on the slave nodes in the YARN cluster mode?
Thanks, Aravind On Mon, Nov 3, 2014 at 4:11 PM, Aravind Srinivasan <arav...@altiscale.com> wrote: > Team, > > We are running a build of spark 1.1.1 for hadoop 2.2. We can't get the > code to read LZO or snappy files in YARN. It fails to find the native libs. > I have tried many different ways of defining the lib path - > LD_LIBRARY_PATH, --driver-class-path, spark.executor.extraLibraryPath in > spark-defaults.conf, --driver-java-options, and SPARK_LIBRARY_PATH. But > none of them seem to take effect. What am I missing? Or is this a known > issue? > > The example below (HdfsTest) works with plain text on both cluster and > local mode. LZO and snappy files work on local mode, but both fail in the > YARN cluster mode > > LD_LIBRARY_PATH=/opt/hadoop/lib/native/ MASTER=yarn > SPARK_EXAMPLES_JAR=./examples/target/spark-examples_2.10-1.1.1.jar > ./bin/run-example HdfsTest /user/input/part-r-00000.snappy > > Stack Trace: > Exception in thread "main" org.apache.spark.SparkException: Job aborted > due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent > failure: Lost task 0.3 in stage 0.0 (TID 3, 101-26-03.sc1.verticloud.com): > ExecutorLostFailure (executor lost) > Driver stacktrace: > at org.apache.spark.scheduler.DAGScheduler.org > $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) > at scala.Option.foreach(Option.scala:236) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) > at akka.actor.ActorCell.invoke(ActorCell.scala:456) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > Thanks, > Aravind > >
