Hi wangyi:
have more detail information？
I guess it maybe caused by need a jars that havn't upload to the
workers,such as your main class.
./bin/spark-class org.apache.spark.deploy.Client launch
[client-options] \
<cluster-url> <application-jar-url> <main-class> \
[application-options]
application-jar-url: Path to a bundled jar including your application
and all dependencies. Currently, the URL must be globally visible
inside of your cluster, for instance, an `hdfs://` path or a `file://`
path that is present on all nodes.
Hi ,
I use spark 0.9 run a simple computation, but it failled when I use
standalone mode
code:
* val sc = new SparkContext(args(0), "BayesAnalysis",
System.getenv("SPARK_HOME"), SparkContext.jarOfClass(this.getClass).toSeq) *
* val dataSet = sc.textFile(args(1)).map(_.split(",")).filter(_.length
== 14).collect; for(record <- dataSet){
println(record.mkString(" ")) }*
run on local mode, it sucessfully done.
run on standalone mode, it failed
java -classpath realclaspath mainclass sparkMaster hdfsFile
*14/08/11 15:10:07 INFO scheduler.DAGScheduler: Failed to run collect at
BayesAnalysis.scala:416 Exception in thread "main"
org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times
(most recent failure: unknown) *
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.org
<http://org.apache.spark.scheduler.dagscheduler.org/>
$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2014-08-11 15:41 GMT+08:00 jeanlyn92 <jeanly...@gmail.com>:
> Hi wangyi:
> have more detail information？
> I guess it maybe cause by need a jars that havn't upload to the works.
>
> ./bin/spark-class org.apache.spark.deploy.Client launch
> [client-options] \
> <cluster-url> <application-jar-url> <main-class> \
> [application-options]
>
>
> application-jar-url: Path to a bundled jar including your application and all
> dependencies. Currently, the URL must be globally visible inside of your
> cluster, for instance, an `hdfs://` path or a `file://` path that is present
> on all nodes.
>
>
>
>
>
> Hi ,
> I use spark 0.9 run a simple computation, but it failled when I use
> standalone mode
>
> code:
> * val sc = new SparkContext(args(0), "BayesAnalysis",
> System.getenv("SPARK_HOME"), SparkContext.jarOfClass(this.getClass).toSeq) *
>
>
>
>
> * val dataSet = sc.textFile(args(1)).map(_.split(",")).filter(_.length
> == 14).collect; for(record <- dataSet){
> println(record.mkString(" ")) }*
>
> run on local mode, it sucessfully done.
> run on standalone mode, it failed
> java -classpath realclaspath mainclass sparkMaster hdfsFile
>
>
> *14/08/11 15:10:07 INFO scheduler.DAGScheduler: Failed to run collect at
> BayesAnalysis.scala:416 Exception in thread "main"
> org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times
> (most recent failure: unknown) *
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
