hi there, I see a spark exception , in my spark job after running into
Thread-69, after ever 12 hrs or so ?, and NIFI receiver stops receiving any
more data into spark application.Is this a bug in NIFI receiver jar file? or
something I am doing wrong, I can send you my spark code if you are
interested.Please advice.
ERROR cluster.YarnScheduler: Lost executor 7 on sxn33.dtcc.localdomain:
remote Rpc client disassociated
*16/07/07 10:38:51 ERROR scheduler.TaskSetManager: Task 0 in stage 3.0
failed 4 times; aborting job
Exception in thread "Thread-69" org.apache.spark.SparkException: Job aborted
due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent
failure: Lost task 0.3 in stage 3.0 (TID 26932, sxn33.dtcc.localdomain):
ExecutorLostFailure (executor 7 lost)*
Driver stacktrace:
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1273)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1264)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1263)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1263)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1457)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1418)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
--
View this message in context:
http://apache-nifi-developer-list.39713.n7.nabble.com/thread-69-exception-in-spark-application-tp12690.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
