I hope that helps https://stackoverflow.com/questions/40623957/slave-lost-and-very-slow-join-in-spark
Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2017-07-26 3:13 GMT+02:00 Debabrata Ghosh <mailford...@gmail.com>: > Hi, > While executing a SparkSQL query, I am hitting the > following error. Wonder, if you can please help me with a possible cause > and resolution. Here is the stacktrace for the same: > > 07/25/2017 02:41:58 PM - DataPrep.py 323 - __main__ - ERROR - An error > occurred while calling o49.sql. > > : org.apache.spark.SparkException: Job aborted due to stage failure: Task > 0 in stage 12.2 failed 4 times, most recent failure: Lost task 0.3 in stage > 12.2 (TID 2242, bicservices.hdp.com): ExecutorLostFailure (executor 25 > exited caused by one of the running tasks) Reason: Slave lost > > Driver stacktrace: > > at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$ > scheduler$DAGScheduler$$failJobAndIndependentStages( > DAGScheduler.scala:1433) > > at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply( > DAGScheduler.scala:1421) > > at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply( > DAGScheduler.scala:1420) > > at scala.collection.mutable.ResizableArray$class.foreach( > ResizableArray.scala:59) > > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > > at org.apache.spark.scheduler.DAGScheduler.abortStage( > DAGScheduler.scala:1420) > > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:801) > > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:801) > > at scala.Option.foreach(Option.scala:236) > > at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed( > DAGScheduler.scala:801) > > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > doOnReceive(DAGScheduler.scala:1642) > > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1601) > > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1590) > > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > > at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:622) > > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1856) > > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1869) > > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1946) > > at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.saveAsHiveFile( > InsertIntoHiveTable.scala:84) > > at org.apache.spark.sql.hive.execution.InsertIntoHiveTable. > sideEffectResult$lzycompute(InsertIntoHiveTable.scala:201) > > at org.apache.spark.sql.hive.execution.InsertIntoHiveTable. > sideEffectResult(InsertIntoHiveTable.scala:127) > > at org.apache.spark.sql.hive.execution.InsertIntoHiveTable. > doExecute(InsertIntoHiveTable.scala:276) > > at org.apache.spark.sql.execution.SparkPlan$$anonfun$ > execute$5.apply(SparkPlan.scala:132) > > at org.apache.spark.sql.execution.SparkPlan$$anonfun$ > execute$5.apply(SparkPlan.scala:130) > > at org.apache.spark.rdd.RDDOperationScope$.withScope( > RDDOperationScope.scala:150) > > at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) > > at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute( > QueryExecution.scala:55) > > at org.apache.spark.sql.execution.QueryExecution. > toRdd(QueryExecution.scala:55) > > at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145) > > at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130) > > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) > > at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) > > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) > > at py4j.Gateway.invoke(Gateway.java:259) > > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) > > at py4j.commands.CallCommand.execute(CallCommand.java:79) > > at py4j.GatewayConnection.run(GatewayConnection.java:209) > at java.lang.Thread.run(Thread.java:745) > > Cheers, > > Debu >
