These seem like s3 connection errors for the table data. Wondering, since
we don't see that many failures on hive. I also set the spark.task.maxFailures
= 15.

On Fri, Oct 24, 2014 at 12:15 PM, Sadhan Sood <[email protected]> wrote:

> Hi,
>
> Trying to run a query on spark-sql but it keeps failing with this error on
> the cli ( we are running spark-sql on a yarn cluster):
>
>
> org.apache.spark.SparkException: Job cancelled because SparkContext was
> shut down
>   at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:700)
>   at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:699)
>   at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
>   at
> org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:699)
>   at
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor.postStop(DAGScheduler.scala:1405)
>   at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
>   at
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundPostStop(DAGScheduler.scala:1352)
>   at
> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
>   at
> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
>   at akka.actor.ActorCell.terminate(ActorCell.scala:369)
>   at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
>   at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>   at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>   at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> On the UI, I see there are connection failures in mapPartition stage:
>
> java.net.SocketTimeoutException: Read timed out
>         java.net.SocketInputStream.socketRead0(Native Method)
>         java.net.SocketInputStream.read(SocketInputStream.java:150)
>         java.net.SocketInputStream.read(SocketInputStream.java:121)
>         java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>         java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>         java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>         sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:703)
>         sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
>         
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534)
>         
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
>         org.apache.spark.util.Utils$.fetchFile(Utils.scala:362)
>         
> org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:331)
>         
> org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$6.apply(Executor.scala:329)
>         
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
>         
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>         
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>         
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
>         scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>         scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
>         
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
>         
> org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:329)
>         org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:162)
>         
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         java.lang.Thread.run(Thread.java:745)
>
>

Reply via email to