Setting the following while creating the sparkContext will sort it out.
.set("spark.core.connection.ack.wait.timeout","600")
.set("spark.akka.frameSize","50")
On 27 Oct 2014 21:15, "shahab" <shahab.mok...@gmail.com> wrote:
> Hi,
>
> I have a stand alone Spark Cluster, where worker and master reside on the
> same machine. I submit a job to the cluster, the job is executed for a
> while and suddenly I get this exception with no additional trace.
>
> ConnectionManager: key already cancelled ?
> sun.nio.ch.SelectionKeyImpl@2490dce9
> java.nio.channels.CancelledKeyException at
> org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at
> org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
>
>
> Any idea where should I look for the cause?
>
> best,
> /shahab
>
> This following is the part of printout from "driver application" logs:
>
> 14/10/27 15:21:15 INFO BlockManagerInfo: Removed broadcast_1_piece0 on
> ip-10-89-32-179.eu-west-1.compute.internal:40479 in memory (size: 3.4 KB,
> free: 1565.6 MB)
> 14/10/27 15:21:15 INFO ContextCleaner: Cleaned broadcast 1
> 14/10/27 15:21:15 INFO ShuffleBlockManager: Could not find files for
> shuffle 1 for deleting
> 14/10/27 15:21:15 INFO ContextCleaner: Cleaned shuffle 1
> 14/10/27 15:21:15 INFO ShuffleBlockManager: Could not find files for
> shuffle 0 for deleting
> 14/10/27 15:21:15 INFO ContextCleaner: Cleaned shuffle 0
> 14/10/27 15:21:15 INFO BlockManagerInfo: Removed taskresult_9 on
> ip-10-zz.xx-yy:40479 in memory (size: 24.1 MB, free: 1589.8 MB)
> 14/10/27 15:21:16 INFO DAGScheduler: Stage 7 (collect at
> TimeBenchmarking_SimpleModel.scala:55) finished in 3.209 s
> 14/10/27 15:21:16 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID
> 9) in 2640 ms onip-10-zz.xx-yy (1/1)
> 14/10/27 15:21:16 INFO SparkContext: Job finished: collect at
> TimeBenchmarking_SimpleModel.scala:55, took 102.661420511 s
> 14/10/27 15:21:16 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks
> have all completed, from pool
> 14/10/27 15:21:16 INFO SparkUI: Stopped Spark web UI at
> http://ip-10-zz.xx-yy:4040
> 14/10/27 15:21:16 INFO DAGScheduler: Stopping DAGScheduler
> 14/10/27 15:21:16 INFO SparkDeploySchedulerBackend: Shutting down all
> executors
> 14/10/27 15:21:16 INFO SparkDeploySchedulerBackend: Asking each executor
> to shut down
> 14/10/27 15:21:16 INFO ConnectionManager: Removing ReceivingConnection to
> ConnectionManagerId(ip-10-zz.xx-yy, 40479)
> 14/10/27 15:21:16 INFO ConnectionManager: Removing SendingConnection to
> ConnectionManagerId(ip-10-zz.xx-yy,40479)
> 14/10/27 15:21:16 INFO ConnectionManager: Removing SendingConnection to
> ConnectionManagerId(ip-10-zz.xx-yy,40479)
> 14/10/27 15:21:16 INFO ConnectionManager: Key not valid ?
> sun.nio.ch.SelectionKeyImpl@2490dce9
> 14/10/27 15:21:16 INFO ConnectionManager: key already cancelled ?
> sun.nio.ch.SelectionKeyImpl@2490dce9
> java.nio.channels.CancelledKeyException
> at
> org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
> at
> org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/10/27 15:21:17 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor
> stopped!
> 14/10/27 15:21:17 INFO ConnectionManager: Selector thread was interrupted!
>
