Hi, I just tried sample PI calculation on Spark Cluster, after returning the Pi result, it shows ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(m37,35662) not found
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://m33:7077 --executor-memory 512m --total-executor-cores 40 examples/target/spark-examples_2.10-1.1.0.jar 100 14/10/23 05:09:03 INFO TaskSetManager: Finished task 87.0 in stage 0.0 (TID 87) in 346 ms on m134.emblocsoft.net (99/100) 14/10/23 05:09:03 INFO TaskSetManager: Finished task 98.0 in stage 0.0 (TID 98) in 262 ms on m134.emblocsoft.net (100/100) 14/10/23 05:09:03 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 14/10/23 05:09:03 INFO DAGScheduler: Stage 0 (reduce at SparkPi.scala:35) finished in 2.597 s 14/10/23 05:09:03 INFO SparkContext: Job finished: reduce at SparkPi.scala:35, took 2.725328861 s Pi is roughly 3.1414948 14/10/23 05:09:03 INFO SparkUI: Stopped Spark web UI at http://m33:4040 14/10/23 05:09:03 INFO DAGScheduler: Stopping DAGScheduler 14/10/23 05:09:03 INFO SparkDeploySchedulerBackend: Shutting down all executors 14/10/23 05:09:03 INFO SparkDeploySchedulerBackend: Asking each executor to shut down 14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@37852165 14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(m37,35662) 14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(m37,35662) 14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(m37,35662) not found 14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@37852165 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(m36,34230) 14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(m35,50371) 14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(m36,34230) 14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(m34,41562) 14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(m36,34230) not found 14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(m35,50371) 14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(m35,50371) not found 14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(m33,39517) 14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(m33,39517) 14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(m34,41562) 14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(m33,39517) not found 14/10/23 05:09:04 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(m34,41562) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/10/23 05:09:04 INFO ConnectionManager: Handling connection error on connection to ConnectionManagerId(m34,41562) 14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(m34,41562) 14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2e0b5c4a 14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2e0b5c4a java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@653f8844 14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@653f8844 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@ac2a183 14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@ac2a183 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(m34,41562) 14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@6bd815e3 14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@6bd815e3 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/10/23 05:09:04 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped! 14/10/23 05:09:04 INFO ConnectionManager: Selector thread was interrupted! 14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(m37,35662) 14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(m37,35662) not found 14/10/23 05:09:04 WARN ConnectionManager: All connections not cleaned up 14/10/23 05:09:04 INFO ConnectionManager: ConnectionManager stopped 14/10/23 05:09:04 INFO MemoryStore: MemoryStore cleared 14/10/23 05:09:04 INFO BlockManager: BlockManager stopped 14/10/23 05:09:04 INFO BlockManagerMaster: BlockManagerMaster stopped 14/10/23 05:09:04 INFO SparkContext: Successfully stopped SparkContext 14/10/23 05:09:04 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. How to handle this? Regards Arthur
