Chen Avnery created SPARK-3549: ---------------------------------- Summary: Yarn client mode does not shutdown clearnly Key: SPARK-3549 URL: https://issues.apache.org/jira/browse/SPARK-3549 Project: Spark Issue Type: Bug Components: PySpark, YARN Affects Versions: 1.1.0 Environment: Cloudera Hadoop 2.3, YARN fair-scheduler, PySpark 1.1.0 Reporter: Chen Avnery Priority: Minor
When I call SparkContext.stop() at the end of my pyspark client mode application, I get a barrage of exceptions: {quote} 14/09/16 17:44:37 INFO SparkContext: Job finished: foreach at /home/chena/projects/sparkapps /dummy.py:34, took 24.597733296 s 14/09/16 17:44:37 INFO SparkUI: Stopped Spark web UI at http://workhorse.dummycorp.local:4040 14/09/16 17:44:37 INFO DAGScheduler: Stopping DAGScheduler 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Shutting down all executors 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Asking each executor to shut down 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 26 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 26 on g012.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 90 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 90 on g017.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 70 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 70 on g019.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 152 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 152 on g027.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 140 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 140 on g019.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 18 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 18 on g021.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 145 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 145 on g027.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 166 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 166 on g027.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 9 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 9 on g027.dummycorp.local: remo te Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 32 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 32 on g021.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 3 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 3 on g029.dummycorp.local: remo te Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 163 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 163 on g029.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 149 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 149 on g029.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 17 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 17 on g029.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 89 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 89 on g007.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 128 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 128 on g029.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 80 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 80 on g023.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 5 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 5 on g007.dummycorp.local: remo te Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 57 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 57 on g015.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 124 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 124 on g027.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 157 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 157 on g021.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 150 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 150 on g021.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 105 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 105 on g024.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 51 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 51 on g027.dummycorp.local: rem ote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 159 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 159 on g027.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Executor 107 disconnected, so removing it 14/09/16 17:44:37 ERROR YarnClientClusterScheduler: Lost executor 107 on g027.dummycorp.local: re mote Akka client disassociated 14/09/16 17:44:37 INFO YarnClientSchedulerBackend: Stopped 14/09/16 17:44:37 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,39146) 14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,39146) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48080) 14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,58341) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,58341) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,58341) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,35349) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,35349) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,57831) 14/09/16 17:44:37 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,48080) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,48080) not found 14/09/16 17:44:37 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,39146) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,57831) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,57831) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,35349) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2eb3ec34 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@2eb3ec 34 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1d321764 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1d3217 64 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1ee061c2 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1ee061 c2 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,39250) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,37654) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54108) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,55074) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,55515) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,56089) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,48853) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,36788) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,57170) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,60009) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,60881) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,41322) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,36171) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,41093) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,53156) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60519) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,44689) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,33434) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,39250) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,39250) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,45596) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,32898) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,54108) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,54108) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,46744) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48837) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,56582) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,53880) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,33434) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,33434) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,33434) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36358) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,60519) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,60519) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,54831) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,60881) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54400) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,56582) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,56582) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,59023) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,36131) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,36358) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,36358) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,48517) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,48898) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,34401) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,46744) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,46744) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,36171) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,36171) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,48898) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,48898) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,48898) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,49829) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,40583) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,48440) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54647) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54647) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,54647) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,45596) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,45596) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,41759) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,48837) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,48837) not found 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,60881) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g021.dummycorp.local,47580) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,47580) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,47580) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,36748) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,39250) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,59257) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44584) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,45485) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,59023) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,59023) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,35412) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,54817) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,40583) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,40583) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,46499) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,36788) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,36788) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36788) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,37654) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,37654) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,36171) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,49829) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,49829) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,33175) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,45485) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,45485) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,50451) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,39260) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,54217) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,34576) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,60009) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,60009) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,44584) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,44584) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,54214) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,47763) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,35412) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,35412) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,59836) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55705) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,59836) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,59836) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,32898) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,32898) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,44689) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,44689) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,44689) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,50848) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,48853) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,48853) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,54817) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,54817) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,59257) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,59257) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,52380) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@5259e682 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@5259e6 82 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@bf7e726 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@bf7e72 6 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@2444569 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@244456 9 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@63f0beab 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@63f0be ab java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@3743a434 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@3743a4 34 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@406875ed 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@406875 ed java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@1ab75e8d 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@1ab75e 8d java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,60207) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,50848) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,50848) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,50848) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,51799) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,55705) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,55705) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,51799) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,51799) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,51799) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,54817) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,50451) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,50451) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,50451) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,34576) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,34576) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,46499) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,46499) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,46499) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,33175) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,33175) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,33175) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,54214) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,54214) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,60303) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,47763) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,47763) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,41759) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,41759) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,60207) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,60207) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,48440) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,48440) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,34401) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,34401) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,39260) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,39260) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,39260) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,55806) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,57170) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,57170) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,57170) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,46077) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,36748) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,36748) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,36748) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,53880) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,53880) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,53880) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g027.dummycorp.local,60303) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g027.dummycorp.local,60303) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,41093) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,41093) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,41322) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,41322) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,46077) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,46077) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g021.dummycorp.local,53156) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 21.dummycorp.local,53156) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,48517) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,48517) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,48517) 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g019.dummycorp.local,36131) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g019.dummycorp.local,36131) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g019.dummycorp.local,36131) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,54217) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,54217) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,55515) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,55515) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g019.dummycorp.local,54831) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 19.dummycorp.local,54831) not found 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g015.dummycorp.local,56089) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 15.dummycorp.local,56089) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g029.dummycorp.local,54400) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g029.dummycorp.local,54400) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54400) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g029.dummycorp.local,52380) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g0 29.dummycorp.local,52380) not found 14/09/16 17:44:38 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionMa nagerId(g015.dummycorp.local,55074) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:390) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 17:44:38 INFO ConnectionManager: Handling connection error on connection to ConnectionManag erId(g015.dummycorp.local,55074) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g015.dummycorp.local,55074) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g027.dummycorp.local,60303) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214) 14/09/16 17:44:38 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(g029.dummycorp.local,54214) 14/09/16 17:44:38 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(g027.dummycorp.local,55806) 14/09/16 17:44:38 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(g027.dummycorp.local,55806) not found 14/09/16 17:44:38 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@653f8d49 14/09/16 17:44:38 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@653f8d 49 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 14/09/16 17:44:38 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped! 14/09/16 17:44:38 INFO ConnectionManager: Selector thread was interrupted! 14/09/16 17:44:38 INFO ConnectionManager: ConnectionManager stopped 14/09/16 17:44:38 INFO MemoryStore: MemoryStore cleared 14/09/16 17:44:38 INFO BlockManager: BlockManager stopped 14/09/16 17:44:38 INFO BlockManagerMaster: BlockManagerMaster stopped 14/09/16 17:44:38 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 14/09/16 17:44:38 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceedin g with flushing remote transports. 14/09/16 17:44:39 INFO SparkContext: Successfully stopped SparkContext {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org