Hi everyone, I'm writing a program that update a cassandra table.
I've writen a first shot where I update the table row by row from a rdd trhough a map. Now I want to build a batch of updates using the same kind of syntax as in this thread : https://groups.google.com/forum/#!msg/spark-users/LUb7ZysYp2k/MhymcFddb8cJ But as soon as I use a mappartition I get a " key already cancelled error". The program updates the table properly but it seems that the problem appears when the driver try to shut down the ressources. 15/01/26 00:07:00 INFO SparkContext: Job finished: collect at CustomerIdReconciliation.scala:143, took 1.998601568 s 15/01/26 00:07:00 INFO SparkUI: Stopped Spark web UI at http://cim1-dev:4044 15/01/26 00:07:00 INFO DAGScheduler: Stopping DAGScheduler 15/01/26 00:07:00 INFO SparkDeploySchedulerBackend: Shutting down all executors 15/01/26 00:07:00 INFO SparkDeploySchedulerBackend: Asking each executor to shut down 15/01/26 00:07:00 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(cim1-dev2,52516) 15/01/26 00:07:00 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(cim1-dev2,52516) 15/01/26 00:07:00 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(cim1-dev2,52516) not found 15/01/26 00:07:00 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@7cedcb23 15/01/26 00:07:00 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@7cedcb23 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 15/01/26 00:07:00 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@38e8c534 15/01/26 00:07:00 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@38e8c534 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:310) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 15/01/26 00:07:00 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(cim1-dev,44773) 15/01/26 00:07:00 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(cim1-dev3,29293) 15/01/26 00:07:00 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(cim1-dev3,29293) 15/01/26 00:07:00 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@159adcf5 15/01/26 00:07:00 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@159adcf5 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 15/01/26 00:07:00 INFO ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(cim1-dev,44773) 15/01/26 00:07:00 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(cim1-dev,44773) not found 15/01/26 00:07:00 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@329a6d86 15/01/26 00:07:00 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@329a6d86 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:310) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 15/01/26 00:07:00 INFO ConnectionManager: Key not valid ? sun.nio.ch.SelectionKeyImpl@3d3e86d5 15/01/26 00:07:00 INFO ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@3d3e86d5 java.nio.channels.CancelledKeyException at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:310) at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139) 15/01/26 00:07:01 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped! 15/01/26 00:07:01 INFO ConnectionManager: Selector thread was interrupted! 15/01/26 00:07:01 INFO ConnectionManager: ConnectionManager stopped 15/01/26 00:07:01 INFO MemoryStore: MemoryStore cleared 15/01/26 00:07:01 INFO BlockManager: BlockManager stopped 15/01/26 00:07:01 INFO BlockManagerMaster: BlockManagerMaster stopped 15/01/26 00:07:01 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 15/01/26 00:07:01 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 15/01/26 00:07:01 INFO SparkContext: Successfully stopped SparkContext I've tried to set these 2 options but it doesn't change anything : set("spark.core.connection.ack.wait.timeout","600") set("spark.akka.frameSize","50") Thanks for your help. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/key-already-cancelled-error-tp21357.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
