Re: ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId

[Spidy]

Hi,

Can you please explain which settings did you changed?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/ERROR-ConnectionManager-Corresponding-SendingConnection-to-ConnectionManagerId-tp17050p21035.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId

[Aaron Davidson]

Do note that this problem may be fixed in Spark 1.2, as we changed the
default transfer service to use a Netty-based one rather than the
ConnectionManager.

On Thu, Jan 8, 2015 at 7:05 AM, Spidy yoni...@gmail.com wrote:

 Hi,

 Can you please explain which settings did you changed?



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/ERROR-ConnectionManager-Corresponding-SendingConnection-to-ConnectionManagerId-tp17050p21035.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org

Re: ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId

[Tom Seddon]

Yes please can you share.  I am getting this error after expanding my
application to include a large broadcast variable. Would be good to know if
it can be fixed with configuration.

On 23 October 2014 18:04, Michael Campbell michael.campb...@gmail.com
wrote:

 Can you list what your fix was so others can benefit?

 On Wed, Oct 22, 2014 at 8:15 PM, arthur.hk.c...@gmail.com 
 arthur.hk.c...@gmail.com wrote:

 Hi,

 I have managed to resolve it because a wrong setting. Please ignore this .

 Regards
 Arthur

 On 23 Oct, 2014, at 5:14 am, arthur.hk.c...@gmail.com 
 arthur.hk.c...@gmail.com wrote:


 14/10/23 05:09:04 WARN ConnectionManager: All connections not cleaned up

ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId not found

[Dai, Kevin]

Hi, all

My job failed and there are a lot of  ERROR ConnectionManager: Corresponding 
SendingConnection to ConnectionManagerId not found information in the log.

Can anyone tell me what's wrong and how to fix it?

Best Regards,
Kevin.

ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId

[arthur.hk.c...@gmail.com]

Hi,

I just tried sample PI calculation on Spark Cluster, after returning the Pi 
result, it shows ERROR ConnectionManager: Corresponding SendingConnection to 
ConnectionManagerId(m37,35662) not found

./bin/spark-submit   --class org.apache.spark.examples.SparkPi   --master 
spark://m33:7077   --executor-memory 512m  --total-executor-cores 40  
examples/target/spark-examples_2.10-1.1.0.jar 100

14/10/23 05:09:03 INFO TaskSetManager: Finished task 87.0 in stage 0.0 (TID 87) 
in 346 ms on m134.emblocsoft.net (99/100)
14/10/23 05:09:03 INFO TaskSetManager: Finished task 98.0 in stage 0.0 (TID 98) 
in 262 ms on m134.emblocsoft.net (100/100)
14/10/23 05:09:03 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have 
all completed, from pool 
14/10/23 05:09:03 INFO DAGScheduler: Stage 0 (reduce at SparkPi.scala:35) 
finished in 2.597 s
14/10/23 05:09:03 INFO SparkContext: Job finished: reduce at SparkPi.scala:35, 
took 2.725328861 s
Pi is roughly 3.1414948
14/10/23 05:09:03 INFO SparkUI: Stopped Spark web UI at http://m33:4040
14/10/23 05:09:03 INFO DAGScheduler: Stopping DAGScheduler
14/10/23 05:09:03 INFO SparkDeploySchedulerBackend: Shutting down all executors
14/10/23 05:09:03 INFO SparkDeploySchedulerBackend: Asking each executor to 
shut down
14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? 
sun.nio.ch.SelectionKeyImpl@37852165
14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to 
ConnectionManagerId(m37,35662)
14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to 
ConnectionManagerId(m37,35662)
14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to 
ConnectionManagerId(m37,35662) not found
14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? 
sun.nio.ch.SelectionKeyImpl@37852165
java.nio.channels.CancelledKeyException
at 
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at 
org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to 
ConnectionManagerId(m36,34230)
14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to 
ConnectionManagerId(m35,50371)
14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to 
ConnectionManagerId(m36,34230)
14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to 
ConnectionManagerId(m34,41562)
14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to 
ConnectionManagerId(m36,34230) not found
14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to 
ConnectionManagerId(m35,50371)
14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to 
ConnectionManagerId(m35,50371) not found
14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to 
ConnectionManagerId(m33,39517)
14/10/23 05:09:04 INFO ConnectionManager: Removing ReceivingConnection to 
ConnectionManagerId(m33,39517)
14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to 
ConnectionManagerId(m34,41562)
14/10/23 05:09:04 ERROR ConnectionManager: Corresponding SendingConnection to 
ConnectionManagerId(m33,39517) not found
14/10/23 05:09:04 ERROR SendingConnection: Exception while reading 
SendingConnection to ConnectionManagerId(m34,41562)
java.nio.channels.ClosedChannelException
at 
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at 
org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/10/23 05:09:04 INFO ConnectionManager: Handling connection error on 
connection to ConnectionManagerId(m34,41562)
14/10/23 05:09:04 INFO ConnectionManager: Removing SendingConnection to 
ConnectionManagerId(m34,41562)
14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? 
sun.nio.ch.SelectionKeyImpl@2e0b5c4a
14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? 
sun.nio.ch.SelectionKeyImpl@2e0b5c4a
java.nio.channels.CancelledKeyException
at 
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at 
org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/10/23 05:09:04 INFO ConnectionManager: Key not valid ? 
sun.nio.ch.SelectionKeyImpl@653f8844
14/10/23 05:09:04 INFO ConnectionManager: key already cancelled ? 
sun.nio.ch.SelectionKeyImpl@653f8844
java.nio.channels.CancelledKeyException
at 
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at 
org.apache.spark.network.ConnectionManager$$anon$4.run

Re: ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId

[arthur.hk.c...@gmail.com]

Hi,

I have managed to resolve it because a wrong setting. Please ignore this .

Regards
Arthur

On 23 Oct, 2014, at 5:14 am, arthur.hk.c...@gmail.com 
arthur.hk.c...@gmail.com wrote:

 
 14/10/23 05:09:04 WARN ConnectionManager: All connections not cleaned up