Re: Too many TCP discovery accepted incoming connections

2021-05-05 Thread VeenaMithare 
Hi Ilya, 


I was wondering if it was a case of Node1 trying to connect to the old node
id of Node2 , but the Node2 has got a new node id since it was last
connected.  ( That is the issue we faced in that link shared ) . 

Maybe that is not what is happening here. 
Please ignore if so,

regards,
Veena.








--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Too many TCP discovery accepted incoming connections

2021-05-04 Thread Ilya Kasnacheev 
Hello!

I'm not sure. Why?

Regards,
-- 
Ilya Kasnacheev


вс, 2 мая 2021 г. в 11:13, VeenaMithare :

> Hi Ilya,
>
> Could the issue be as mentioned here :
>
>
> http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-tp33854p35224.html
>
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Too many TCP discovery accepted incoming connections

2021-05-02 Thread VeenaMithare 
Hi Ilya, 

Could the issue be as mentioned here : 

http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-tp33854p35224.html

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Too many TCP discovery accepted incoming connections

2021-04-28 Thread Ilya Kasnacheev 
Hello!

Please consider the following messages:

[2021-04-21T14:55:09,203][WARN
][tcp-comm-worker-#1%EDIFCustomer%][TcpCommunicationSpi] Connect timed out
(consider increasing 'failureDetectionTimeout' configuration property)
[addr=/10.40.0.78:47100, failureDetectionTimeout=6]
[2021-04-21T14:55:09,203][WARN
][tcp-comm-worker-#1%EDIFCustomer%][TcpCommunicationSpi] Failed to connect
to a remote node (make sure that destination node is alive and operating
system firewall is disabled on local and remote hosts) [addrs=[/
10.40.0.78:47100, /127.0.0.1:47100]]

I can see that communication threads will spend a lot of time on connect(),
indicating network or firewall issues:
Thread [name="tcp-comm-worker-#1%EDIFCustomer%", id=365, state=RUNNABLE,
blockCnt=1294, waitCnt=12569]
at sun.nio.ch.Net.poll(Native Method)
at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
- locked java.lang.Object@65ec5b09
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:110)
- locked java.lang.Object@9ecd49c
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3299)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.access$6000(TcpCommunicationSpi.java:271)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4489)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4294)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

I think this is the root cause. Your server node cannot connect to some of
your remaining nodes' communication port. Maybe your server node is behind
NAT or firewall. Consider enabling NAT traversal feature:
https://ignite.apache.org/docs/latest/clustering/running-client-nodes-behind-nat

Regards,
-- 
Ilya Kasnacheev


чт, 22 апр. 2021 г. в 21:58, Gangaiah Gundeboina :

> HI Ilya,
>
> Please find attached full log file.
>
> Regards,
> Gangaiah
>
> server_log.zip
> 
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Too many TCP discovery accepted incoming connections

2021-04-22 Thread Gangaiah Gundeboina 
HI Ilya,

Please find attached full log file.

Regards,
Gangaiah

server_log.zip
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Too many TCP discovery accepted incoming connections

2021-04-22 Thread Ilya Kasnacheev 
Hello!

I can see that a node has just joined, but I'm not sure why there are all
those messages.

However, the log file seems truncated. Can you provide the rest?

Regards,
-- 
Ilya Kasnacheev


чт, 22 апр. 2021 г. в 16:47, Gangaiah Gundeboina :

> Hi Ilya,
>
>  There is no jvm pause, only one entry in the server log but  that too
> after
> affect. The incoming connections started at 2021-04-21T14:52:50,035 and jvm
> pause could see at 2021-04-21T14:54:18. I have attached logs file, could
> you
> please check.
>
> [2021-04-21T14:54:18,035][WARN
> ][jvm-pause-detector-worker][IgniteKernal%EDIFCustomer] Possible too long
> JVM pause: 2020 milliseconds.
>
>
> server_log.zip
> 
>
>
> Regards,
> Gangaiah
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Too many TCP discovery accepted incoming connections

2021-04-22 Thread Gangaiah Gundeboina 
Hi Ilya,

 There is no jvm pause, only one entry in the server log but  that too after
affect. The incoming connections started at 2021-04-21T14:52:50,035 and jvm
pause could see at 2021-04-21T14:54:18. I have attached logs file, could you
please check.

[2021-04-21T14:54:18,035][WARN
][jvm-pause-detector-worker][IgniteKernal%EDIFCustomer] Possible too long
JVM pause: 2020 milliseconds.


server_log.zip
  

Regards,
Gangaiah



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Too many TCP discovery accepted incoming connections

2021-04-22 Thread Ilya Kasnacheev 
Hello!

Can you share complete logs from that node? You can also search them for
"JVM pause" messages or just for any reason why these connections were
closed in the first place.

Regards,
-- 
Ilya Kasnacheev


чт, 22 апр. 2021 г. в 14:39, Gangaiah Gundeboina :

> Hi Igniters,
>
> Some times we are seeing too many 'tcp discovery accepted incoming
> connections' from client servers in server logs. Is it due to network
> glitch
> or connectivity break between servers and clients?. This leads to system
> critical errors and many clients disconnected and reconnected. During the
> issue time cluster is not responding for some time around 5 mints and
> recovered it self. We haven't put any network and jointimeouts at discovery
> level, using default time outs.. Cluster should be respond for other
> clients, do we need to increase the timeouts. Could you please help us,
> below are few entries in server log.
>
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
> incoming connection [rmtAddr=/CIPS8.152, rmtPort=48986]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning
> a
> new thread for connection [rmtAddr=/CIPS8.152, rmtPort=48986]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
> incoming connection [rmtAddr=/CIPS3.29, rmtPort=52172]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning
> a
> new thread for connection [rmtAddr=/CIPS3.29, rmtPort=52172]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-sock-reader-#58331%EDIFCustomer%][TcpDiscoverySpi] Started
> serving remote node connection [rmtAddr=/CIPS8.152:48986, rmtPort=48986]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
> incoming connection [rmtAddr=/CIPS4.29, rmtPort=35815]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning
> a
> new thread for connection [rmtAddr=/CIPS4.29, rmtPort=35815]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-sock-reader-#58332%EDIFCustomer%][TcpDiscoverySpi] Started
> serving remote node connection [rmtAddr=/CIPS3.29:52172, rmtPort=52172]
> [2021-04-21T14:52:54,697][INFO
> ][tcp-disco-sock-reader-#58333%EDIFCustomer%][TcpDiscoverySpi] Started
> serving remote node connection [rmtAddr=/CIPS4.29:35815, rmtPort=35815]
> [2021-04-21T14:52:54,698][INFO
> ][tcp-disco-sock-reader-#47336%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS3.29:56843, rmtPort=56843
> [2021-04-21T14:52:54,751][INFO
> ][tcp-disco-sock-reader-#47340%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS6.107:55739, rmtPort=55739
> [2021-04-21T14:52:54,755][INFO
> ][tcp-disco-sock-reader-#38863%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS2.48:25981, rmtPort=25981
> [2021-04-21T14:52:54,797][INFO
> ][tcp-disco-sock-reader-#47296%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS7.31:43122, rmtPort=43122
> [2021-04-21T14:52:54,904][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
> incoming connection [rmtAddr=/CIPS2.45, rmtPort=39922]
> [2021-04-21T14:52:54,904][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning
> a
> new thread for connection [rmtAddr=/CIPS2.45, rmtPort=39922]
> [2021-04-21T14:52:54,904][INFO
> ][tcp-disco-sock-reader-#58337%EDIFCustomer%][TcpDiscoverySpi] Started
> serving remote node connection [rmtAddr=/CIPS2.45:39922, rmtPort=39922]
> [2021-04-21T14:52:55,040][INFO
> ][tcp-disco-sock-reader-#47606%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS2.155:38128, rmtPort=38128
> [2021-04-21T14:52:55,261][INFO
> ][tcp-disco-sock-reader-#47254%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS3.44:44047, rmtPort=44047
> [2021-04-21T14:52:55,392][INFO
> ][tcp-disco-sock-reader-#47342%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS4.22:55079, rmtPort=55079
> [2021-04-21T14:52:55,433][INFO
> ][tcp-disco-sock-reader-#47302%EDIFCustomer%][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/CIPS7.44:49161, rmtPort=49161
> [2021-04-21T14:52:55,477][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
> incoming connection [rmtAddr=/CIPS2.57, rmtPort=57963]
> [2021-04-21T14:52:55,477][INFO
> ][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery s

Too many TCP discovery accepted incoming connections

2021-04-22 Thread Gangaiah Gundeboina 
Hi Igniters,

Some times we are seeing too many 'tcp discovery accepted incoming
connections' from client servers in server logs. Is it due to network glitch
or connectivity break between servers and clients?. This leads to system
critical errors and many clients disconnected and reconnected. During the
issue time cluster is not responding for some time around 5 mints and
recovered it self. We haven't put any network and jointimeouts at discovery
level, using default time outs.. Cluster should be respond for other
clients, do we need to increase the timeouts. Could you please help us,
below are few entries in server log.

[2021-04-21T14:52:54,697][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
incoming connection [rmtAddr=/CIPS8.152, rmtPort=48986]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning a
new thread for connection [rmtAddr=/CIPS8.152, rmtPort=48986]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
incoming connection [rmtAddr=/CIPS3.29, rmtPort=52172]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning a
new thread for connection [rmtAddr=/CIPS3.29, rmtPort=52172]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-sock-reader-#58331%EDIFCustomer%][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/CIPS8.152:48986, rmtPort=48986]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
incoming connection [rmtAddr=/CIPS4.29, rmtPort=35815]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning a
new thread for connection [rmtAddr=/CIPS4.29, rmtPort=35815]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-sock-reader-#58332%EDIFCustomer%][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/CIPS3.29:52172, rmtPort=52172]
[2021-04-21T14:52:54,697][INFO
][tcp-disco-sock-reader-#58333%EDIFCustomer%][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/CIPS4.29:35815, rmtPort=35815]
[2021-04-21T14:52:54,698][INFO
][tcp-disco-sock-reader-#47336%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS3.29:56843, rmtPort=56843
[2021-04-21T14:52:54,751][INFO
][tcp-disco-sock-reader-#47340%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS6.107:55739, rmtPort=55739
[2021-04-21T14:52:54,755][INFO
][tcp-disco-sock-reader-#38863%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS2.48:25981, rmtPort=25981
[2021-04-21T14:52:54,797][INFO
][tcp-disco-sock-reader-#47296%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS7.31:43122, rmtPort=43122
[2021-04-21T14:52:54,904][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
incoming connection [rmtAddr=/CIPS2.45, rmtPort=39922]
[2021-04-21T14:52:54,904][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning a
new thread for connection [rmtAddr=/CIPS2.45, rmtPort=39922]
[2021-04-21T14:52:54,904][INFO
][tcp-disco-sock-reader-#58337%EDIFCustomer%][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/CIPS2.45:39922, rmtPort=39922]
[2021-04-21T14:52:55,040][INFO
][tcp-disco-sock-reader-#47606%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS2.155:38128, rmtPort=38128
[2021-04-21T14:52:55,261][INFO
][tcp-disco-sock-reader-#47254%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS3.44:44047, rmtPort=44047
[2021-04-21T14:52:55,392][INFO
][tcp-disco-sock-reader-#47342%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS4.22:55079, rmtPort=55079
[2021-04-21T14:52:55,433][INFO
][tcp-disco-sock-reader-#47302%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS7.44:49161, rmtPort=49161
[2021-04-21T14:52:55,477][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
incoming connection [rmtAddr=/CIPS2.57, rmtPort=57963]
[2021-04-21T14:52:55,477][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery spawning a
new thread for connection [rmtAddr=/CIPS2.57, rmtPort=57963]
[2021-04-21T14:52:55,477][INFO
][tcp-disco-sock-reader-#58339%EDIFCustomer%][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/CIPS2.57:57963, rmtPort=57963]
[2021-04-21T14:52:55,596][INFO
][tcp-disco-sock-reader-#34121%EDIFCustomer%][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/CIPS6.100:40969, rmtPort=40969
[2021-04-21T14:52:55,608][INFO
][tcp-disco-srvr-#3%EDIFCustomer%][TcpDiscoverySpi] TCP discovery accepted
incoming connection [rmtAddr=/CIPS6.101, rmtPort=50848]







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/