Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

HI Alexey, 

Thank you for the response, 

The below are the details from our Linux env. test :

The log files from the test are attached here : 
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-tp33854p35247.html

Version we use is : 2.8.1

Start type :  a shell script that calls java

size of the server nodes cluster : 3 

The IDE reproducer that i used on my windows env is attached here  ( however
i dont use this on the linux env. On Linux I test against the actual server
cluster ): 
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-tp33854p35021.html

I intend to do some more checks as well in a day or two on this issue. Will
get back on these.

regards,
Veena










--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[alexey.plotnik]

Hi!

So, for now, to be on one line, what are exacts steps you reproduced that?

I mean the following:

1. OS (win or linux)
2. Start type (sh, cmd, Intellij IDEA (run or debug mode)
3. Exact version of ignite
4. Size of cluster
5. configuation files

Great if you have some code-snipped or IDE project as reproducer.

Thanks!





-
Alexey Plotnik
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi all, 

Continuation of this post (
Page 1 :
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-td33854.html

Page 2 :
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-td33854i20.html#a35238
) . Created a new post , since the old post has got paginated for some
reason. 

I checked, this issue is still reproducible in the 2.10 version. 

I noticed that Alexey has created a issue for this : 

https://issues.apache.org/jira/browse/IGNITE-14104

Is there any workaround till this issue is resolved.

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Alexey, 

Please find the debug logs attached. 

This log "Client creation failed" is present in server2.log

regards,
Veena.
serverlogs.zip
  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Sure. Let me get back on this.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[alexey.plotnik]

Could you plz attach DEBUG log of the server node running on Linux machine
from the moment server starts until client re-connect after being stopped?
So, full logs basically



-
Alexey Plotnik
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Alexey,

I am able to reproduce this on my linux env. by doing a kill -9 .  Please
find the logs from my linux env. attached. 
temp.log
  
Also note in our linux env. we have dual ip for our boxes - not sure if that
contributes to this issue. 

Also I do not see this log in the INFO mode in our client logs : 
Sending local node ID (bc6ae633-b643-44ba-afab-321dbf01aaf1 to newly
accepted session ...

I will try shutting down the client with ctrl+c on windows and check if that
helps.

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[alexey.plotnik]

VeenaMithare wrote
> The reproducer code works fine on my home mac ( I dont see the 100s of
> logs)- so not sure if the issue is because of a. Change in OS - at home I
> use mac and office it is windows/linux.b. Change in the network ( between
> my home and office )Any thoughts on how I can debug further  ?--Sent from:
> http://apache-ignite-users.70518.x6.nabble.com/

Hi there.It's reproduced both in Intellij Idea and from cmd on Win 10. It's
NOT reproduced on Linux cli.What happens here: server node does not receive
TcpDiscoveryNodeLeftMessage to let client node left the cluster. Instead
client app is killed, server node understands it and try to reconnect,
client node continues stay in topology and if yo start it in short time you
will see in server logs a lot of handshake errorsand in client log a lot of
You can see client node sends old Node ID.It looks like server node does not
receive "TcpDiscoveryNodeLeftMessage" message from client when you stop the
client app. When stopped from Idea or by closing CMD window the target
process does not receive SIGINT signal.You can try to stop the client app on
windows with "Ctrl-C" shortcut, then client will send left node message, you
will see it in client logs:then in server logs:When using IDEA please try to
run client app in debug mode, it sends SIGINT signal to app when you click
"Stop" button.Anyway when running on Linux I cant reproduce the same
behaviour when stopping the client process with something like "kill
-9".I'll try the situation whne number of client nodes is more than one.



-
Alexey Plotnik
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[alexey.plotnik]

VeenaMithare wrote
> The reproducer code works fine on my home mac ( I dont see the 100s of
> logs)
> - so not sure if the issue is because of 
> a. Change in OS - at home I use mac and office it is windows/linux.
> b. Change in the network ( between my home and office )
> 
> Any thoughts on how I can debug further  ?
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Hi there.

It's reproduced both in Intellij Idea and from cmd on Win 10. It's NOT
reproduced on Linux cli.

What happens here: server node does not receive TcpDiscoveryNodeLeftMessage
to let client node left the cluster. Instead client app is killed, server
node understands this case and try to reconnect, client node continues stay
in topology and if you start it again in short time you will see in server
logs a lot of handshake errors:



and in client log a lot of 



You can see client node sends old Node ID.

It looks like server node does not receive "TcpDiscoveryNodeLeftMessage"
message from client when you stop the client app. When stopped from Idea or
by closing CMD window the target process does not receive SIGINT signal.

You can try to stop the client app on windows with "Ctrl-C" shortcut or run
it in Idea in *debug mode*, then client will send left node message, you
will see it in client logs:



then in server logs:



When using IDEA please try to run client app in debug mode, it sends SIGINT
signal to app when you click "Stop" button so the client app signal handler
is invoked properly.

Anyway when running on Linux I cant reproduce the same behaviour when
stopping the client process with something like "kill -9".

I'll try the situation whne number of client nodes is more than one.





-
Alexey Plotnik
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Unfortunately , will be difficult to try this - since we are working from
home and are connected to office network through citrix . 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[Ilya Kasnacheev]

Hello!

Network may be relevant here.

Can you try disconnect from your network (turn off wi/fi, unplug ethernet)
and re-try?

Regards,
-- 
Ilya Kasnacheev


вт, 19 янв. 2021 г. в 12:45, VeenaMithare :

> The reproducer code works fine on my home mac ( I dont see the 100s of
> logs)
> - so not sure if the issue is because of
> a. Change in OS - at home I use mac and office it is windows/linux.
> b. Change in the network ( between my home and office )
>
> Any thoughts on how I can debug further  ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

The reproducer code works fine on my home mac ( I dont see the 100s of logs)
- so not sure if the issue is because of 
a. Change in OS - at home I use mac and office it is windows/linux.
b. Change in the network ( between my home and office )

Any thoughts on how I can debug further  ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[Ilya Kasnacheev]

Hello!

It's interesting, but I still can't reproduce the issue. Does not seem to
happen to me.

Maybe somebody else will chime in.

Regards,
-- 
Ilya Kasnacheev


чт, 14 янв. 2021 г. в 20:47, VeenaMithare :

> Hi Ilya,
>
> Please find the video recording of the issue I face.
>
>
> https://drive.google.com/file/d/1fDqxmgjiL3o_eenHoUosdiYlMMBJ4xVj/view?usp=sharing
>
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

Please find the video recording of the issue I face.

https://drive.google.com/file/d/1fDqxmgjiL3o_eenHoUosdiYlMMBJ4xVj/view?usp=sharing

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[Ilya Kasnacheev]

Hello!

I am using your reproducer and I did not change dependencies, so I assume
it's on 2.8.1 if your reproducer uses it.

Yes, I'm running both server and client from the reproducer.

Maybe you could record a screencast where you run it and get the result
that you describe?

Regards,
-- 
Ilya Kasnacheev


ср, 13 янв. 2021 г. в 19:14, VeenaMithare :

> Kindly confirm the below :
>
> a. Are you on :2.8.1
> b. Is this cod uncommented :
>  ignite.active(true);
> addPersistentCacheConfiguration(
> ignite);
> c. using both server and client on the reproducer?
>
> I get this on all env. ( even in our linux env. )
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Kindly confirm the below : 

a. Are you on :2.8.1
b. Is this cod uncommented : 
 ignite.active(true);
addPersistentCacheConfiguration(
ignite);
c. using both server and client on the reproducer?

I get this on all env. ( even in our linux env. )



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[Ilya Kasnacheev]

Hello!

Maybe it's some kind of Windows thing or otherwise depends on your
environment? I have tried to run your reproducer, but I never get these
exceptions. Instead, I will only see:

янв 13, 2021 6:38:56 PM org.apache.ignite.logger.java.JavaLogger info
INFO: Accepted incoming communication connection
[locAddr=/0:0:0:0:0:0:0:1:47100, rmtAddr=/0:0:0:0:0:0:0:1:34890]

Once on the server node after client restart.

Regards,
-- 
Ilya Kasnacheev


вт, 12 янв. 2021 г. в 18:11, VeenaMithare :

> This issue is also observed if two different clients exist on the same
> box.
> Steps to reproduce  :
>
> 1. Both the clients are not running
> 2. Start the client1
> 3. Stop the client 1 and start client 2 .
> 4. The huge set of logs are visible on client 2 logs.
>
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

This issue is also observed if two different clients exist on the same box. 
Steps to reproduce  : 

1. Both the clients are not running
2. Start the client1
3. Stop the client 1 and start client 2 . 
4. The huge set of logs are visible on client 2 logs. 

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

Did you get a chance to look in to this ..?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

Please find the reproducer project attached. 

Steps to reproduce : 
1. Run the server.java 
2. Run the ListenerClient.java. Wait till it starts fully.
3. Stop and start the client again ( within the failuredetectiontimeout of
10 secs. )

You will see these logs on the console.

Another thing I noticed today is that , the logs seem to be connected to
this code in Server.java
ignite.active(true);
addPersistentCacheConfiguration(
ignite);

If I comment out the above lines and try the steps above, I dont get those
logs.

regards,
Veena. ClientConnectProject.zip

  







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[Ilya Kasnacheev]

Hello!

We have a lot of tests which do exactly that, and they don't seem to
exhibit that behavior. Please provide a reproducer.

Regards,
-- 
Ilya Kasnacheev


вт, 3 нояб. 2020 г. в 11:13, VeenaMithare :

> Hi Ilya,
>
> This is easy to reproduce. Have a server node and a client node in a
> cluster. Stop and start the client immediately so that the start happens
> within the failure detection timeout ( 10 sec typically ). You will see
> these messages in the client log as it is starting up the second time.
>
> Let me know if you still need a reproducer.
>
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

This is easy to reproduce. Have a server node and a client node in a
cluster. Stop and start the client immediately so that the start happens
within the failure detection timeout ( 10 sec typically ). You will see
these messages in the client log as it is starting up the second time.

Let me know if you still need a reproducer.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[Ilya Kasnacheev]

Hello!

Can you please perhaps share a reproducer project?

Regards,
-- 
Ilya Kasnacheev


вт, 29 сент. 2020 г. в 19:34, VeenaMithare :

> HI Anton,
>
> The setup I am working on is as below :
> server nodes : 3
> client : 1
>
> I get these logs when I start the client node within 10 secs of stopping
> it.
> There is no clue on the server nodes as to why these logs are printed.
>
> It looks as if the client node that was killed, is trying to join back ,
> since it was not marked as DEAD yet by the servers.
>
> It is very easy to reproduce this. Stop and start any client node within 10
> seconds of stopping it.
>
> regards,
> Veena.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

HI Anton,

The setup I am working on is as below : 
server nodes : 3
client : 1

I get these logs when I start the client node within 10 secs of stopping it.
There is no clue on the server nodes as to why these logs are printed. 

It looks as if the client node that was killed, is trying to join back ,
since it was not marked as DEAD yet by the servers.

It is very easy to reproduce this. Stop and start any client node within 10
seconds of stopping it.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[akurbanov]

Hello Veena,

It would make sense to check for the other node logs, because it's the other
node that tries to initiate communication connection with this one.

The clue for the reason why is it trying to connect might be in other node
log.

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/