Re: partition-exchanger system-critical thread blocked

Hello! I don't think running client nodes in kubernetes is supported. You can only use thin client inside k8s if your server nodes are outside. Regards, -- Ilya Kasnacheev вт, 15 дек. 2020 г. в 20:25, Gangaiah Gundeboina : > Hi Ilya, > > Our server nodes not running in kubernetes, so we have

Re: partition-exchanger system-critical thread blocked

Hi Ilya, Our server nodes not running in kubernetes, so we have simple ignite cluster no relation to kubernetes. But we have few clients which running inside kubernetes cluster and few clients are out side kubernets, both are connecting to same ignite cluster. - Thanks and Regards, Gangai

Re: partition-exchanger system-critical thread blocked

Hello! I think it is only available in 2.9.0. Prior to 2.9.0, clients outside of kubernetes may not connect to k8s cluster. Regards, -- Ilya Kasnacheev вт, 15 дек. 2020 г. в 14:55, Gangaiah Gundeboina : > Hi Ilya, > > We are using 2.7.6 version, is it available in the version. > > And also,

Re: partition-exchanger system-critical thread blocked

Hi Ilya, We are using 2.7.6 version, is it available in the version. And also, the clients which are running in kubernetes able to connect to the cluster. Sometimes facing the issue whenever we restart the multiple clients/adding multiple clients to cluster and suddenly sometimes the error printi

Re: partition-exchanger system-critical thread blocked

Hi Ilya, We are using 2.7.6 version, is it available in the version. Thanks and Regards, Gangaiah - Thanks and Regards, Gangaiah -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: partition-exchanger system-critical thread blocked

Hello! Thread [name="tcp-comm-worker-#1%EDIFCustomerCC%", id=369, state=RUNNABLE, blockCnt=1570, waitCnt=8053] at sun.nio.ch.Net.poll(Native Method) at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954) - locked java.lang.Object@77988a56 at sun.nio.ch.Soc

Re: partition-exchanger system-critical thread blocked

Hi Ilya, We have configured atomictyMode 'ATOMIC' for all caches it's not transactional, below is our cache configuration for one of the caches. ### Cache Configuration I ha

Re: partition-exchanger system-critical thread blocked

Hello! Can you provide stack trace from that thread? No, this is a different thread than exchange worker. I don't think that cluster is supposed to fail clients in this case. We have timeout for transactions waiting on PME now. Regards, -- Ilya Kasnacheev сб, 12 дек. 2020 г. в 10:23, Gangaia

Re: partition-exchanger system-critical thread blocked

Hi Ilya, Again we got same issue 'tcp-comm-worker-#1' blocked. We have observed one thing whenever multiple clients joining to cluster at same time, the error is printing. I could see there is only one thread(tcp-comm-worker), checked from logs and also from visualvm. >From documentation also, c

Re: partition-exchanger system-critical thread blocked

Hi Ilya, Regarding exchange worker, the jira saying that 2.8.1 and 2.9 but we are using 2.7.6 version. So it could be in 2.7.6 as well. But still I am not understanding here is, there is no node join/left or cluster activation or cache creation near to the error logging time. Don't know why PME tr

Re: partition-exchanger system-critical thread blocked

Hello! For the first 4 thread dump, the problem was in establishing communication connection to one of nodes: [2020-11-09T02:31:41,105][WARN ][tcp-comm-worker-#1%EDIFCustomerCC%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/k

Re: partition-exchanger system-critical thread blocked

Hi Ilya, Please find attached entire log file. Regards, Gangaiah 18.zip - Thanks and Regards, Gangaiah -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: partition-exchanger system-critical thread blocked

Hello! We need to see all the lines after the following (but before the next Thread): Line 41312: [2020-11-09T06:44:13,606][WARN ][tcp-disco-msg-worker-#2%EDIFCustomerCC%][G] Thread [name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=TIMED_WAITING, blockCnt=16, waitCnt=246

Re: partition-exchanger system-critical thread blocked

HI Ilya Kasnacheev, Below are log entries with the tread name 'partition-exchanger' Line 41311: [2020-11-09T06:44:13,605][ERROR][tcp-disco-msg-worker-#2%EDIFCustomerCC%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined beh

Re: partition-exchanger system-critical thread blocked

Hello! Can you please find an actual stack trace from partition-exchanger thread in that log? One that starts with Thread [name="partition-exchanger" ? Regards, -- Ilya Kasnacheev вт, 10 нояб. 2020 г. в 15:54, Kamlesh Joshi : > Hi Igniters, > > > > Sometimes 'Blocked system-critical thread'