Hello!
We need to see all the lines after the following (but before the next
Thread):
Line 41312: [2020-11-09T06:44:13,606][WARN
][tcp-disco-msg-worker-#2%EDIFCustomerCC%][G] Thread
[name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=TIMED_WAITING,
blockCnt=16, waitCnt=2469710]
Line 43302: Thread
[name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=RUNNABLE,
blockCnt=16, waitCnt=2469710]
Line 47326: [2020-11-09T10:55:18,888][WARN
][sys-stripe-118-#119%EDIFCustomerCC%][G] Thread
[name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=TIMED_WAITING,
blockCnt=16, waitCnt=2469961]
Line 49473: Thread
[name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=TIMED_WAITING,
blockCnt=16, waitCnt=2469961]
Regards,
--
Ilya Kasnacheev
чт, 12 нояб. 2020 г. в 19:57, Gangaiah Gundeboina <gangaiah.ya...@gmail.com
>:
> HI Ilya Kasnacheev,
>
> Below are log entries with the tread name 'partition-exchanger'
>
> ############################
>
> Line 41311:
>
> [2020-11-09T06:44:13,605][ERROR][tcp-disco-msg-worker-#2%EDIFCustomerCC%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=partition-exchanger,
> blockedFor=60s]
> Line 41315:
> [2020-11-09T06:44:13,606][ERROR][tcp-disco-msg-worker-#2%EDIFCustomerCC%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=EDIFCustomerCC,
> finished=false, heartbeatTs=1604884393601]]]
> Line 41316: org.apache.ignite.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=EDIFCustomerCC,
> finished=false, heartbeatTs=1604884393601]
> Line 47325:
> [2020-11-09T10:55:18,888][ERROR][sys-stripe-118-#119%EDIFCustomerCC%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=partition-exchanger,
> blockedFor=60s]
> Line 47329:
> [2020-11-09T10:55:18,889][ERROR][sys-stripe-118-#119%EDIFCustomerCC%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=EDIFCustomerCC,
> finished=false, heartbeatTs=1604899458881]]]
> Line 47330: org.apache.ignite.IgniteException: GridWorker
> [name=partition-exchanger, igniteInstanceName=EDIFCustomerCC,
> finished=false, heartbeatTs=1604899458881]
> #################################################
>
>
> Below is the stack trace for 'exchange-worker-#344' worker thread,
>
> #################################################
> Line 41109: [2020-11-09T04:11:42,068][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridCachePartitionExchangeManager]
> Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
> [topVer=2791, minorTopVer=0], force=false, evt=NODE_JOINED,
> node=f11cfea2-5ece-4867-b276-e94fa8458f47]
> Line 41116: [2020-11-09T04:11:52,060][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][time] Started exchange init
> [topVer=AffinityTopologyVersion [topVer=2792, minorTopVer=0],
> mvccCrd=MvccCoordinator [nodeId=08260e5f-ae8d-44f2-b10a-dc3490133ee8,
> crdVer=1602946449957, topVer=AffinityTopologyVersion [topVer=6,
> minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED,
> evtNode=5c548b34-defa-4fcd-9bc7-364a4fbec8da, customEvt=null,
> allowMerge=true]
> Line 41117: [2020-11-09T04:11:52,062][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridDhtPartitionsExchangeFuture]
> Finish exchange future [startVer=AffinityTopologyVersion [topVer=2792,
> minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2792,
> minorTopVer=0],
> err=null]
> Line 41118: [2020-11-09T04:11:52,066][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridDhtPartitionsExchangeFuture]
> Completed partition exchange
> [localNode=30b55ea5-18a4-4c15-b45f-7fe420ac00bd,
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
> [topVer=2792, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode
> [id=5c548b34-defa-4fcd-9bc7-364a4fbec8da, addrs=[0:0:0:0:0:0:0:1%lo,
> AA.BB.CC.DD, 127.0.0.1, 2405:200:a10:fc05:a2d3:c1ff:fef2:25a0%eth0],
> sockAddrs=[0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
> 2405:200:a10:fc05:a2d3:c1ff:fef2:25a0%eth0:0, AA.BB.CC.DD/AA.BB.CC.DD:0],
> discPort=0, order=2792, intOrder=1527, lastExchangeTime=1604875311250,
> loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], done=true],
> topVer=AffinityTopologyVersion [topVer=2792, minorTopVer=0],
> durationFromInit=0]
> Line 41119: [2020-11-09T04:11:52,067][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][time] Finished exchange init
> [topVer=AffinityTopologyVersion [topVer=2792, minorTopVer=0], crd=false]
> Line 41120: [2020-11-09T04:11:52,072][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridCachePartitionExchangeManager]
> Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
> [topVer=2792, minorTopVer=0], force=false, evt=NODE_JOINED,
> node=5c548b34-defa-4fcd-9bc7-364a4fbec8da]
> Line 41124: [2020-11-09T04:11:52,451][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][time] Started exchange init
> [topVer=AffinityTopologyVersion [topVer=2793, minorTopVer=0],
> mvccCrd=MvccCoordinator [nodeId=08260e5f-ae8d-44f2-b10a-dc3490133ee8,
> crdVer=1602946449957, topVer=AffinityTopologyVersion [topVer=6,
> minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED,
> evtNode=076584a1-154f-4e3c-8f45-6ee096a71868, customEvt=null,
> allowMerge=true]
> Line 41125: [2020-11-09T04:11:52,453][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridDhtPartitionsExchangeFuture]
> Finish exchange future [startVer=AffinityTopologyVersion [topVer=2793,
> minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2793,
> minorTopVer=0],
> err=null]
> Line 41126: [2020-11-09T04:11:52,458][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridDhtPartitionsExchangeFuture]
> Completed partition exchange
> [localNode=30b55ea5-18a4-4c15-b45f-7fe420ac00bd,
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
> [topVer=2793, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode
> [id=076584a1-154f-4e3c-8f45-6ee096a71868, addrs=[0:0:0:0:0:0:0:1%lo,
> AA.BB.CC.DD, 127.0.0.1, 2405:200:a10:fc05:a2d3:c1ff:fef2:25a0%eth0],
> sockAddrs=[0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
> 2405:200:a10:fc05:a2d3:c1ff:fef2:25a0%eth0:0, AA.BB.CC.DD/AA.BB.CC.DD:0],
> discPort=0, order=2793, intOrder=1528, lastExchangeTime=1604875311403,
> loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=true], done=true],
> topVer=AffinityTopologyVersion [topVer=2793, minorTopVer=0],
> durationFromInit=10]
> Line 41127: [2020-11-09T04:11:52,458][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][time] Finished exchange init
> [topVer=AffinityTopologyVersion [topVer=2793, minorTopVer=0], crd=false]
> Line 41128: [2020-11-09T04:11:52,472][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridCachePartitionExchangeManager]
> Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
> [topVer=2793, minorTopVer=0], force=false, evt=NODE_JOINED,
> node=076584a1-154f-4e3c-8f45-6ee096a71868]
> Line 41136: [2020-11-09T04:12:13,569][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][time] Started exchange init
> [topVer=AffinityTopologyVersion [topVer=2794, minorTopVer=0],
> mvccCrd=MvccCoordinator [nodeId=08260e5f-ae8d-44f2-b10a-dc3490133ee8,
> crdVer=1602946449957, topVer=AffinityTopologyVersion [topVer=6,
> minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED,
> evtNode=5ee5d2c6-76d6-4aba-a160-d084d3ad9e21, customEvt=null,
> allowMerge=true]
> Line 41137: [2020-11-09T04:12:13,571][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridDhtPartitionsExchangeFuture]
> Finish exchange future [startVer=AffinityTopologyVersion [topVer=2794,
> minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2794,
> minorTopVer=0],
> err=null]
> Line 41138: [2020-11-09T04:12:13,575][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridDhtPartitionsExchangeFuture]
> Completed partition exchange
> [localNode=30b55ea5-18a4-4c15-b45f-7fe420ac00bd,
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
> [topVer=2794, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode
> [id=5ee5d2c6-76d6-4aba-a160-d084d3ad9e21, addrs=[0:0:0:0:0:0:0:1%lo,
> AA.BB.CC.DD, 127.0.0.1], sockAddrs=[AA.BB.CC.DD/AA.BB.CC.DD:0,
> 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0], discPort=0, order=2794,
> intOrder=1529,
> lastExchangeTime=1604875332557, loc=false,
> ver=2.7.6#20190911-sha1:21f7ca41,
> isClient=true], done=true], topVer=AffinityTopologyVersion [topVer=2794,
> minorTopVer=0], durationFromInit=0]
> Line 41139: [2020-11-09T04:12:13,576][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][time] Finished exchange init
> [topVer=AffinityTopologyVersion [topVer=2794, minorTopVer=0], crd=false]
> Line 41140: [2020-11-09T04:12:13,586][INFO
> ][exchange-worker-#344%EDIFCustomerCC%][GridCachePartitionExchangeManager]
> Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
> [topVer=2794, minorTopVer=0], force=false, evt=NODE_JOINED,
> node=5ee5d2c6-76d6-4aba-a160-d084d3ad9e21]
> Line 41312: [2020-11-09T06:44:13,606][WARN
> ][tcp-disco-msg-worker-#2%EDIFCustomerCC%][G] Thread
> [name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=TIMED_WAITING,
> blockCnt=16, waitCnt=2469710]
> Line 43302: Thread
> [name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=RUNNABLE,
> blockCnt=16, waitCnt=2469710]
> Line 47326: [2020-11-09T10:55:18,888][WARN
> ][sys-stripe-118-#119%EDIFCustomerCC%][G] Thread
> [name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=TIMED_WAITING,
> blockCnt=16, waitCnt=2469961]
> Line 49473: Thread
> [name="exchange-worker-#344%EDIFCustomerCC%", id=391, state=TIMED_WAITING,
> blockCnt=16, waitCnt=2469961]
>
> Thanks and Regards,
> Gangaiah
>
>
>
> -----
> Thanks and Regards,
> Gangaiah
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
