[ https://issues.apache.org/jira/browse/IGNITE-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Voronkin updated IGNITE-9793: ----------------------------------- Description: There is coordinator and ring of nodes coordinator -> 1 -> 2 - > 3 -> 4 coordinator deactivated: 2018-09-24 15:09:01.609 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:01.620 [DEBUG][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.l.ExchangeLatchManager] Server latch is created [latch=CompletableLatchUid{id='exchange', topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]}, participantsSize=160] 2018-09-24 15:09:01.621 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%] nodes 1, 2, 3, 4 were deactivated: 2018-09-24 15:09:01.609 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:03.328 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=22a58223-47b5-43c2-897b-e70e8e50edf7, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:03.334 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=973eb8ce-3b8c-463d-a6ab-00ac66d93f13, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:03.332 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=a904bac4-aaed-4f69-90f3-c13bc4d331d1, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] Node 2 SEGMENTED 2018-09-24 15:17:50.068 [WARN ][tcp-disco-msg-worker-#2%DPL_GRID%DplGridNodeName%|#2%DPL_GRID%DplGridNodeName%][o.a.i.s.d.tcp.TcpDiscoverySpi] Node is out of topology (probably, due to short-time network problems). 2018-09-24 15:17:50.069 [WARN ][disco-event-worker-#152%DPL_GRID%DplGridNodeName%|#152%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager] Local node SEGMENTED: TcpDiscoveryNode [id=a904bac4-aaed-4f69-90f3-c13bc4d331d1, addrs=ArrayList [10.116.206.98], sockAddrs=HashSet [grid724.domain/10.116.206.98:47500], discPort=47500, order=110, intOrder=110, lastExchangeTime=1537791470063, loc=true, ver=2.5.1#20180906-sha1:ebde6c79, isClient=false] Coordinator started activation on topology without node2 2018-09-24 15:19:48.686 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Start activation process [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, topVer=AffinityTopologyVersion [topVer=188, minorTopVer=1]] But node 3 which is next to node 2 haven't received activation message. Coordinator sent activation to all except 3. Node 3 haven't received activation message. 2018-09-24 15:24:25.911 [INFO ][sys-#28144%DPL_GRID%DplGridNodeName%|#28144%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=188, minorTopVer=1], node=073f1598-6b70-49df-8f45-126735611775, allReceived=false] GridDhtPartitionsExchangeFuture hangs forever. was: There is coordinator and ring of nodes coordinator -> 1 -> 2 - > 3 -> 4 coordinator deactivated: 2018-09-24 15:09:01.609 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:01.620 [DEBUG][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.l.ExchangeLatchManager] Server latch is created [latch=CompletableLatchUid\{id='exchange', topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]}, participantsSize=160] 2018-09-24 15:09:01.621 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%] nodes 1, 2, 3, 4 were deactivated: 2018-09-24 15:09:01.609 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:03.328 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=22a58223-47b5-43c2-897b-e70e8e50edf7, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:03.334 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=973eb8ce-3b8c-463d-a6ab-00ac66d93f13, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] 2018-09-24 15:09:03.332 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Successfully deactivated data structures, services and caches [nodeId=a904bac4-aaed-4f69-90f3-c13bc4d331d1, client=false, topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] Node 2 SEGMENTED 2018-09-24 15:17:50.068 [WARN ][tcp-disco-msg-worker-#2%DPL_GRID%DplGridNodeName%][o.a.i.s.d.tcp.TcpDiscoverySpi] Node is out of topology (probably, due to short-time network problems). 2018-09-24 15:17:50.069 [WARN ][disco-event-worker-#152%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager] Local node SEGMENTED: TcpDiscoveryNode [id=a904bac4-aaed-4f69-90f3-c13bc4d331d1, addrs=ArrayList [10.116.206.98], sockAddrs=HashSet [grid724.domain/10.116.206.98:47500], discPort=47500, order=110, intOrder=110, lastExchangeTime=1537791470063, loc=true, ver=2.5.1#20180906-sha1:ebde6c79, isClient=false] Coordinator started activation on topology without node2 2018-09-24 15:19:48.686 [INFO ][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Start activation process [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, topVer=AffinityTopologyVersion [topVer=188, minorTopVer=1]] But node 3 which is next to node 2 haven't received activation message. Coordinator sent activation to all except 2018-09-24 15:24:25.911 [INFO ][sys-#28144%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=188, minorTopVer=1], node=073f1598-6b70-49df-8f45-126735611775, allReceived=false] GridDhtPartitionsExchangeFuture hangs forever. > Deactivation, segmentation of one node, activation may lead to hang > activation forever > -------------------------------------------------------------------------------------- > > Key: IGNITE-9793 > URL: https://issues.apache.org/jira/browse/IGNITE-9793 > Project: Ignite > Issue Type: Bug > Affects Versions: 2.5 > Reporter: Pavel Voronkin > Priority: Major > > There is coordinator and ring of nodes > coordinator -> 1 -> 2 - > 3 -> 4 > coordinator deactivated: > 2018-09-24 15:09:01.609 [INFO > ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] > Successfully deactivated data structures, services and caches > [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, > topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] > 2018-09-24 15:09:01.620 > [DEBUG][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.l.ExchangeLatchManager] > Server latch is created [latch=CompletableLatchUid{id='exchange', > topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]}, > participantsSize=160] > 2018-09-24 15:09:01.621 [INFO > ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%] > nodes 1, 2, 3, 4 were deactivated: > 2018-09-24 15:09:01.609 [INFO > ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] > Successfully deactivated data structures, services and caches > [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, client=false, > topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] > 2018-09-24 15:09:03.328 [INFO > ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] > Successfully deactivated data structures, services and caches > [nodeId=22a58223-47b5-43c2-897b-e70e8e50edf7, client=false, > topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] > 2018-09-24 15:09:03.334 [INFO > ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] > Successfully deactivated data structures, services and caches > [nodeId=973eb8ce-3b8c-463d-a6ab-00ac66d93f13, client=false, > topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] > 2018-09-24 15:09:03.332 [INFO > ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] > Successfully deactivated data structures, services and caches > [nodeId=a904bac4-aaed-4f69-90f3-c13bc4d331d1, client=false, > topVer=AffinityTopologyVersion [topVer=183, minorTopVer=1]] > Node 2 SEGMENTED > 2018-09-24 15:17:50.068 [WARN > ][tcp-disco-msg-worker-#2%DPL_GRID%DplGridNodeName%|#2%DPL_GRID%DplGridNodeName%][o.a.i.s.d.tcp.TcpDiscoverySpi] > Node is out of topology (probably, due to short-time network problems). > 2018-09-24 15:17:50.069 [WARN > ][disco-event-worker-#152%DPL_GRID%DplGridNodeName%|#152%DPL_GRID%DplGridNodeName%][o.a.i.i.m.d.GridDiscoveryManager] > Local node SEGMENTED: TcpDiscoveryNode > [id=a904bac4-aaed-4f69-90f3-c13bc4d331d1, addrs=ArrayList [10.116.206.98], > sockAddrs=HashSet [grid724.domain/10.116.206.98:47500], discPort=47500, > order=110, intOrder=110, lastExchangeTime=1537791470063, loc=true, > ver=2.5.1#20180906-sha1:ebde6c79, isClient=false] > Coordinator started activation on topology without node2 > 2018-09-24 15:19:48.686 [INFO > ][exchange-worker-#153%DPL_GRID%DplGridNodeName%|#153%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] > Start activation process [nodeId=e002e011-8d1c-4353-a0f3-b71264c5b0f4, > client=false, topVer=AffinityTopologyVersion [topVer=188, minorTopVer=1]] > But node 3 which is next to node 2 haven't received activation message. > Coordinator sent activation to all except 3. > Node 3 haven't received activation message. > 2018-09-24 15:24:25.911 [INFO > ][sys-#28144%DPL_GRID%DplGridNodeName%|#28144%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture] > Coordinator received single message [ver=AffinityTopologyVersion > [topVer=188, minorTopVer=1], node=073f1598-6b70-49df-8f45-126735611775, > allReceived=false] > GridDhtPartitionsExchangeFuture hangs forever. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)