I encountered a problem when restarting ignite, a node seems to be unable to process the request anymore, because its CacheStore did not process the update operations of database. Here is the log it printed:
2018-12-12 07:06:27:556 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:06:37:557 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:06:47:558 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:06:57:559 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:07:05:177 [db-checkpoint-thread-#47] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=0ms, reason='timeout'] 2018-12-12 07:07:05:565 [grid-timeout-worker-#23] INFO o.a.i.i.IgniteKernal:478 - Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=14135687, uptime=00:09:00.052] ^-- H/N/C [hosts=13, nodes=13, CPUs=44] ^-- CPU [cur=0.2%, avg=37.69%, GC=0%] ^-- PageMemory [pages=324003] ^-- Heap [used=1876MB, free=38.91%, comm=3072MB] ^-- Non heap [used=70MB, free=-1%, comm=72MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=8, qSize=0] 2018-12-12 07:07:05:565 [grid-timeout-worker-#23] INFO o.a.i.i.IgniteKernal:478 - FreeList [name=null, buckets=256, dataPages=204429, reusePages=0] 2018-12-12 07:07:07:559 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:07:17:560 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:07:27:561 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:07:37:562 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:07:47:563 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:07:57:563 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] 2018-12-12 07:08:05:574 [grid-timeout-worker-#23] INFO o.a.i.i.IgniteKernal:478 - Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=14135687, uptime=00:10:00.054] ^-- H/N/C [hosts=13, nodes=13, CPUs=44] ^-- CPU [cur=0.17%, avg=36.59%, GC=0%] ^-- PageMemory [pages=324003] ^-- Heap [used=1878MB, free=38.85%, comm=3072MB] ^-- Non heap [used=70MB, free=-1%, comm=72MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=8, qSize=0] 2018-12-12 07:08:05:575 [grid-timeout-worker-#23] INFO o.a.i.i.IgniteKernal:478 - FreeList [name=null, buckets=256, dataPages=204429, reusePages=0] 2018-12-12 07:08:07:564 [exchange-worker-#42] WARN o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await partitions release latch within timeout: ClientLatch [coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c, addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]] In addition, other nodes also printed some logs. Please review the attachment. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/