Hello,
Could you please attach additional logs from a coordinator node? An id of
that node you may notice in "Unable to await partitions release latch"
message.
Also, it would be good to have logs from the client machine and from any
other server node in the cluster.
пн, 24 дек. 2018 г. в 09:13, aMark <feku.fa...@gmail.com>:
> Hi,
>
> We are using Ignite 2.6 as persistent store in Partitioned Mode having 6
> cluster node running, each node is running on different machine.
>
> We have noticed that on all the server nodes were trying to rebalance due
> to
> 'too many dirty pages':
> 2018-12-22 14:56:17,161 INFO
> [db-checkpoint-thread-#104%d66a2109-94b4-4eb3-bb3c-e611aa842a2a%] {}
>
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager
> - Checkpoint started [
> checkpointId=cabfb20d-d53d-4dd5-8e2b-d270f062c010, startPtr=FileWALPointer
> [idx=5025, fileOff=65789181, len=5487], checkpointLockWait=34ms,
> checkpointLockHoldTime=20ms, walCpRecordFsyncDuration=68ms, pages=1
> 820016, reason='too many dirty pages']
>
> then I can see following log after a minute:
> 2018-12-22 14:57:26,040 WARN
> [exchange-worker-#102%d66a2109-94b4-4eb3-bb3c-e611aa842a2a%] {}
>
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
> - Unable to await partitions release latch within timeout: ClientLatch
>
>
> After this cluster became unresponsive for 30 minutes and timeout started
> happening while writing to cluster at the client end.
> But at the same time number of ignite related objects increased by many
> threshold over the time.
>
> Following is histogram from one of the client machine :
> 1: 675519770 16212474480
> org.apache.ignite.internal.util.future.GridFutureAdapter$Node
> 2: 6637390 3837988712 [Ljava.lang.Object;
> 3: 6557471 262298840
> org.apache.ignite.internal.util.future.GridCompoundFuture
> 4: 6627708 159064992 java.util.ArrayList
> 5: 177242 36609304 [B
>
> And eventually client machine went in Full GC mode.
> Following is the code to write in the ignite cache :
> try(IgniteDataStreamer<BenchmarkCacheStoreKey, OptionalDouble> streamer =
> ignite.dataStreamer(cacheName)){
> igniteMap.forEach((key,value) -> streamer.addData(key, value));
>
>
> }catch(CacheException|IgniteInterruptedException|IllegalStateException|IgniteDataStreamerTimeoutException
> e){
> ignite.log().error("Entries not written to Ignite Cache, please
> check the logs.");
> throw new IgniteException(e);
> }
>
>
> Any help will be much appreciated.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
