Hi,
Throttling is disabled in ignite config as mentioned in prev reply. What do
you suggest to make ignite catchup with SSD limits on checkpointing.
On Mon, 12 Sept 2022, 11:32 Zhenya Stanilovsky via user, <
user@ignite.apache.org> wrote:
>
>
>
>
>
> We have observed one interesting issue with checkpointing. We are using
> 64G RAM 12 CPU with 3K iops/128mbps SSDs. Our application fills up the WAL
> directory really fast and hence the RAM. We made the following observations
>
> 0. Not so bad news first, it resumes processing after getting stuck for
> several minutes.
>
> 1. WAL and WAL Archive writes are a lot faster than writes to the work
> directory through checkpointing. Very curious to know why this is the case.
> checkpointing writes never exceeds 15 mbps while wal and wal archive go
> really high upto max limits of ssd
>
>
> Very simple example : sequential changing of 1 key, so in wal you obtain
> all changes and in (in your terms — checkpointing) only one key change.
>
>
>
> 2. We observed that when offheap memory usage tend to zero , checkpointing
> takes minutes to complete , sometimes 30+ minutes which stalls the
> application writes completely on all nodes. It means the whole cluster
> freezes.
>
>
> Seems ignite enables throttling in such a case, you need some system and
> cluster tuning.
>
>
>
> 3. Checkpointing thread get stuck at checkpointing page futures.get and
> after several minutes, it logs this error and grid resumes processing
>
> "sys-stripe-0-#1" #19 prio=5 os_prio=0 cpu=86537.69ms elapsed=2166.63s
> tid=0x00007fa52a6f1000 nid=0x3b waiting on condition [0x00007fa4c58be000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.14.1/Native Method)
> at java.util.concurrent.locks.LockSupport.park(java.base@11.0.14.1/Unknown
> Source)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.getUninterruptibly(GridFutureAdapter.java:146)
> at
> org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock.checkpointReadLock(CheckpointTimeoutLock.java:144)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1613)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3313)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:143)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:322)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:317)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1151)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:592)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:393)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
> at
> org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(java.base@11.0.14.1/Unknown Source)
> CheckpointProgress pages = checkpointer.scheduleCheckpoint(0, "too many
> dirty pages");
>
> checkpointReadWriteLock.readUnlock();
>
> if (timeout > 0 && U.currentTimeMillis() - start >= timeout)
> failCheckpointReadLock();
>
> try {
> pages
> .futureFor(LOCK_RELEASED)
> .getUninterruptibly();
> }
>
>
> [2022-09-09 18:58:35,148][ERROR][sys-stripe-9-#10][CheckpointTimeoutLock]
> Checkpoint read lock acquisition has been timed out.
> class
> org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock$CheckpointReadLockTimeoutException:
> Checkpoint read lock acquisition has been timed out.
> at
> org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock.failCheckpointReadLock(CheckpointTimeoutLock.java:210)
> at
> org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointTimeoutLock.checkpointReadLock(CheckpointTimeoutLock.java:108)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1613)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3313)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:143)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:322)
> [2022-09-09 18:58:35,148][INFO ][sys-stripe-7-#8][FailureProcessor] Thread
> dump is hidden due to throttling settings. Set
> IGNITE_DUMP_THREADS_ON_FAILURE_THROTTLING_TIMEOUT property to 0 to see all
> thread dumps.
>
>
> 4. Other nodes printy below logs during the window problematic node is
> stuck at checkpointing
>
> [2022-09-09 18:58:35,153][WARN ][push-metrics-exporter-#80][G] >>>
> Possible starvation in striped pool.
> Thread name: sys-stripe-5-#6
> Queue:
> [o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$DeferredUpdateTimeout@eb9f832,
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1,
> arr=[351148]]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse
> [futIds=GridLongList [idx=2, arr=[273841,273843]]]]], Message closure
> [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false,
> timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> [futId=1662749921887, key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> flags=1, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0],
> subjId=12746da1-ac0d-4ba1-933e-5aa3f92d2f68, taskNameHash=0, createTtl=-1,
> accessTtl=-1, txLbl=null, mvccSnapshot=null]]], Message closure
> [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false,
> timeout=0, skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse
> [futIds=GridLongList [idx=1, arr=[351149]]]]],
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$DeferredUpdateTimeout@110ec0fa,
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=10,
> arr=[414638,414655,414658,414661,414662,414663,414666,414668,414673,414678]]]]],
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$DeferredUpdateTimeout@63ae8204,
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$DeferredUpdateTimeout@2d3cc0b,
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1,
> arr=[414667]]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse
> [futIds=GridLongList [idx=4, arr=[351159,351162,351163,351164]]]]], Message
> closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1,
> arr=[290762]]]]], Message closure [msg=GridIoMessage [plc=2,
> topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0,
> skipOnTimeout=false, msg=GridDhtAtomicDeferredUpdateResponse
> [futIds=GridLongList [idx=1, arr=[400357]]]]],
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$DeferredUpdateTimeout@71887193,
> Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDhtAtomicSingleUpdateRequest [key=BinaryObjectImpl [arr= true,
> ctx=false, start=0], val=BinaryObjectImpl [arr= true, ctx=false, start=0],
> prevVal=null, super=GridDhtAtomicAbstractUpdateRequest [onRes=false,
> nearNodeId=null, nearFutId=0, flags=]]]], Message closure
> [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false,
> timeout=0, skipOnTimeout=false, msg=GridNearAtomicSingleUpdateRequest
> [key=BinaryObjectImpl [arr= true, ctx=false, start=0],
> parent=GridNearAtomicAbstractSingleUpdateRequest [nodeId=null,
> futId=1324019, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0],
> parent=GridNearAtomicAbstractUpdateRequest [res=null, flags=]]]]]]
> Deadlock: false
> Completed: 205703
>
>
> On Wed, Sep 7, 2022 at 4:25 PM Zhenya Stanilovsky via user <
> user@ignite.apache.org
> <//e.mail.ru/compose/?mailto=mailto%3au...@ignite.apache.org>> wrote:
>
> Ok, Raymond i understand. But seems no one have good answer here, it
> depends on appropriate fs and near (probably cloud) layer implementation.
> If you not observe «throttling» messages (described in prev link) seems
> it`s all ok, but of course you can benchmark your io by yourself with 3-rd
> party tool.
>
>
>
> Thanks Zhenya.
>
> I have seen the link you provide has a lot of good information on this
> system. But it does not talk about the check point writers in any detail.
>
> I appreciate this cannot be a bottleneck, my question is more related to:
> "If I have more check pointing threads will check points take less time".
> In our case we use AWS EFS so if each checkpoint thread is spending
> relatively long times blocking on write I/O to the persistent store then
> more check points allow more concurrent writes to take place. Of course, if
> the check point threads themselves utilise async I/O tasks and
> interleave I/O activities on that basis then there may not be an
> opportunity for performance improvement, but I am not an expert in the
> Ignite code base :)
>
> Raymond.
>
>
> On Wed, Sep 7, 2022 at 7:51 PM Zhenya Stanilovsky via user <
> user@ignite.apache.org
> <http://e.mail.ru/compose/?mailto=mailto%3au...@ignite.apache.org>> wrote:
>
>
> No, there is no any log and metrics suggestions and as i told earlier
> — this place can`t became a bottleneck, if you have any performance
> problems — describe them somehow wider and interesting reading here [1]
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
>
>
>
> Thanks Zhenya.
>
> Is there any logging or metrics that would indicate if there was value
> increasing the size of this pool?
>
>
> On Fri, 2 Sep 2022 at 8:20 PM, Zhenya Stanilovsky via user <
> user@ignite.apache.org
> <http://e.mail.ru/compose/?mailto=mailto%3au...@ignite.apache.org>> wrote:
>
> Hi Raymond
>
> checkpoint threads is responsible for dumping modified pages, so you may
> consider it as io bound only operation and pool size is amount of
> disc writing workers.
> I think that default is enough and no need for raising it, but it also up
> to you.
>
>
>
> Hi,
>
> I am looking at our configuration of the Ignite checkpointing system to
> ensure we have it tuned correctly.
>
> There is a checkpointing thread pool defined, which defaults to 4 threads
> in size. I have not been able to find much of a discussion on when/how this
> pool size should be changed to reflect the node size Ignite is running on.
>
> In our case, we are running 16 core servers with 128 GB RAM with
> persistence on an NFS storage layer.
>
> Given the number of cores, and the relative latency of NFS compared to
> local SSD, is 4 checkpointing threads appropriate, or are we likely to see
> better performance if we increased it to 8 (or more)?
>
> If there is a discussion related to this a pointer to it would be good
> (it's not really covered in the performance tuning section).
>
> Thanks,
> Raymond.
>
> --
>
> Raymond Wilson
> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> 11 Birmingham Drive
> <https://www.google.com/maps/search/11+Birmingham+Drive+%7C+Christchurch,+New+Zealand?entry=gmail&source=g>
> |
> <https://www.google.com/maps/search/11+Birmingham+Drive+%7C+Christchurch,+New+Zealand?entry=gmail&source=g>
> Christchurch, New Zealand
> <https://www.google.com/maps/search/11+Birmingham+Drive+%7C+Christchurch,+New+Zealand?entry=gmail&source=g>
> raymond_wil...@trimble.com
> <http://e.mail.ru/compose/?mailto=mailto%3araymond_wil...@trimble.com>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Raymond Wilson
> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> 11 Birmingham Drive | Christchurch, New Zealand
> raymond_wil...@trimble.com
> <http://e.mail.ru/compose/?mailto=mailto%3araymond_wil...@trimble.com>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
