hope it would be helpful too:
https://www.jeffgeerling.com/blog/2018/getting-best-performance-out-amazon-efs
https://docs.aws.amazon.com/efs/latest/ug/storage-classes.html
>
>Hi Zhenya,
>
>The matching checkpoint finished log is this:
>
>2020-12-15 19:07:39,253 [106] INF [MutableCacheComputeServer] Checkpoint
>finished [cpId=e2c31b43-44df-43f1-b162-6b6cefa24e28, pages=33421,
>markPos=FileWALPointer [idx=6339, fileOff=243287334, len=196573],
>walSegmentsCleared=0, walSegmentsCovered=[], markDuration=218ms,
>pagesWrite=1150ms, fsync=37104ms, total=38571ms]
>
>Regards your comment that 3/4 of pages in whole data region need to be dirty
>to trigger this, can you confirm this is 3/4 of the maximum size of the data
>region, or of the currently used size (eg: if Min is 1Gb, and Max is 4Gb, and
>used is 2Gb, would 1.5Gb of dirty pages trigger this?)
>
>Are data regions independently checkpointed, or are they checkpointed as a
>whole, so that a 'too many dirty pages' condition affects all data regions in
>terms of write blocking?
>
>Can you comment on my query regarding should we set Min and Max size of the
>data region to be the same? Ie: Don't bother with growing the data region
>memory use on demand, just allocate the maximum?
>
>In terms of the checkpoint lock hold time metric, of the checkpoints quoting
>'too many dirty pages' there is one instance apart from the one I have
>provided earlier violating this limit, ie:
>
>2020-12-17 18:56:39,086 [104] INF [MutableCacheComputeServer] Checkpoint
>started [checkpointId=e9ccf0ca-f813-4f91-ac93-5483350fdf66,
>startPtr=FileWALPointer [idx=7164, fileOff=389224517, len=196573],
>checkpointBeforeLockTime=276ms, checkpointLockWait=0ms,
>checkpointListenersExecuteTime=16ms, checkpointLockHoldTime=39ms,
>walCpRecordFsyncDuration=254ms, writeCheckpointEntryDuration=32ms,
>splitAndSortCpPagesDuration=276ms, pages=77774, reason=' too many dirty pages
>']
>
>This is out of a population of 16 instances I can find. The remainder have
>lock times of 16-17ms.
>
>Regarding writes of pages to the persistent store, does the check pointing
>system parallelise writes across partitions ro maximise throughput?
>
>Thanks,
>Raymond.
>
>
>On Thu, Dec 31, 2020 at 1:17 AM Zhenya Stanilovsky < arzamas...@mail.ru >
>wrote:
>>
>>All write operations will be blocked for this timeout :
>>checkpointLockHoldTime=32ms (Write Lock holding) If you observe huge amount
>>of such messages : reason=' too many dirty pages ' may be you need to
>>store some data in not persisted regions for example or reduce indexes (if
>>you use them). And please attach other part of cp message starting with :
>>Checkpoint finished.
>>
>>
>>
>>>In (
>>>https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood
>>> ), there is a mention of a dirty pages limit that is a factor that can
>>>trigger check points.
>>>
>>>I also found this issue:
>>>http://apache-ignite-users.70518.x6.nabble.com/too-many-dirty-pages-td28572.html
>>> where "too many dirty pages" is a reason given for initiating a checkpoint.
>>>
>>>After reviewing our logs I found this: (one example)
>>>
>>>2020-12-15 19:07:00,999 [106] INF [MutableCacheComputeServer] Checkpoint
>>>started [checkpointId=e2c31b43-44df-43f1-b162-6b6cefa24e28,
>>>startPtr=FileWALPointer [idx=6339, fileOff=243287334, len=196573],
>>>checkpointBeforeLockTime=99ms, checkpointLockWait=0ms,
>>>checkpointListenersExecuteTime=16ms, checkpointLockHoldTime=32ms,
>>>walCpRecordFsyncDuration=113ms, writeCheckpointEntryDuration=27ms,
>>>splitAndSortCpPagesDuration=45ms, pages=33421, reason=' too many dirty pages
>>>']
>>>
>>>Which suggests we may have the issue where writes are frozen until the check
>>>point is completed.
>>>
>>>Looking at the AI 2.8.1 source code, the dirty page limit fraction appears
>>>to be 0.1 (10%), via this entry in GridCacheDatabaseSharedManager.java:
>>>
>>> /**
>>> * Threshold to calculate limit for pages list on-heap caches.
>>> * <p>
>>> * Note: When a checkpoint is triggered, we need some amount of page
>>>memory to store pages list on-heap cache.
>>> * If a checkpoint is triggered by "too many dirty pages" reason and
>>>pages list cache is rather big, we can get
>>>* {@code IgniteOutOfMemoryException}. To prevent this, we can limit the
>>>total amount of cached page list buckets,
>>> * assuming that checkpoint will be triggered if no more then 3/4 of
>>>pages will be marked as dirty (there will be
>>> * at least 1/4 of clean pages) and each cached page list bucket can be
>>>stored to up to 2 pages (this value is not
>>> * static, but depends on PagesCache.MAX_SIZE, so if PagesCache.MAX_SIZE
>>>> PagesListNodeIO#getCapacity it can take
>>> * more than 2 pages). Also some amount of page memory needed to store
>>>page list metadata.
>>> */
>>> private static final double PAGE_LIST_CACHE_LIMIT_THRESHOLD =
>>>0.1 ;
>>>
>>>This raises two questions:
>>>
>>>1. The data region where most writes are occurring has 4Gb allocated to it,
>>>though it is permitted to start at a much lower level. 4Gb should be
>>>1,000,000 pages, 10% of which should be 100,000 dirty pages.
>>>
>>>The 'limit holder' is calculated like this:
>>>
>>> /**
>>> * @return Holder for page list cache limit for given data region.
>>> */
>>> public AtomicLong pageListCacheLimitHolder ( DataRegion
>>>dataRegion ) {
>>> if ( dataRegion . config (). isPersistenceEnabled ()) {
>>> return pageListCacheLimits . computeIfAbsent ( dataRegion .
>>>config (). getName (), name -> new AtomicLong (
>>> ( long )(((PageMemoryEx) dataRegion . pageMemory ()).
>>>totalPages () * PAGE_LIST_CACHE_LIMIT_THRESHOLD)));
>>> }
>>> return null ;
>>> }
>>>
>>>... but I am unsure if totalPages() is referring to the current size of the
>>>data region, or the size it is permitted to grow to. ie: Could the 'dirty
>>>page limit' be a sliding limit based on the growth of the data region? Is it
>>>better to set the initial and maximum sizes of data regions to be the same
>>>number?
>>>
>>>2. We have two data regions, one supporting inbound arrival of data (with
>>>low numbers of writes), and one supporting storage of processed results from
>>>the arriving data (with many more writes).
>>>
>>>The block on writes due to the number of dirty pages appears to affect all
>>>data regions, not just the one which has violated the dirty page limit. Is
>>>that correct? If so, is this something that can be improved?
>>>
>>>Thanks,
>>>Raymond.
>>>
>>>On Wed, Dec 30, 2020 at 9:17 PM Raymond Wilson < raymond_wil...@trimble.com
>>>> wrote:
>>>>I'm working on getting automatic JVM thread stack dumping occurring if we
>>>>detect long delays in put (PutIfAbsent) operations. Hopefully this will
>>>>provide more information.
>>>>On Wed, Dec 30, 2020 at 7:48 PM Zhenya Stanilovsky < arzamas...@mail.ru >
>>>>wrote:
>>>>>
>>>>>Don`t think so, checkpointing work perfectly well already before this fix.
>>>>>Need additional info for start digging your problem, can you share ignite
>>>>>logs somewhere?
>>>>>
>>>>>>I noticed an entry in the Ignite 2.9.1 changelog:
>>>>>>* Improved checkpoint concurrent behaviour
>>>>>>I am having trouble finding the relevant Jira ticket for this in the
>>>>>>2.9.1 Jira area at
>>>>>>https://issues.apache.org/jira/browse/IGNITE-13876?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1%20and%20status%20%3D%20Resolved
>>>>>>
>>>>>>Perhaps this change may improve the checkpointing issue we are seeing?
>>>>>>
>>>>>>Raymond.
>>>>>>
>>>>>>On Tue, Dec 29, 2020 at 8:35 PM Raymond Wilson <
>>>>>>raymond_wil...@trimble.com > wrote:
>>>>>>>Hi Zhenya,
>>>>>>>
>>>>>>>1. We currently use AWS EFS for primary storage, with provisioned IOPS
>>>>>>>to provide sufficient IO. Our Ignite cluster currently tops out at ~10%
>>>>>>>usage (with at least 5 nodes writing to it, including WAL and WAL
>>>>>>>archive), so we are not saturating the EFS interface. We use the default
>>>>>>>page size (experiments with larger page sizes showed instability when
>>>>>>>checkpointing due to free page starvation, so we reverted to the default
>>>>>>>size).
>>>>>>>
>>>>>>>2. Thanks for the detail, we will look for that in thread dumps when we
>>>>>>>can create them.
>>>>>>>
>>>>>>>3. We are using the default CP buffer size, which is max(256Mb,
>>>>>>>DataRagionSize / 4) according to the Ignite documentation, so this
>>>>>>>should have more than enough checkpoint buffer space to cope with
>>>>>>>writes. As additional information, the cache which is displaying very
>>>>>>>slow writes is in a data region with relatively slow write traffic.
>>>>>>>There is a primary (default) data region with large write traffic, and
>>>>>>>the vast majority of pages being written in a checkpoint will be for
>>>>>>>that default data region.
>>>>>>>
>>>>>>>4. Yes, this is very surprising. Anecdotally from our logs it appears
>>>>>>>write traffic into the low write traffic cache is blocked during
>>>>>>>checkpoints.
>>>>>>>
>>>>>>>Thanks,
>>>>>>>Raymond.
>>>>>>>
>>>>>>>
>>>>>>>On Tue, Dec 29, 2020 at 7:31 PM Zhenya Stanilovsky < arzamas...@mail.ru
>>>>>>>> wrote:
>>>>>>>>*
>>>>>>>>Additionally to Ilya reply you can check vendors page for additional
>>>>>>>>info, all in this page are applicable for ignite too [1]. Increasing
>>>>>>>>threads number leads to concurrent io usage, thus if your have
>>>>>>>>something like nvme — it`s up to you but in case of sas possibly better
>>>>>>>>would be to reduce this param.
>>>>>>>>* Log will shows you something like :
>>>>>>>>Parking thread=%Thread name% for timeout(ms)= %time% and appropriate :
>>>>>>>>Unparking thread=
>>>>>>>>* No additional looging with cp buffer usage are provided. cp buffer
>>>>>>>>need to be more than 10% of overall persistent DataRegions size.
>>>>>>>>* 90 seconds or longer — Seems like problems in io or system
>>>>>>>>tuning, it`s very bad score i hope.
>>>>>>>>[1]
>>>>>>>>https://www.gridgain.com/docs/latest/perf-troubleshooting-guide/persistence-tuning
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>Hi,
>>>>>>>>>
>>>>>>>>>We have been investigating some issues which appear to be related to
>>>>>>>>>checkpointing. We currently use the IA 2.8.1 with the C# client.
>>>>>>>>>
>>>>>>>>>I have been trying to gain clarity on how certain aspects of the
>>>>>>>>>Ignite configuration relate to the checkpointing process:
>>>>>>>>>
>>>>>>>>>1. Number of check pointing threads. This defaults to 4, but I don't
>>>>>>>>>understand how it applies to the checkpointing process. Are more
>>>>>>>>>threads generally better (eg: because it makes the disk IO parallel
>>>>>>>>>across the threads), or does it only have a positive effect if you
>>>>>>>>>have many data storage regions? Or something else? If this could be
>>>>>>>>>clarified in the documentation (or a pointer to it which Google has
>>>>>>>>>not yet found), that would be good.
>>>>>>>>>
>>>>>>>>>2. Checkpoint frequency. This is defaulted to 180 seconds. I was
>>>>>>>>>thinking that reducing this time would result in smaller less
>>>>>>>>>disruptive check points. Setting it to 60 seconds seems pretty safe,
>>>>>>>>>but is there a practical lower limit that should be used for use cases
>>>>>>>>>with new data constantly being added, eg: 5 seconds, 10 seconds?
>>>>>>>>>
>>>>>>>>>3. Write exclusivity constraints during checkpointing. I understand
>>>>>>>>>that while a checkpoint is occurring ongoing writes will be supported
>>>>>>>>>into the caches being check pointed, and if those are writes to
>>>>>>>>>existing pages then those will be duplicated into the checkpoint
>>>>>>>>>buffer. If this buffer becomes full or stressed then Ignite will
>>>>>>>>>throttle, and perhaps block, writes until the checkpoint is complete.
>>>>>>>>>If this is the case then Ignite will emit logging (warning or
>>>>>>>>>informational?) that writes are being throttled.
>>>>>>>>>
>>>>>>>>>We have cases where simple puts to caches (a few requests per second)
>>>>>>>>>are taking up to 90 seconds to execute when there is an active check
>>>>>>>>>point occurring, where the check point has been triggered by the
>>>>>>>>>checkpoint timer. When a checkpoint is not occurring the time to do
>>>>>>>>>this is usually in the milliseconds. The checkpoints themselves can
>>>>>>>>>take 90 seconds or longer, and are updating up to 30,000-40,000 pages,
>>>>>>>>>across a pair of data storage regions, one with 4Gb in-memory space
>>>>>>>>>allocated (which should be 1,000,000 pages at the standard 4kb page
>>>>>>>>>size), and one small region with 128Mb. There is no 'throttling'
>>>>>>>>>logging being emitted that we can tell, so the checkpoint buffer
>>>>>>>>>(which should be 1Gb for the first data region and 256 Mb for the
>>>>>>>>>second smaller region in this case) does not look like it can fill up
>>>>>>>>>during the checkpoint.
>>>>>>>>>
>>>>>>>>>It seems like the checkpoint is affecting the put operations, but I
>>>>>>>>>don't understand why that may be given the documented checkpointing
>>>>>>>>>process, and the checkpoint itself (at least via Informational
>>>>>>>>>logging) is not advertising any restrictions.
>>>>>>>>>
>>>>>>>>>Thanks,
>>>>>>>>>Raymond.
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>Raymond Wilson
>>>>>>>>>Solution Architect, Civil Construction Software Systems (CCSS)
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>Raymond Wilson
>>>>>>>Solution Architect, Civil Construction Software Systems (CCSS)
>>>>>>>11 Birmingham Drive | Christchurch, New Zealand
>>>>>>>+64-21-2013317 Mobile
>>>>>>>raymond_wil...@trimble.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>Raymond Wilson
>>>>>>Solution Architect, Civil Construction Software Systems (CCSS)
>>>>>>11 Birmingham Drive | Christchurch, New Zealand
>>>>>>+64-21-2013317 Mobile
>>>>>>raymond_wil...@trimble.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>Raymond Wilson
>>>>Solution Architect, Civil Construction Software Systems (CCSS)
>>>>11 Birmingham Drive | Christchurch, New Zealand
>>>>+64-21-2013317 Mobile
>>>>raymond_wil...@trimble.com
>>>>
>>>>
>>>
>>> --
>>>
>>>Raymond Wilson
>>>Solution Architect, Civil Construction Software Systems (CCSS)
>>>11 Birmingham Drive | Christchurch, New Zealand
>>>+64-21-2013317 Mobile
>>>raymond_wil...@trimble.com
>>>
>>>
>>
>>
>>
>>
>
> --
>
>Raymond Wilson
>Solution Architect, Civil Construction Software Systems (CCSS)
>11 Birmingham Drive | Christchurch, New Zealand
>+64-21-2013317 Mobile
>raymond_wil...@trimble.com
>
>
>
>
>
