Hi,
We have a 4 node cluster with 64G RAM and 40G DISK per node attached for
/work and /walarchive each. WAL dir is 10G per node
Below is our data region configuration and jvm_opts. We are getting
timeouts on checkpointing and WalArchive is getting filled up and no data
is moving to the /work directory. Checkpointing error is mentioned below
could you suggest what's wrong with these configs(Note : with
walarchveSize as 16G and checkpointingBUffSize as 4G, it was writing but
very slow and throttling rate was 35% so I increased it, but it started
showing timeout errors)
JVM_OPTS: -XX:MaxDirectMemorySize=2g -Xms20g -Xmx25g
-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
-XX:+DisableExplicitGC
Ignite config:
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="walBufferSize" value="#{256L * 1024 *
1024}"/>
<property name="checkpointFrequency" value="30000"/>
<property name="checkpointThreads" value="12"/>
<property name="walSegmentSize" value="#{512L * 1024 *
1024}"/>
<property name="maxWalArchiveSize" value="#{32L * 1024 *
1024 * 1024}"/>
<property name="writeThrottlingEnabled" value="true"/>
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<!--
https://ignite.apache.org/docs/latest/persistence/persistence-tuning#adjusting-checkpointing-buffer-size--
>
<property name="checkpointPageBufferSize"
value="#{8L * 1024 * 1024 * 1024}"/>
<!--<property name="pageReplacementMode"
value="SEGMENTED_LRU"/>-->
</bean>
</property>
<property name="walPath" value="/ignite/wal"/>
<property name="walArchivePath" value="/ignite/walarchive"/>
</bean>
</property>
Checkpointing logs:
[14:56:55,209][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint
started [checkpointId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653,
startPtr=WALPointer [idx=930, fileOff=527943956, len=51411],
checkpointBeforeLockTime=9ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=9ms,
walCpRecordFsyncDuration=7ms, writeCheckpointEntryDuration=3ms,
splitAndSortCpPagesDuration=45ms, pages=35542, reason='timeout']
[14:56:55,921][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint
finished [cpId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653, pages=35542,
markPos=WALPointer [idx=930, fileOff=527943956, len=51411],
walSegmentsCovered=[], markDuration=64ms, pagesWrite=108ms, fsync=604ms,
total=785ms]
