Hi Surinder, > I increased it, but it started showing timeout errors
These messages are INFO messages but not errors. Checkpoints can be triggered by several causes, some of them are below: 1) Checkpoint timeout. You specified 'checkpointFrequency' equal to 30 seconds, which means that checkpoint should be attempted at least each 30 seconds. 2) Too many dirty pages. 3) Creating a snapshot. 4) WAL segments rollover. ср, 23 мар. 2022 г. в 18:10, Surinder Mehra <redni...@gmail.com>: > Hi, > We have a 4 node cluster with 64G RAM and 40G DISK per node attached for > /work and /walarchive each. WAL dir is 10G per node > > Below is our data region configuration and jvm_opts. We are getting > timeouts on checkpointing and WalArchive is getting filled up and no data > is moving to the /work directory. Checkpointing error is mentioned below > > could you suggest what's wrong with these configs(Note : with > walarchveSize as 16G and checkpointingBUffSize as 4G, it was writing but > very slow and throttling rate was 35% so I increased it, but it started > showing timeout errors) > > JVM_OPTS: -XX:MaxDirectMemorySize=2g -Xms20g -Xmx25g > -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC > -XX:+DisableExplicitGC > > Ignite config: > > <property name="dataStorageConfiguration"> > <bean > class="org.apache.ignite.configuration.DataStorageConfiguration"> > <property name="walBufferSize" value="#{256L * 1024 * > 1024}"/> > <property name="checkpointFrequency" value="30000"/> > <property name="checkpointThreads" value="12"/> > <property name="walSegmentSize" value="#{512L * 1024 * > 1024}"/> > <property name="maxWalArchiveSize" value="#{32L * 1024 * > 1024 * 1024}"/> > <property name="writeThrottlingEnabled" value="true"/> > <property name="defaultDataRegionConfiguration"> > <bean > class="org.apache.ignite.configuration.DataRegionConfiguration"> > <property name="persistenceEnabled" value="true"/> > <!-- > https://ignite.apache.org/docs/latest/persistence/persistence-tuning#adjusting-checkpointing-buffer-size-- > > > <property name="checkpointPageBufferSize" > value="#{8L * 1024 * 1024 * 1024}"/> > <!--<property name="pageReplacementMode" > value="SEGMENTED_LRU"/>--> > </bean> > </property> > > <property name="walPath" value="/ignite/wal"/> > <property name="walArchivePath" > value="/ignite/walarchive"/> > </bean> > > </property> > > > Checkpointing logs: > > [14:56:55,209][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint > started [checkpointId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653, > startPtr=WALPointer [idx=930, fileOff=527943956, len=51411], > checkpointBeforeLockTime=9ms, checkpointLockWait=0ms, > checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=9ms, > walCpRecordFsyncDuration=7ms, writeCheckpointEntryDuration=3ms, > splitAndSortCpPagesDuration=45ms, pages=35542, reason='timeout'] > [14:56:55,921][INFO][db-checkpoint-thread-#105][Checkpointer] Checkpoint > finished [cpId=4b071279-8f25-4cd0-a5ef-e6f58f3a5653, pages=35542, > markPos=WALPointer [idx=930, fileOff=527943956, len=51411], > walSegmentsCovered=[], markDuration=64ms, pagesWrite=108ms, fsync=604ms, > total=785ms] >