Hi,
We have an application with ignite thick clients which writes to ignite
caches on ignite grid deployed separately. Below is the ignite
configuration per node
With this configuration, we see throttling happening and checkpointing time
is between 20-30 seconds. Did we miss something in configuration or any
other settings we can enable. Any suggestions will be of great help.
* 100-200 concurrent writes to 25 node cluster
* #partitions 512
* cache backups = 2
* cache mode partitioned
* syncronizationMode : primary Sync
* Off Heap caches
* Server nodes : 25
* RAM : 64G
* maxmemoryDirectSize : 4G
* Heap: 25G
* persistenceEnabled: true
* data region size : 24GB
* checkPointingBufferSize: 6gb
* walSegmentSize: 1G
* walBufferSize : 256MB
* walarchiveSize: 24G
* writeThrotlingEnabled: true
* checkPointingfreq : 60 sec
* checkPointingThreads: 4
* DirectIO enabled: true
SSDs atatched:
work volume : 20G
wal volume : 15G
Wal archive volume : 26G
Checkpointing logs:
[10:27:13,237][INFO][db-checkpoint-thread-#230][Checkpointer] Checkpoint
started [checkpointId=11749dc0-fd0d-4b5f-8b9a-510e774fec38,
startPtr=WALPointer [idx=26, fileOff=385214751, len=16683],
checkpointBeforeLockTime=29ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=2ms, checkpointLockHoldTime=3ms,
walCpRecordFsyncDuration=11ms, writeCheckpointEntryDuration=3ms,
splitAndSortCpPagesDuration=30ms, pages=40505, reason='timeout']
[10:27:13,242][INFO][sys-stripe-7-#8][PageMemoryImpl] Throttling is applied
to page modifications [percentOfPartTime=0.88, markDirty=2121 pages/sec,
checkpointWrite=1219 pages/sec, estIdealMarkDirty=0 pages/sec,
curDirty=0.00, maxDirty=0.02, avgParkTime=410172 ns, pages: (total=40505,
evicted=0, written=10, synced=0, cpBufUsed=133, cpBufTotal=1554645)]
[10:27:29,935][INFO][grid-timeout-worker-#30][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=214f3c2b, uptime=00:45:00.227]
^-- Cluster [hosts=45, CPUs=540, servers=25, clients=20, topVer=75,
minorTopVer=0]
^-- Network [addrs=[127.0.0.1, 192.168.98.141], discoPort=47500,
commPort=47100]
^-- CPU [CPUs=12, curLoad=3.67%, avgLoad=0.82%, GC=0%]
^-- Heap [used=5330MB, free=79.18%, comm=20480MB]
^-- Off-heap memory [used=1019MB, free=95.92%, allocated=24775MB]
^-- Page memory [pages=257976]
^-- sysMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%,
allocRam=99MB, allocTotal=0MB]
^-- default region [type=default, persistence=true, lazyAlloc=true,
... initCfg=24576MB, maxCfg=24576MB, usedRam=1018MB, freeRam=95.86%,
allocRam=24576MB, allocTotal=3820MB]
^-- metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=1MB, freeRam=98.78%,
allocRam=0MB, allocTotal=1MB]
^-- TxLog region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=99MB, allocTotal=0MB]
^-- volatileDsMemPlc region [type=user, persistence=false,
lazyAlloc=true,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=0MB]
^-- Ignite persistence [used=3821MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=7, qSize=0]
^-- Striped thread pool [active=0, idle=12, qSize=0]
[10:27:38,261][INFO][db-checkpoint-thread-#230][Checkpointer] Checkpoint
finished [cpId=11749dc0-fd0d-4b5f-8b9a-510e774fec38, pages=40505,
markPos=WALPointer [idx=26, fileOff=385214751, len=16683],
walSegmentsCovered=[], markDuration=47ms, pagesWrite=25018ms, fsync=6ms,
total=25100ms]
