Hello! It seems that you have run out of available memory. I.e., your operating system could not allocate more memory even though the demand was still in the range permitted by data region configuration. How much RAM do you have on that machine?
That you still have heap left is irrelevant here, since the allocation is for non-heap memory. Regards, -- Ilya Kasnacheev пн, 23 нояб. 2020 г. в 21:44, Scott Prater <spra...@gmail.com>: > Hello, > > I recently ran into an out-of-memory error on a durable persistent cache I > set up a few weeks ago. I have a single node, with durable persistence > enabled, as well as WAL archiving. I'm running Ignite ver. > 2.8.1#20200521-sha1:86422096. > > I looked at the stack trace, but I couldn't get a clear fix on what part > of the system ran out of memory, or what parameters I should change to fix > the problem. From what I could tell of the stack dump, it looks like the > WAL archive ran out of memory; but the memory usage report that occurred > just a minute before the exception showed plenty of memory was available. > > Can someone with more experience tuning Ignite memory point me towards the > configuration parameters I should adjust? Below are my log and my > configuration. ( I have read the wiki page on memory tuning, but I'm happy > to be referred back to it.) > > The log, with the metrics right before the OOM exception, then the OOM > exception: > > [2020-11-22T19:20:39,787][INFO ][grid-timeout-worker-#22][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=2845fe3e, uptime=5 days, 15:08:38.033] > ^-- Cluster [hosts=1, CPUs=4, servers=1, clients=0, topVer=1, > minorTopVer=1] > ^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, xxx.xxx.xxx.xxx, 127.0.0.1, > yyy.yyy.yyy.yyy], discoPort=47500, commPort=47100] > ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=0.29%, GC=0%] > ^-- Heap [used=316MB, free=62.34%, comm=812MB] > ^-- Off-heap memory [used=4288MB, free=33.45%, allocated=6344MB] > ^-- Page memory [pages=1085139] > ^-- sysMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, > allocRam=100MB, allocTotal=0MB] > ^-- default_region region [type=default, persistence=true, > lazyAlloc=true, > ... initCfg=256MB, maxCfg=6144MB, usedRam=4288MB, freeRam=30.2%, > allocRam=6144MB, allocTotal=4240MB] > ^-- metastoreMemPlc region [type=internal, persistence=true, > lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.94%, > allocRam=0MB, allocTotal=0MB] > ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, > ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, > allocRam=100MB, allocTotal=0MB] > ^-- Ignite persistence [used=4240MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=6, qSize=0] > [2020-11-22T19:21:15,585][ERROR][db-checkpoint-thread-#63][] Critical > system error detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=CRITICAL_ERROR, > err=java.lang.OutOfMemoryError]] > java.lang.OutOfMemoryError: null > at sun.misc.Unsafe.allocateMemory(Native Method) ~[?:1.8.0_121] > at > org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1205) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.util.GridUnsafe.allocateBuffer(GridUnsafe.java:264) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.wal.ByteBufferExpander.<init>(ByteBufferExpander.java:36) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.<init>(AbstractWalRecordsIterator.java:125) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2701) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2637) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:944) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:920) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.initIfNeeded(CheckpointEntry.java:347) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry$GroupStateLazyStore.access$300(CheckpointEntry.java:243) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.initIfNeeded(CheckpointEntry.java:122) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointEntry.groupState(CheckpointEntry.java:104) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCpToEarliestCpMap(CheckpointHistory.java:242) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointHistory.addCheckpoint(CheckpointHistory.java:175) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:3952) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3515) > ~[ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3404) > [ignite-core-2.9.0.jar:2.9.0] > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > [ignite-core-2.9.0.jar:2.9.0] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] > > My configuration: > > <beans xmlns="http://www.springframework.org/schema/beans"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation=" > http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd";> > <bean id="ignite.cfg" > class="org.apache.ignite.configuration.IgniteConfiguration"> > <property name="gridLogger"> > <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger"> > <constructor-arg type="java.lang.String" > value="/etc/apache-ignite/log4j2.xml"/> > </bean> > </property> > <property name="dataStorageConfiguration"> > <bean > class="org.apache.ignite.configuration.DataStorageConfiguration"> > <property name="storagePath" > value="/data/ignite/persistent-cache"/> > <property name="walPath" value="/data/ignite/wal"/> > <property name="walArchivePath" > value="/data/ignite/wal/archive"/> > <property name="defaultDataRegionConfiguration"> > <bean > class="org.apache.ignite.configuration.DataRegionConfiguration"> > <property name="name" value="default_region"/> > <property name="persistenceEnabled" value="true"/> > <property name="maxSize" value="#{6L * 1024 * 1024 * 1024}"/> > </bean> > </property> > <property name="pageSize" value="#{4 * 1024}"/> > <property name="maxWalArchiveSize" value="#{100L * 1024 * 1024 * > 1024}"/> > </bean> > </property> > <property name="cacheConfiguration"> > <list> > <bean class="org.apache.ignite.configuration.CacheConfiguration"> > <property name="name" value="default" /> > <property name="atomicityMode" value="ATOMIC" /> > <property name="backups" value="1"/> > <property name="dataRegionName" value="default_region"/> > </bean> > </list> > </property> > <property name="discoverySpi"> > <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> > <property name="ipFinder"> > <bean > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> > <property name="addresses"> > <list> > <value>127.0.0.1</value> > </list> > </property> > </bean> > </property> > </bean> > </property> > <property name="communicationSpi"> > <bean > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> > <property name="idleConnectionTimeout" value="60000"/> > </bean> > </property> > </bean> > </beans> > > Thanks in advance, > > -- Scott >
