Hi,
We are using Ignite 1.9, OFFHEAP with SWAP disabled. We are creating caches
programmatic and want to use SQL.
At one instance creating 350 empty cache ran into out-of-memory. We are
already setting low queue size for delete history
(IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE=100)
Log metrics showing 25% free heap memory and 87% free off-heap. Can you
please help me understand this issue and possible resolution? OOM with heap
or offheap or H2 ?
Below is Ignite log with stack trace as well as periodic metrics.
=============
2017-06-15 11:41:41,404,[exchange-worker-#68%TESTNODE%],Skipping rebalancing
(nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
minorTopVer=348], evt=DISCOVERY_CUSTOM_EVT,
node=0c23715a-8c78-4852-9b1a-a5f101d65b7b]
2017-06-15 11:41:43,345,[exchange-worker-#68%TESTNODE%],Failed to
reinitialize local partitions (preloading will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1,
minorTopVer=349], nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT]
class
org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
Failed to allocate memory [total=0, failed=16384]
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(GridUnsafeMemory.java:182)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocateSystem(GridUnsafeMemory.java:145)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(GridUnsafeMap.java:273)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(GridUnsafeMap.java:248)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<init>(GridUnsafePartitionedMap.java:111)
at
org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.unsafePartitionedMap(GridOffHeapMapFactory.java:224)
at
org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(GridOffHeapProcessor.java:72)
at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.initOffHeap(GridCacheSwapManager.java:251)
at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(GridCacheSwapManager.java:138)
at
org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(GridCacheManagerAdapter.java:50)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1091)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1745)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1636)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:382)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:581)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:464)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
2017-06-15 11:41:48,111,[grid-timeout-worker-#54%TESTNODE%],
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=0c23715a, name=TESTNODE, uptime=00:10:06:422]
^-- H/N/C [hosts=1, nodes=1, CPUs=2]
^-- CPU [cur=0.27%, avg=20.87%, GC=0%]
^-- Heap [used=1461MB, free=26.63%, comm=1991MB]
^-- Non heap [used=195MB, free=87.16%, comm=208MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
^-- Outbound messages queue [size=0]
2017-06-15 11:41:48,333,[exchange-worker-#68%TESTNODE%],Failed to wait for
completion of partition map exchange (preloading will not start):
GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], topVer=1, nodeId8=0c23715a, msg=null,
type=DISCOVERY_CUSTOM_EVT, tstamp=1497507101444]], crd=TcpDiscoveryNode
[id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1,
lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1:a8169d0a,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=349],
nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT], added=true,
initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1497507101444,
endTime=1497507107577, ignoreInterrupts=false, state=DONE], init=false,
topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=null,
skipPreload=false, clientOnlyExchange=false, initTs=1497507101444,
centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[],
srvNodes=[TcpDiscoveryNode [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1,
intOrder=1, lastExchangeTime=1497506480276, loc=true,
ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], super=GridFutureAdapter
[resFlag=1, res=class o.a.i.i.util.offheap.GridOffHeapOutOfMemoryException:
Failed to allocate memory [total=0, failed=16384], startTime=1497507101444,
endTime=1497507107577, ignoreInterrupts=false, state=DONE]]
class org.apache.ignite.IgniteCheckedException: Failed to allocate memory
[total=0, failed=16384]
at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7239)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:198)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1680)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: class
org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException:
Failed to allocate memory [total=0, failed=16384]
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0(GridUnsafeMemory.java:182)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocateSystem(GridUnsafeMemory.java:145)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init(GridUnsafeMap.java:273)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>(GridUnsafeMap.java:248)
at
org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.<init>(GridUnsafePartitionedMap.java:111)
at
org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory.unsafePartitionedMap(GridOffHeapMapFactory.java:224)
at
org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create(GridOffHeapProcessor.java:72)
at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.initOffHeap(GridCacheSwapManager.java:251)
at
org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0(GridCacheSwapManager.java:138)
at
org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start(GridCacheManagerAdapter.java:50)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache(GridCacheProcessor.java:1091)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1745)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1636)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:382)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:581)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:464)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674)
... 2 more
=============
Thanks,
-Sam
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Out-Of-Memory-tp13829.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
