[
https://issues.apache.org/jira/browse/IGNITE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389681#comment-16389681
]
Zbyszek B commented on IGNITE-7319:
-----------------------------------
{noformat}
C:\Dev>"C:\Program Files\Java\jdk1.8.0_111\bin\java" -Xmx3G
"-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition
2017.2.5\lib\idea_rt.jar=4917
1:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2017.2.5\bin"
-Dfile.encoding=UTF-8 -classpath "C:\Program
Files\Java\jdk1.8.0_111\jre\lib\charsets
.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\deploy.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\access-bridge-64.jar;C:\Program
Files\Java\jdk1.8.0_11
1\jre\lib\ext\cldrdata.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\dnsns.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\jaccess.jar;C:\Program Files\
Java\jdk1.8.0_111\jre\lib\ext\jfxrt.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\localedata.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\nashorn.jar
;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\sunec.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\sunjce_provider.jar;C:\Program
Files\Java\jdk1.8.0_111\
jre\lib\ext\sunmscapi.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\sunpkcs11.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\zipfs.jar;C:\Program Files
\Java\jdk1.8.0_111\jre\lib\javaws.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\jce.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\jfr.jar;C:\Program Files\Jav
a\jdk1.8.0_111\jre\lib\jfxswt.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\jsse.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\management-agent.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\plugin.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\resources.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\rt.jar;C:\Dev\Ig
niteMemLeak2\out\production\IgniteDemo;C:\Dev\IgniteMemLeak2\lib\2.3\cache-api-1.0.0.jar;C:\Dev\IgniteMemLeak2\lib\2.3\commons-codec-1.6.jar;C:\Dev\IgniteMemLea
k2\lib\2.3\h2-1.4.195.jar;C:\Dev\IgniteMemLeak2\lib\2.3\ignite-core-2.3.0.jar;C:\Dev\IgniteMemLeak2\lib\2.3\ignite-indexing-2.3.0.jar;C:\Dev\IgniteMemLeak2\lib\
2.3\lucene-analyzers-common-5.5.2.jar;C:\Dev\IgniteMemLeak2\lib\2.3\lucene-core-5.5.2.jar;C:\Dev\IgniteMemLeak2\lib\2.3\lucene-queryparser-5.5.2.jar"
p1.Demo
>>> Starting Ignite on MyLeakingNode; work directory C:\Users\testuser\ignite
>>> ...
Mar 06, 2018 6:12:16 PM java.util.logging.LogManager$RootLogger log
SEVERE: Failed to resolve default logging config file:
config/java.util.logging.properties
[18:12:16] __________ ________________
[18:12:16] / _/ ___/ |/ / _/_ __/ __/
[18:12:16] _/ // (7 7 // / / / / _/
[18:12:16] /___/\___/_/|_/___/ /_/ /___/
[18:12:16]
[18:12:16] ver. 2.3.0#20171028-sha1:8add7fd5
[18:12:16] 2017 Copyright(C) Apache Software Foundation
[18:12:16]
[18:12:16] Ignite documentation: http://ignite.apache.org
[18:12:16]
[18:12:16] Quiet mode.
[18:12:16] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
"-v" to ignite.{sh|bat}
[18:12:16]
[18:12:16] OS: Windows 7 6.1 amd64
[18:12:16] VM information: Java(TM) SE Runtime Environment 1.8.0_111-b14 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 25.111-b14
[18:12:16] Initial heap size is 256MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[18:12:16] Configured plugins:
[18:12:16] ^-- None
[18:12:16]
[18:12:16] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message qu
eues growth on sender and receiver sides.
[18:12:16] Security status [authentication=off, tls/ssl=off]
[18:12:18] Performance suggestions for grid (fix if possible)
[18:12:18] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[18:12:18] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
[18:12:18] ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
[18:12:18] ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[18:12:18] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[18:12:18]
[18:12:18] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[18:12:18]
[18:12:18] Ignite node started OK (id=8219875e)
[18:12:18] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, heap=2.7GB]
>>> Ignite started on MyLeakingNode (8219875e-97d5-4e0e-852a-15eb83759b7d)
>>> successfully!
s
Queue size: 1
Caches created: 387307
Caches destroyed: 387303
Mar 07, 2018 4:20:56 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Error when executing timeout callback: CancelableTask
[id=c4ad4810261-a07dab65-3c81-4349-af11-68b9d200e3b3, endTime=1520392859178,
period=5000, cancel=f
alse,
task=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@1b3cca44]
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850)
at
java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1962)
at
org.apache.ignite.internal.util.GridConcurrentSkipListSet.add(GridConcurrentSkipListSet.java:143)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.addTimeoutObject(GridTimeoutProcessor.java:95)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:267)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:163)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Mar 07, 2018 4:21:29 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: GridWorker
[name=grid-timeout-worker, igniteInstanceName=null, finished=false,
hashCode=223614316,
interrupted=false, runner=grid-timeout-worker-#23]
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850)
at
java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1962)
at
org.apache.ignite.internal.util.GridConcurrentSkipListSet.add(GridConcurrentSkipListSet.java:143)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.addTimeoutObject(GridTimeoutProcessor.java:95)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:267)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:163)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "grid-timeout-worker-#23" java.lang.OutOfMemoryError: GC
overhead limit exceeded
at
java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850)
at
java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1962)
at
org.apache.ignite.internal.util.GridConcurrentSkipListSet.add(GridConcurrentSkipListSet.java:143)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.addTimeoutObject(GridTimeoutProcessor.java:95)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:267)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:163)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Mar 07, 2018 4:23:36 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to unmarshal discovery custom message.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.HashMap$KeySet.iterator(HashMap.java:916)
at java.util.HashSet.iterator(HashSet.java:172)
at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
at
org.apache.ignite.internal.managers.discovery.DiscoCache.<init>(DiscoCache.java:123)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.createDiscoCache(GridDiscoveryManager.java:2280)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.access$1700(GridDiscoveryManager.java:160)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:712)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery(GridDiscoveryManager.java:571)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:5463)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processCustomMessage(ServerImpl.java:5282)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2656)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2447)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6648)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2533)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Mar 07, 2018 4:24:13 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to initialize cache. Will try to rollback cache start routine.
[cacheName=localCache11251307]
class org.apache.ignite.IgniteCheckedException: Failed to register MBean for
component:
org.apache.ignite.internal.processors.cache.CacheClusterMetricsMXBeanImp
l@204fbdb1
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.registerMbean(GridCacheProcessor.java:3600)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1566)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1799)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:751)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeF
uture.java:882)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:588)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2279)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at
org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:112)
at
org.apache.ignite.internal.util.IgniteUtils.makeCacheMBeanName(IgniteUtils.java:4484)
at
org.apache.ignite.internal.util.IgniteUtils.registerCacheMBean(IgniteUtils.java:4574)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.registerMbean(GridCacheProcessor.java:3596)
... 8 more
Mar 07, 2018 9:51:20 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1708, name=tcp-disco-sock-reader-#8, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#8" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Mar 07, 2018 9:51:30 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: IgniteSpiThread
[name=tcp-disco-ip-finder-cleaner-#5]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-ip-finder-cleaner-#5"
java.lang.OutOfMemoryError: GC overhead limit exceeded
Mar 07, 2018 9:51:49 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1706, name=tcp-disco-sock-reader-#6, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#6" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Mar 07, 2018 9:52:05 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1707, name=tcp-disco-sock-reader-#7, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#7" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Mar 07, 2018 9:52:37 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1709, name=tcp-disco-sock-reader-#9, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#9" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Mar 07, 2018 9:53:13 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1711, name=tcp-disco-sock-reader-#11, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#11" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Mar 07, 2018 9:53:24 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1710, name=tcp-disco-sock-reader-#10, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#10" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Mar 07, 2018 9:54:45 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1712, name=tcp-disco-sock-reader-#12, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#12" java.lang.OutOfMemoryError: GC
overhead limit exceeded
Mar 07, 2018 9:54:52 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: Socket reader
[id=1713, name=tcp-disco-sock-reader-#13, nodeId=null]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "tcp-disco-sock-reader-#13" java.lang.OutOfMemoryError: GC
overhead limit exceeded
{noformat}
> Memory leak during creating/destroying local cache
> --------------------------------------------------
>
> Key: IGNITE-7319
> URL: https://issues.apache.org/jira/browse/IGNITE-7319
> Project: Ignite
> Issue Type: Bug
> Components: cache
> Affects Versions: 2.3
> Reporter: Mikhail Cherkasov
> Assignee: Alexey Goncharuk
> Priority: Major
> Fix For: 2.5
>
>
> The following code creates local caches:
> {code}
> private IgniteCache<String, BinaryObject> createLocalCache(String name) {
> CacheConfiguration<String, BinaryObject> cCfg = new
> CacheConfiguration<>();
> cCfg.setName(name);
> cCfg.setGroupName("localCaches"); // without group leak is much
> bigger!
> cCfg.setStoreKeepBinary(true);
> cCfg.setCacheMode(CacheMode.LOCAL);
> cCfg.setOnheapCacheEnabled(false);
> cCfg.setCopyOnRead(false);
> cCfg.setBackups(0);
> cCfg.setWriteBehindEnabled(false);
> cCfg.setReadThrough(false);
> cCfg.setReadFromBackup(false);
> cCfg.setQueryEntities(<some basic query here>);
> return ignite.createCache(cCfg).withKeepBinary();
> }
> {code}
> The caches are placed in the queue and are picked up by the worker thread
> which just destroys them after removing from the queue.
> This setup seems to generate a memory leak of about 1GB per day.
> When looking at heap dump, I see all space is occupied by instances of
> java.util.concurrent.ConcurrentSkipListMap$Node.
> User list:
> http://apache-ignite-users.70518.x6.nabble.com/Memory-leak-in-GridCachePartitionExchangeManager-tt18995.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
