[ https://issues.apache.org/jira/browse/DERBY-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vamsavardhana Reddy updated DERBY-7075: --------------------------------------- Description: We have two runtime environments v7.5.4 and v7.5.3 similar in terms of OS memory, java version and java launch command settings for the derby and other java processes required for our application in which similar projects/workloads are running. But in one of the environments we are seeing out of memory (OOM) exceptions in the derby server java process. In this environment where OOM is occurring, we even refreshed Apache derby to the latest version available for Java 8 and still encountered OOM exceptions. Please help resolve this OOM issue. Exception in thread "DRDAConnThread_39" Exception in thread "DRDAConnThread_41" java/lang/OutOfMemoryError: Java heap space at java/nio/CharBuffer.wrap (CharBuffer.java:373) at sun/nio/cs/StreamEncoder.implWrite (StreamEncoder.java:277) at sun/nio/cs/StreamEncoder.write (StreamEncoder.java:137) at java/io/OutputStreamWriter.write (OutputStreamWriter.java:244) at java/io/BufferedWriter.flushBuffer (BufferedWriter.java:140) at java/io/PrintStream.write (PrintStream.java:550) at java/io/PrintStream.print (PrintStream.java:693) at java/io/PrintStream.append (PrintStream.java:1089) at java/io/PrintStream.append (PrintStream.java:70) at com/ibm/oti/util/Util.appendTo (Util.java:225) at java/lang/Throwable.printStackTrace (Throwable.java:506) at java/lang/Throwable.printStackTrace (Throwable.java:302) at java/lang/Throwable.printStackTrace (Throwable.java:247) Exception in thread "DRDAConnThread_43" java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError(?)java.lang.OutOfMemoryError: Java heap space at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:872) at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:8java/lang/OutOfMemoryError: Java heap space at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:872) at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866) at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866) at java/lang/Thread.uncaughtException (Thread.java:1332) ----------------------- :*v7.5.4* OS: RHELv8 with 4GB RAM Java: IBM Java SDK v8.0.6.0 Apache Derby v10.8.3.3, Apache Derby v10.14.2.0 [bceglc385/Standalone ~]# /usr/java/default/jre/bin/java -version java version "1.8.0_231" Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6)) IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191106_432135 (JIT enabled, AOT enabled) OpenJ9 - f0b6be7 OMR - 18d8f94 IBM - 233dfb5) JCL - 20191016_01 based on Oracle jdk8u231-b10 /usr/java/default/bin/java -server -Xdump:heap+system+snap:none -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.ph -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 -XX:+HeapManagementMXBeanCompatibility -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Dderby.system.home=/usr/ironhide/var/db/staging -Dcom.sun.management.jmxremote -Dcom.approuter.management.jmxremote.connectorserver.port=31997 -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote -Dcom.sun.management.jmxremote.port=1997 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access -Dcom.approuter.logging.filename=derbyserver.log -Djava.rmi.server.hostname=9.121.242.177 -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true -Dcom.approuter.sysconf.appliance=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 -Dosgi.clean=true -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true -Declipse.consoleLog=true -Dcom.approuter.orcmon.purger.compressionmode=sequential -DIH_ROOT=/usr/ironhide -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar [bceglc390/Standalone ~]# cat /proc/meminfo MemTotal: 4057920 kB MemFree: 152896 kB Buffers: 342624 kB Cached: 729112 kB SwapCached: 3428 kB Active: 2644068 kB Inactive: 1066404 kB Active(anon): 1951704 kB Inactive(anon): 712696 kB Active(file): 692364 kB Inactive(file): 353708 kB Unevictable: 20 kB Mlocked: 20 kB SwapTotal: 3999736 kB SwapFree: 3988760 kB Dirty: 720 kB Writeback: 0 kB AnonPages: 2636560 kB Mapped: 38508 kB Shmem: 25648 kB Slab: 129600 kB SReclaimable: 48320 kB SUnreclaim: 81280 kB KernelStack: 6568 kB PageTables: 11116 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 6028696 kB Committed_AS: 1658244 kB VmallocTotal: 34359738367 kB VmallocUsed: 152396 kB VmallocChunk: 34359582800 kB AnonHugePages: 2433024 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 4096 kB DirectMap2M: 4190208 kB ---------- *v7.5.3* OS: Bedrock v5 which is custom OS based on RHELv7 with 4GB RAM Java: IBM Java SDK v8.0.6.0 Apache Derby v10.8.3.3 [bceglc390/Standalone ~]# /usr/java/default/jre/bin/java -version java version "1.8.0_231" Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6)) IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191106_432135 (JIT enabled, AOT enabled) OpenJ9 - f0b6be7 OMR - 18d8f94 IBM - 233dfb5) JCL - 20191016_01 based on Oracle jdk8u231-b10 /usr/java/default/bin/java -server -Xdump:heap+system+snap:none -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.phd -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 -XX:+HeapManagementMXBeanCompatibility -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Dderby.system.home=/usr/ironhide/var/db/staging -Dcom.sun.management.jmxremote -Dcom.approuter.management.jmxremote.connectorserver.port=31997 -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote -Dcom.sun.management.jmxremote.port=1997 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access -Dcom.approuter.logging.filename=derbyserver.log -Djava.rmi.server.hostname=9.124.112.240 -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true -Dcom.approuter.sysconf.appliance=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 -Dosgi.clean=true -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true -Declipse.consoleLog=true -Dcom.approuter.orcmon.purger.compressionmode=sequential -DIH_ROOT=/usr/ironhide -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar [cldegdh9/Standalone log]# cat /proc/meminfo MemTotal: 3872868 kB MemFree: 343700 kB MemAvailable: 374676 kB Buffers: 33100 kB Cached: 170624 kB SwapCached: 3848 kB Active: 2540784 kB Inactive: 710092 kB Active(anon): 2429812 kB Inactive(anon): 621324 kB Active(file): 110972 kB Inactive(file): 88768 kB Unevictable: 20 kB Mlocked: 20 kB SwapTotal: 5345272 kB SwapFree: 5280760 kB Dirty: 1880 kB Writeback: 0 kB AnonPages: 3044312 kB Mapped: 33676 kB Shmem: 3984 kB Slab: 119980 kB SReclaimable: 71120 kB SUnreclaim: 48860 kB KernelStack: 15116 kB PageTables: 13936 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 7281704 kB Committed_AS: 1755044 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 2885632 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 126912 kB DirectMap2M: 4067328 kB was: We have two runtime environments v7.5.4 and v7.5.3 similar in terms of OS memory, java version and java launch command settings for the derby and other java processes required for our application in which similar projects/workloads are running. But in one of the environments we are seeing out of memory (OOM) exceptions in the derby server java process. In this environment where OOM is occurring, we even refreshed Apache derby to the latest version available for Java 8 and still encountered OOM exceptions. Please help resolve this OOM issue. :*v7.5.4* OS: RHELv8 with 4GB RAM Java: IBM Java SDK v8.0.6.0 Apache Derby v10.8.3.3, Apache Derby v10.14.2.0 [bceglc385/Standalone ~]# /usr/java/default/jre/bin/java -version java version "1.8.0_231" Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6)) IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191106_432135 (JIT enabled, AOT enabled) OpenJ9 - f0b6be7 OMR - 18d8f94 IBM - 233dfb5) JCL - 20191016_01 based on Oracle jdk8u231-b10 /usr/java/default/bin/java -server -Xdump:heap+system+snap:none -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.ph -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 -XX:+HeapManagementMXBeanCompatibility -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Dderby.system.home=/usr/ironhide/var/db/staging -Dcom.sun.management.jmxremote -Dcom.approuter.management.jmxremote.connectorserver.port=31997 -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote -Dcom.sun.management.jmxremote.port=1997 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access -Dcom.approuter.logging.filename=derbyserver.log -Djava.rmi.server.hostname=9.121.242.177 -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true -Dcom.approuter.sysconf.appliance=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 -Dosgi.clean=true -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true -Declipse.consoleLog=true -Dcom.approuter.orcmon.purger.compressionmode=sequential -DIH_ROOT=/usr/ironhide -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar [bceglc390/Standalone ~]# cat /proc/meminfo MemTotal: 4057920 kB MemFree: 152896 kB Buffers: 342624 kB Cached: 729112 kB SwapCached: 3428 kB Active: 2644068 kB Inactive: 1066404 kB Active(anon): 1951704 kB Inactive(anon): 712696 kB Active(file): 692364 kB Inactive(file): 353708 kB Unevictable: 20 kB Mlocked: 20 kB SwapTotal: 3999736 kB SwapFree: 3988760 kB Dirty: 720 kB Writeback: 0 kB AnonPages: 2636560 kB Mapped: 38508 kB Shmem: 25648 kB Slab: 129600 kB SReclaimable: 48320 kB SUnreclaim: 81280 kB KernelStack: 6568 kB PageTables: 11116 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 6028696 kB Committed_AS: 1658244 kB VmallocTotal: 34359738367 kB VmallocUsed: 152396 kB VmallocChunk: 34359582800 kB AnonHugePages: 2433024 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 4096 kB DirectMap2M: 4190208 kB ---------- *v7.5.3* OS: Bedrock v5 which is custom OS based on RHELv7 with 4GB RAM Java: IBM Java SDK v8.0.6.0 Apache Derby v10.8.3.3 [bceglc390/Standalone ~]# /usr/java/default/jre/bin/java -version java version "1.8.0_231" Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6)) IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191106_432135 (JIT enabled, AOT enabled) OpenJ9 - f0b6be7 OMR - 18d8f94 IBM - 233dfb5) JCL - 20191016_01 based on Oracle jdk8u231-b10 /usr/java/default/bin/java -server -Xdump:heap+system+snap:none -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.phd -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 -XX:+HeapManagementMXBeanCompatibility -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Dderby.system.home=/usr/ironhide/var/db/staging -Dcom.sun.management.jmxremote -Dcom.approuter.management.jmxremote.connectorserver.port=31997 -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote -Dcom.sun.management.jmxremote.port=1997 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access -Dcom.approuter.logging.filename=derbyserver.log -Djava.rmi.server.hostname=9.124.112.240 -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true -Dcom.approuter.sysconf.appliance=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 -Dosgi.clean=true -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true -Declipse.consoleLog=true -Dcom.approuter.orcmon.purger.compressionmode=sequential -DIH_ROOT=/usr/ironhide -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar [cldegdh9/Standalone log]# cat /proc/meminfo MemTotal: 3872868 kB MemFree: 343700 kB MemAvailable: 374676 kB Buffers: 33100 kB Cached: 170624 kB SwapCached: 3848 kB Active: 2540784 kB Inactive: 710092 kB Active(anon): 2429812 kB Inactive(anon): 621324 kB Active(file): 110972 kB Inactive(file): 88768 kB Unevictable: 20 kB Mlocked: 20 kB SwapTotal: 5345272 kB SwapFree: 5280760 kB Dirty: 1880 kB Writeback: 0 kB AnonPages: 3044312 kB Mapped: 33676 kB Shmem: 3984 kB Slab: 119980 kB SReclaimable: 71120 kB SUnreclaim: 48860 kB KernelStack: 15116 kB PageTables: 13936 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 7281704 kB Committed_AS: 1755044 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 2885632 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 126912 kB DirectMap2M: 4067328 kB > OutOfMemory Exceptions in derby server process > ---------------------------------------------- > > Key: DERBY-7075 > URL: https://issues.apache.org/jira/browse/DERBY-7075 > Project: Derby > Issue Type: Bug > Components: Network Server > Affects Versions: 10.8.3.0, 10.14.2.0 > Reporter: Vamsavardhana Reddy > Priority: Major > Attachments: javaheapdump-oom-derby-101420.zip, > javaheapdump-oom-derby-v10833.zip > > > We have two runtime environments v7.5.4 and v7.5.3 similar in terms of OS > memory, java version and java launch command settings for the derby and other > java processes required for our application in which similar > projects/workloads are running. But in one of the environments we are seeing > out of memory (OOM) exceptions in the derby server java process. In this > environment where OOM is occurring, we even refreshed Apache derby to the > latest version available for Java 8 and still encountered OOM exceptions. > Please help resolve this OOM issue. > > Exception in thread "DRDAConnThread_39" Exception in thread > "DRDAConnThread_41" java/lang/OutOfMemoryError: Java heap space > at java/nio/CharBuffer.wrap (CharBuffer.java:373) > at sun/nio/cs/StreamEncoder.implWrite (StreamEncoder.java:277) > at sun/nio/cs/StreamEncoder.write (StreamEncoder.java:137) > at java/io/OutputStreamWriter.write (OutputStreamWriter.java:244) > at java/io/BufferedWriter.flushBuffer (BufferedWriter.java:140) > at java/io/PrintStream.write (PrintStream.java:550) > at java/io/PrintStream.print (PrintStream.java:693) > at java/io/PrintStream.append (PrintStream.java:1089) > at java/io/PrintStream.append (PrintStream.java:70) > at com/ibm/oti/util/Util.appendTo (Util.java:225) > at java/lang/Throwable.printStackTrace (Throwable.java:506) > at java/lang/Throwable.printStackTrace (Throwable.java:302) > at java/lang/Throwable.printStackTrace (Throwable.java:247) > Exception in thread "DRDAConnThread_43" > java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError(?)java.lang.OutOfMemoryError: > Java heap space > at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:872) > at > java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:8java/lang/OutOfMemoryError: > Java heap space > at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:872) > at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866) > at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866) > at java/lang/Thread.uncaughtException (Thread.java:1332) > ----------------------- > :*v7.5.4* > OS: RHELv8 with 4GB RAM > Java: IBM Java SDK v8.0.6.0 > Apache Derby v10.8.3.3, Apache Derby v10.14.2.0 > [bceglc385/Standalone ~]# /usr/java/default/jre/bin/java -version > java version "1.8.0_231" > Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6)) > IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References > 20191106_432135 (JIT enabled, AOT enabled) > OpenJ9 - f0b6be7 > OMR - 18d8f94 > IBM - 233dfb5) > JCL - 20191016_01 based on Oracle jdk8u231-b10 > > /usr/java/default/bin/java -server -Xdump:heap+system+snap:none > -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.ph > -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 > > -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt > -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron > -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: > -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 > -XX:+HeapManagementMXBeanCompatibility > -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger > -Dderby.system.home=/usr/ironhide/var/db/staging > -Dcom.sun.management.jmxremote > -Dcom.approuter.management.jmxremote.connectorserver.port=31997 > -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote > -Dcom.sun.management.jmxremote.port=1997 > -Dcom.sun.management.jmxremote.local.only=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=true > -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password > > -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access > -Dcom.approuter.logging.filename=derbyserver.log > -Djava.rmi.server.hostname=9.121.242.177 > -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true > -Dcom.approuter.sysconf.appliance=true > -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog > -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true > -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 > -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true > -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 > -Dosgi.clean=true > -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver > -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true > -Declipse.consoleLog=true > -Dcom.approuter.orcmon.purger.compressionmode=sequential > -DIH_ROOT=/usr/ironhide > -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties > -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar > > [bceglc390/Standalone ~]# cat /proc/meminfo > MemTotal: 4057920 kB > MemFree: 152896 kB > Buffers: 342624 kB > Cached: 729112 kB > SwapCached: 3428 kB > Active: 2644068 kB > Inactive: 1066404 kB > Active(anon): 1951704 kB > Inactive(anon): 712696 kB > Active(file): 692364 kB > Inactive(file): 353708 kB > Unevictable: 20 kB > Mlocked: 20 kB > SwapTotal: 3999736 kB > SwapFree: 3988760 kB > Dirty: 720 kB > Writeback: 0 kB > AnonPages: 2636560 kB > Mapped: 38508 kB > Shmem: 25648 kB > Slab: 129600 kB > SReclaimable: 48320 kB > SUnreclaim: 81280 kB > KernelStack: 6568 kB > PageTables: 11116 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 6028696 kB > Committed_AS: 1658244 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 152396 kB > VmallocChunk: 34359582800 kB > AnonHugePages: 2433024 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > DirectMap4k: 4096 kB > DirectMap2M: 4190208 kB > > ---------- > *v7.5.3* > OS: Bedrock v5 which is custom OS based on RHELv7 with 4GB RAM > Java: IBM Java SDK v8.0.6.0 > Apache Derby v10.8.3.3 > > [bceglc390/Standalone ~]# /usr/java/default/jre/bin/java -version > java version "1.8.0_231" > Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6)) > IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References > 20191106_432135 (JIT enabled, AOT enabled) > OpenJ9 - f0b6be7 > OMR - 18d8f94 > IBM - 233dfb5) > JCL - 20191016_01 based on Oracle jdk8u231-b10 > > /usr/java/default/bin/java -server -Xdump:heap+system+snap:none > -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.phd > -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 > > -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt > -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron > -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: > -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 > -XX:+HeapManagementMXBeanCompatibility > -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger > -Dderby.system.home=/usr/ironhide/var/db/staging > -Dcom.sun.management.jmxremote > -Dcom.approuter.management.jmxremote.connectorserver.port=31997 > -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote > -Dcom.sun.management.jmxremote.port=1997 > -Dcom.sun.management.jmxremote.local.only=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=true > -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password > > -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access > -Dcom.approuter.logging.filename=derbyserver.log > -Djava.rmi.server.hostname=9.124.112.240 > -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true > -Dcom.approuter.sysconf.appliance=true > -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog > -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true > -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 > -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true > -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 > -Dosgi.clean=true > -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver > -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true > -Declipse.consoleLog=true > -Dcom.approuter.orcmon.purger.compressionmode=sequential > -DIH_ROOT=/usr/ironhide > -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties > -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar > > [cldegdh9/Standalone log]# cat /proc/meminfo > MemTotal: 3872868 kB > MemFree: 343700 kB > MemAvailable: 374676 kB > Buffers: 33100 kB > Cached: 170624 kB > SwapCached: 3848 kB > Active: 2540784 kB > Inactive: 710092 kB > Active(anon): 2429812 kB > Inactive(anon): 621324 kB > Active(file): 110972 kB > Inactive(file): 88768 kB > Unevictable: 20 kB > Mlocked: 20 kB > SwapTotal: 5345272 kB > SwapFree: 5280760 kB > Dirty: 1880 kB > Writeback: 0 kB > AnonPages: 3044312 kB > Mapped: 33676 kB > Shmem: 3984 kB > Slab: 119980 kB > SReclaimable: 71120 kB > SUnreclaim: 48860 kB > KernelStack: 15116 kB > PageTables: 13936 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 7281704 kB > Committed_AS: 1755044 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 0 kB > VmallocChunk: 0 kB > HardwareCorrupted: 0 kB > AnonHugePages: 2885632 kB > ShmemHugePages: 0 kB > ShmemPmdMapped: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > Hugetlb: 0 kB > DirectMap4k: 126912 kB > DirectMap2M: 4067328 kB -- This message was sent by Atlassian Jira (v8.3.4#803005)