[
https://issues.apache.org/jira/browse/DERBY-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085715#comment-17085715
]
Richard N. Hillegas commented on DERBY-7075:
--------------------------------------------
The attached zips contains files with the .phd extension. I believe that this
format is an IBM dump format. I don't have a tool to read these dumps. Can you
give us an idea of what Derby objects are consuming the memory? Thanks.
> OutOfMemory Exceptions in derby server process
> ----------------------------------------------
>
> Key: DERBY-7075
> URL: https://issues.apache.org/jira/browse/DERBY-7075
> Project: Derby
> Issue Type: Bug
> Components: Network Server
> Affects Versions: 10.8.3.0, 10.14.2.0
> Reporter: Vamsavardhana Reddy
> Priority: Major
> Attachments: javaheapdump-oom-derby-101420.zip,
> javaheapdump-oom-derby-v10833.zip
>
>
> We have two runtime environments v7.5.4 and v7.5.3 similar in terms of OS
> memory, java version and java launch command settings for the derby and other
> java processes required for our application in which similar
> projects/workloads are running. But in one of the environments we are seeing
> out of memory (OOM) exceptions in the derby server java process. In this
> environment where OOM is occurring, we even refreshed Apache derby to the
> latest version available for Java 8 and still encountered OOM exceptions.
> Please help resolve this OOM issue.
>
> Exception in thread "DRDAConnThread_39" Exception in thread
> "DRDAConnThread_41" java/lang/OutOfMemoryError: Java heap space
> at java/nio/CharBuffer.wrap (CharBuffer.java:373)
> at sun/nio/cs/StreamEncoder.implWrite (StreamEncoder.java:277)
> at sun/nio/cs/StreamEncoder.write (StreamEncoder.java:137)
> at java/io/OutputStreamWriter.write (OutputStreamWriter.java:244)
> at java/io/BufferedWriter.flushBuffer (BufferedWriter.java:140)
> at java/io/PrintStream.write (PrintStream.java:550)
> at java/io/PrintStream.print (PrintStream.java:693)
> at java/io/PrintStream.append (PrintStream.java:1089)
> at java/io/PrintStream.append (PrintStream.java:70)
> at com/ibm/oti/util/Util.appendTo (Util.java:225)
> at java/lang/Throwable.printStackTrace (Throwable.java:506)
> at java/lang/Throwable.printStackTrace (Throwable.java:302)
> at java/lang/Throwable.printStackTrace (Throwable.java:247)
> Exception in thread "DRDAConnThread_43"
> java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError(?)java.lang.OutOfMemoryError:
> Java heap space
> at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:872)
> at
> java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:8java/lang/OutOfMemoryError:
> Java heap space
> at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:872)
> at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866)
> at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866)
> at java/lang/Thread.uncaughtException (Thread.java:1332)
> -----------------------
> :*v7.5.4*
> OS: RHELv8 with 4GB RAM
> Java: IBM Java SDK v8.0.6.0
> Apache Derby v10.8.3.3, Apache Derby v10.14.2.0
> [bceglc385/Standalone ~]# /usr/java/default/jre/bin/java -version
> java version "1.8.0_231"
> Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6))
> IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References
> 20191106_432135 (JIT enabled, AOT enabled)
> OpenJ9 - f0b6be7
> OMR - 18d8f94
> IBM - 233dfb5)
> JCL - 20191016_01 based on Oracle jdk8u231-b10
>
> /usr/java/default/bin/java -server -Xdump:heap+system+snap:none
> -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.ph
> -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1
>
> -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt
> -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron
> -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar:
> -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000
> -XX:+HeapManagementMXBeanCompatibility
> -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger
> -Dderby.system.home=/usr/ironhide/var/db/staging
> -Dcom.sun.management.jmxremote
> -Dcom.approuter.management.jmxremote.connectorserver.port=31997
> -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote
> -Dcom.sun.management.jmxremote.port=1997
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=true
> -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password
>
> -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access
> -Dcom.approuter.logging.filename=derbyserver.log
> -Djava.rmi.server.hostname=9.121.242.177
> -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true
> -Dcom.approuter.sysconf.appliance=true
> -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog
> -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true
> -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30
> -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true
> -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10
> -Dosgi.clean=true
> -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver
> -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true
> -Declipse.consoleLog=true
> -Dcom.approuter.orcmon.purger.compressionmode=sequential
> -DIH_ROOT=/usr/ironhide
> -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties
> -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar
>
> [bceglc390/Standalone ~]# cat /proc/meminfo
> MemTotal: 4057920 kB
> MemFree: 152896 kB
> Buffers: 342624 kB
> Cached: 729112 kB
> SwapCached: 3428 kB
> Active: 2644068 kB
> Inactive: 1066404 kB
> Active(anon): 1951704 kB
> Inactive(anon): 712696 kB
> Active(file): 692364 kB
> Inactive(file): 353708 kB
> Unevictable: 20 kB
> Mlocked: 20 kB
> SwapTotal: 3999736 kB
> SwapFree: 3988760 kB
> Dirty: 720 kB
> Writeback: 0 kB
> AnonPages: 2636560 kB
> Mapped: 38508 kB
> Shmem: 25648 kB
> Slab: 129600 kB
> SReclaimable: 48320 kB
> SUnreclaim: 81280 kB
> KernelStack: 6568 kB
> PageTables: 11116 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 6028696 kB
> Committed_AS: 1658244 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 152396 kB
> VmallocChunk: 34359582800 kB
> AnonHugePages: 2433024 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 4096 kB
> DirectMap2M: 4190208 kB
>
> ----------
> *v7.5.3*
> OS: Bedrock v5 which is custom OS based on RHELv7 with 4GB RAM
> Java: IBM Java SDK v8.0.6.0
> Apache Derby v10.8.3.3
>
> [bceglc390/Standalone ~]# /usr/java/default/jre/bin/java -version
> java version "1.8.0_231"
> Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6))
> IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References
> 20191106_432135 (JIT enabled, AOT enabled)
> OpenJ9 - f0b6be7
> OMR - 18d8f94
> IBM - 233dfb5)
> JCL - 20191016_01 based on Oracle jdk8u231-b10
>
> /usr/java/default/bin/java -server -Xdump:heap+system+snap:none
> -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.phd
> -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1
>
> -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt
> -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron
> -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar:
> -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000
> -XX:+HeapManagementMXBeanCompatibility
> -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger
> -Dderby.system.home=/usr/ironhide/var/db/staging
> -Dcom.sun.management.jmxremote
> -Dcom.approuter.management.jmxremote.connectorserver.port=31997
> -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote
> -Dcom.sun.management.jmxremote.port=1997
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=true
> -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password
>
> -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access
> -Dcom.approuter.logging.filename=derbyserver.log
> -Djava.rmi.server.hostname=9.124.112.240
> -[Djava.net|https://slack-redir.net/link?url=http%3A%2F%2FDjava.net].preferIPv4Stack=true
> -Dcom.approuter.sysconf.appliance=true
> -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog
> -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true
> -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30
> -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true
> -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10
> -Dosgi.clean=true
> -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver
> -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true
> -Declipse.consoleLog=true
> -Dcom.approuter.orcmon.purger.compressionmode=sequential
> -DIH_ROOT=/usr/ironhide
> -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties
> -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar
>
> [cldegdh9/Standalone log]# cat /proc/meminfo
> MemTotal: 3872868 kB
> MemFree: 343700 kB
> MemAvailable: 374676 kB
> Buffers: 33100 kB
> Cached: 170624 kB
> SwapCached: 3848 kB
> Active: 2540784 kB
> Inactive: 710092 kB
> Active(anon): 2429812 kB
> Inactive(anon): 621324 kB
> Active(file): 110972 kB
> Inactive(file): 88768 kB
> Unevictable: 20 kB
> Mlocked: 20 kB
> SwapTotal: 5345272 kB
> SwapFree: 5280760 kB
> Dirty: 1880 kB
> Writeback: 0 kB
> AnonPages: 3044312 kB
> Mapped: 33676 kB
> Shmem: 3984 kB
> Slab: 119980 kB
> SReclaimable: 71120 kB
> SUnreclaim: 48860 kB
> KernelStack: 15116 kB
> PageTables: 13936 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 7281704 kB
> Committed_AS: 1755044 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 0 kB
> VmallocChunk: 0 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 2885632 kB
> ShmemHugePages: 0 kB
> ShmemPmdMapped: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> Hugetlb: 0 kB
> DirectMap4k: 126912 kB
> DirectMap2M: 4067328 kB
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
