[ https://issues.apache.org/jira/browse/HDDS-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anu Engineer resolved HDDS-2308. -------------------------------- Fix Version/s: 0.5.0 Resolution: Fixed Committed to the build branch. > Switch to centos with the apache/ozone-build docker image > --------------------------------------------------------- > > Key: HDDS-2308 > URL: https://issues.apache.org/jira/browse/HDDS-2308 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Reporter: Marton Elek > Assignee: Marton Elek > Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: hs_err_pid16346.log > > Time Spent: 20m > Remaining Estimate: 0h > > I realized multiple JVM crashes in the daily builds: > > {code:java} > ERROR] ExecutionException The forked VM terminated without properly saying > goodbye. VM crash or System.exit called? > > > [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && > /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m > -XX:+HeapDumpOnOutOfMemoryError -jar > /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar > /workdir/hadoop-ozone/ozonefs/target/surefire > 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp > surefire_947955725320624341206tmp > > > [ERROR] Error occurred in starting fork, check output in log > > > [ERROR] Process Exit Code: 139 > > > [ERROR] Crashed tests: > > > [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename > > > [ERROR] ExecutionException The forked VM terminated without properly > saying goodbye. VM crash or System.exit called? > > > [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && > /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m > -XX:+HeapDumpOnOutOfMemoryError -jar > /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter5429192218879128313.jar > /workdir/hadoop-ozone/ozonefs/target/surefire > 2019-10-06T14-52-40_697-jvmRun1 surefire7227403571189445391tmp > surefire_1011197392458143645283tmp > > > [ERROR] Error occurred in starting fork, check output in log > > > [ERROR] Process Exit Code: 139 > > > [ERROR] Crashed tests: > > > [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDistCp > > > [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: > ExecutionException The forked VM terminated without properly saying goodbye. > VM crash or System.exit called? > > > [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && > /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m > -XX:+HeapDumpOnOutOfMemoryError -jar > /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter1355604543311368443.jar > /workdir/hadoop-ozone/ozonefs/target/surefire > 2019-10-06T14-52-40_697-jvmRun1 surefire3938612864214747736tmp > surefire_933162535733309260236tmp > > > [ERROR] Error occurred in starting fork, check output in log > > > [ERROR] Process Exit Code: 139 > > > [ERROR] ExecutionException The forked VM terminated without properly > saying goodbye. VM crash or System.exit called? > > > [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs && > /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m > -XX:+HeapDumpOnOutOfMemoryError -jar > /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar > /workdir/hadoop-ozone/ozonefs/target/surefire > 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp > surefire_947955725320624341206tmp > > > [ERROR] Error occurred in starting fork, check output in log > > > [ERROR] Process Exit Code: 139 {code} > > Based on the crash log (uploaded) it's related to the rocksdb JNI interface. > In the current ozone-build docker image (which provides the environment for > build) we use alpine where musl libc is used instead of the main glibc. I > think it would be more safe to use the same glibc what is used in production. > I tested with centos based docker image and it seems to be more stable. > Didn't see any more JVM crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org