[
https://issues.apache.org/jira/browse/HADOOP-15711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated HADOOP-15711:
-----------------------------------
Description:
Branch-2 builds have been disabled for a while:
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
A test run here causes hdfs tests to hang:
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
Running hadoop-hdfs tests locally reveal some errors such as:{noformat}[ERROR]
testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed: 0.059
s <<< ERROR!
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:514)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
at
org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
at
org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
I was able to get more tests passing locally by increasing the max user process
count on my machine. But the error suggests that there's an issue in the tests
themselves. Not sure if the error seen locally is the same reason as why
jenkins builds are failing, I wasn't able to confirm based on the jenkins
builds' lack of output.
was:
Branch-2 builds have been disabled for a while:
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
A test run here causes hdfs tests to hang:
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
Running hadoop-hdfs tests locally reveal some errors such as:{noformat}[ERROR]
testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed: 0.059
s <<< ERROR!
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:514)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
at
org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
at
org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
I was able to get more tests passing locally by increasing the max user process
count on my machine. But the error suggests that there's an issue in the tests
themselves.
> Fix branch-2 builds
> -------------------
>
> Key: HADOOP-15711
> URL: https://issues.apache.org/jira/browse/HADOOP-15711
> Project: Hadoop Common
> Issue Type: Task
> Reporter: Jonathan Hung
> Priority: Critical
>
> Branch-2 builds have been disabled for a while:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> A test run here causes hdfs tests to hang:
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86-jhung/4/
> Running hadoop-hdfs tests locally reveal some errors such
> as:{noformat}[ERROR]
> testComplexAppend2(org.apache.hadoop.hdfs.TestFileAppend2) Time elapsed:
> 0.059 s <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1164)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1128)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1172)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:403)
> at
> org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:234)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1080)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:883)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:514)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:473)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend(TestFileAppend2.java:489)
> at
> org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend2(TestFileAppend2.java:543)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){noformat}
> I was able to get more tests passing locally by increasing the max user
> process count on my machine. But the error suggests that there's an issue in
> the tests themselves. Not sure if the error seen locally is the same reason
> as why jenkins builds are failing, I wasn't able to confirm based on the
> jenkins builds' lack of output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
