[jira] [Created] (HDFS-7777) Consolidate the HA NN documentation down to one
Allen Wittenauer created HDFS-: -- Summary: Consolidate the HA NN documentation down to one Key: HDFS- URL: https://issues.apache.org/jira/browse/HDFS- Project: Hadoop HDFS Issue Type: Improvement Reporter: Allen Wittenauer These are nearly the same document now. Let's consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-6759) Namenode Web UI only display 1 datanode when multiple datanodes exist on same hostname under different ports
[ https://issues.apache.org/jira/browse/HDFS-6759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai resolved HDFS-6759. -- Resolution: Duplicate Duplicate of HDFS-7303 Namenode Web UI only display 1 datanode when multiple datanodes exist on same hostname under different ports Key: HDFS-6759 URL: https://issues.apache.org/jira/browse/HDFS-6759 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.4.0, 2.4.1 Environment: Centos 6.5 Reporter: Konnjuta Assignee: liu chang Priority: Trivial NameNode Web UI only display 1 datanode when multiple datanodes exist on same hostname under different ports. This seems to affect only version 2.x Version 1.x working as expected. This is a development environment so multiple datanodes on a single hostname is common. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Hadoop-Hdfs-trunk - Build # 2033 - Still Failing
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2033/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 6614 lines...] [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target/test-dir [INFO] Executed tasks [INFO] [INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-hdfs-project --- [INFO] Skipping javadoc generation [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-checkstyle-plugin:2.12.1:checkstyle (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop HDFS FAILURE [ 02:33 h] [INFO] Apache Hadoop HttpFS .. SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED [INFO] Apache Hadoop HDFS-NFS SKIPPED [INFO] Apache Hadoop HDFS Project SUCCESS [ 2.345 s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:33 h [INFO] Finished at: 2015-02-11T14:37:20+00:00 [INFO] Final Memory: 55M/802M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-hdfs: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports for the individual test results. [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Updating YARN-2246 Updating YARN-3160 Updating HADOOP-11559 Updating HADOOP-10953 Updating HDFS-7753 Updating YARN-3090 Updating HDFS-7769 Updating HADOOP-11495 Updating YARN-2809 Sending e-mails to: hdfs-dev@hadoop.apache.org Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 8 tests failed. REGRESSION: org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart Error Message: org.apache.hadoop.util.ExitUtil$ExitException: Could not sync enough journals to persistent storage due to No journals available to flush. Unsynced transactions: 1 at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:629) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1284) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:360) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1216) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1725) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:865) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownNameNode(MiniDFSCluster.java:1760) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1811) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1791) at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:494) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart(TestLeaseRecovery2.java:433) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at
Build failed in Jenkins: Hadoop-Hdfs-trunk #2033
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2033/changes Changes: [jlowe] YARN-3090. DeletionService can silently ignore deletion task failures. Contributed by Varun Saxena [jlowe] YARN-2809. Implement workaround for linux kernel panic when removing cgroup. Contributed by Nathan Roberts [cmccabe] HADOOP-10953. NetworkTopology#add calls NetworkTopology#toString without holding the netlock (Liang Xie via Colin P. McCabe) [aw] HADOOP-11495. Convert site documentation from apt to markdown (Masatake Iwasaki via aw) [zjshen] YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/appId to avoid duplicate sections. Contributed by Devaraj K. [aw] HADOOP-11559. Add links to RackAwareness and InterfaceClassification to site index (Masatake Iwasaki via aw) [szetszwo] HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir. [shv] HDFS-7753. Fix Multithreaded correctness Warnings in BackupImage. Contributed by Rakesh R and Konstantin Shvachko. [junping_du] YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. (Contributed by Chengbing Liu) -- [...truncated 6421 lines...] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126) at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170) at org.apache.hadoop.hdfs.server.namenode.NameNode.doImmediateShutdown(NameNode.java:1694) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1729) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:865) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownNameNode(MiniDFSCluster.java:1760) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1811) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1791) at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:494) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart(TestLeaseRecovery2.java:433) testLeaseRecoverByAnotherUser(org.apache.hadoop.hdfs.TestLeaseRecovery2) Time elapsed: 0 sec ERROR! java.lang.IllegalStateException: Lease monitor is not running at com.google.common.base.Preconditions.checkState(Preconditions.java:145) at org.apache.hadoop.hdfs.server.namenode.LeaseManager.triggerMonitorCheckNow(LeaseManager.java:557) at org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter.setLeasePeriod(NameNodeAdapter.java:135) at org.apache.hadoop.hdfs.MiniDFSCluster.setLeasePeriod(MiniDFSCluster.java:2352) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser(TestLeaseRecovery2.java:159) testHardLeaseRecovery(org.apache.hadoop.hdfs.TestLeaseRecovery2) Time elapsed: 0.002 sec ERROR! java.io.EOFException: End of File Exception between local host is: asf903.gq1.ygridcore.net/67.195.81.147; destination host is: localhost:34627; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at
[jira] [Resolved] (HDFS-7102) Null dereference in PacketReceiver#receiveNextPacket()
[
https://issues.apache.org/jira/browse/HDFS-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HDFS-7102.
--
Resolution: Later
Null dereference in PacketReceiver#receiveNextPacket()
--
Key: HDFS-7102
URL: https://issues.apache.org/jira/browse/HDFS-7102
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void receiveNextPacket(ReadableByteChannel in) throws IOException {
doRead(in, null);
{code}
doRead() would pass null as second parameter to (line 134):
{code}
doReadFully(ch, in, curPacketBuf);
{code}
which dereferences it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-957) FSImage layout version should be only once file is complete
[ https://issues.apache.org/jira/browse/HDFS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-957. -- Resolution: Won't Fix FSImage layout version should be only once file is complete --- Key: HDFS-957 URL: https://issues.apache.org/jira/browse/HDFS-957 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-957.txt Right now, the FSImage save code writes the LAYOUT_VERSION at the head of the file, along with some other headers, and then dumps the directory into the file. Instead, it should write a special IMAGE_IN_PROGRESS entry for the layout version, dump all of the data, then seek back to the head of the file to write the proper LAYOUT_VERSION. This would make it very easy to detect the case where the FSImage save got interrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7778) Rename FsVolumeListTest to TestFsVolumeList and commit it to branch-2
Lei (Eddy) Xu created HDFS-7778:
---
Summary: Rename FsVolumeListTest to TestFsVolumeList and commit it
to branch-2
Key: HDFS-7778
URL: https://issues.apache.org/jira/browse/HDFS-7778
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
HDFS-7496 mistakenly named {{FsVolumeListTest}}, which causes it out of jenkin
tests. Also it mistakenly removed it from branch-2 patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7083) TestDecommission#testIncludeByRegistrationName sometimes fails
[
https://issues.apache.org/jira/browse/HDFS-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HDFS-7083.
--
Resolution: Cannot Reproduce
TestDecommission#testIncludeByRegistrationName sometimes fails
--
Key: HDFS-7083
URL: https://issues.apache.org/jira/browse/HDFS-7083
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/ :
{code}
REGRESSION:
org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
Error Message:
test timed out after 36 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 36 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName(TestDecommission.java:957)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7779) Improve the HDFS Web UI browser to allow chowning / chmoding and setting replication
Ravi Prakash created HDFS-7779: -- Summary: Improve the HDFS Web UI browser to allow chowning / chmoding and setting replication Key: HDFS-7779 URL: https://issues.apache.org/jira/browse/HDFS-7779 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ravi Prakash -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7781) Use block erasure coder in client stripping
Li Bo created HDFS-7781:
---
Summary: Use block erasure coder in client stripping
Key: HDFS-7781
URL: https://issues.apache.org/jira/browse/HDFS-7781
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Li Bo
In HDFS-7729, raw erasure coder is used in order not to depend on the
{{ErasureCoder}} api defined in HDFS-7662 or even {{ErasureCodec}} api defined
in HDFS-7337 since they're still upcoming.
This is a follow up issue to work on that when the high level constructs are
available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7784) load fsimage in parallel
Walter Su created HDFS-7784: --- Summary: load fsimage in parallel Key: HDFS-7784 URL: https://issues.apache.org/jira/browse/HDFS-7784 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Walter Su When single Namenode has huge amount of files, without using federation, the startup/restart speed is slow. The fsimage loading step takes the most of the time. fsimage loading can seperate to two parts, deserialization and object construction(mostly map insertion). Deserialization takes the most of CPU time. So we can do deserialization in parallel, and add to hashmap in parallel. It will significantly reduce the NN start time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7783) Support appending to a striping layout file
Li Bo created HDFS-7783: --- Summary: Support appending to a striping layout file Key: HDFS-7783 URL: https://issues.apache.org/jira/browse/HDFS-7783 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Li Bo Assignee: Li Bo Currently HDFS-7729 focuses on writing a new file. This sub task works on appending to an existing file in striping layout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7782) Read a striping layout file from client side
Li Bo created HDFS-7782: --- Summary: Read a striping layout file from client side Key: HDFS-7782 URL: https://issues.apache.org/jira/browse/HDFS-7782 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Li Bo Assignee: Li Bo If client wants to read a file, he is not necessary to know and handle what layout the file is. This sub task adds logic to DFSInputStream to support reading striping layout files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
