[jira] [Created] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
Ravindra Babu created HDFS-8914: --- Summary: Documentation conflict regarding fail-over of Namenode Key: HDFS-8914 URL: https://issues.apache.org/jira/browse/HDFS-8914 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.7.1 Environment: Documentation page in live Reporter: Ravindra Babu Priority: Trivial Please refer to these two links and correct one of them. http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html The NameNode machine is a single point of failure for an HDFS cluster. If the NameNode machine fails, manual intervention is necessary. Currently, automatic restart and failover of the NameNode software to another machine is not supported. http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html The HDFS High Availability feature addresses the above problems by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This allows a fast failover to a new NameNode in the case that a machine crashes, or a graceful administrator-initiated failover for the purpose of planned maintenance. Please keep one right version regarding failover statements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8913) Documentation correction regarding Secondary node, Checkpoint node Backup node
Ravindra Babu created HDFS-8913: --- Summary: Documentation correction regarding Secondary node, Checkpoint node Backup node Key: HDFS-8913 URL: https://issues.apache.org/jira/browse/HDFS-8913 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.7.1 Environment: Content in documentation Reporter: Ravindra Babu Assignee: Ravindra Babu Priority: Minor Fix For: 3.0.0 I checked with many people and almost all of them are confused on responsibilities of Secondary Node, Checkpoint Node and Backup node. Link: http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html Confusion: Secondary NameNode The NameNode stores modifications to the file system as a log appended to a native file system file, edits. When a NameNode starts up, it reads HDFS state from an image file, fsimage, and then applies edits from the edits log file. It then writes new HDFS state to the fsimage and starts normal operation with an empty edits file. Since NameNode merges fsimage and edits files only during start up, the edits log file could get very large over time on a busy cluster. Another side effect of a larger edits file is that next restart of NameNode takes longer. Checkpoint Node NameNode persists its namespace using two files: fsimage, which is the latest checkpoint of the namespace and edits, a journal (log) of changes to the namespace since the checkpoint. When a NameNode starts up, it merges the fsimage and edits journal to provide an up-to-date view of the file system metadata. The NameNode then overwrites fsimage with the new HDFS state and begins a new edits journal. Backup Node The Backup node provides the same checkpointing functionality as the Checkpoint node, as well as maintaining an in-memory, up-to-date copy of the file system namespace that is always synchronized with the active NameNode state. Along with accepting a journal stream of file system edits from the NameNode and persisting this to disk, the Backup node also applies those edits into its own copy of the namespace in memory, thus creating a backup of the namespace. Now all three nodes have overlapping functionalities. To add confusion to this point, http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html quotes that NameNode will never make RPC call to other nodes. The Communication Protocols All HDFS communication protocols are layered on top of the TCP/IP protocol. A client establishes a connection to a configurable TCP port on the NameNode machine. It talks the ClientProtocol with the NameNode. The DataNodes talk to the NameNode using the DataNode Protocol. A Remote Procedure Call (RPC) abstraction wraps both the Client Protocol and the DataNode Protocol. By design, the NameNode never initiates any RPCs. Instead, it only responds to RPC requests issued by DataNodes or clients. We need clarification regarding these points. Please enhance your documentation to avoid confusion among readers. 1) Secondary Node, Check point Node Backup node - Clear separation of roles 2) For High Availability, do we require only One of them Or Two of them or All of them? If it's not all of them, what combination is allowed? 3) Without RPC by Name node to data nodes, how writes and read are happening? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8918) Convert BlockUnderConstructionFeature#replicas form list to array
Zhe Zhang created HDFS-8918:
---
Summary: Convert BlockUnderConstructionFeature#replicas form list
to array
Key: HDFS-8918
URL: https://issues.apache.org/jira/browse/HDFS-8918
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.8.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
{{BlockInfoUnderConstruction}} / {{BlockUnderConstructionFeature}} uses a List
to store its {{replicas}}. To reduce memory usage, we can use an array instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8835) Convert BlockInfoUnderConstruction as an interface
[
https://issues.apache.org/jira/browse/HDFS-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang resolved HDFS-8835.
-
Resolution: Invalid
HDFS-8801 has converted {{BlockInfoUC}} as a feature.
Convert BlockInfoUnderConstruction as an interface
--
Key: HDFS-8835
URL: https://issues.apache.org/jira/browse/HDFS-8835
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.7.1
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Per discussion under HDFS-8499, this JIRA aims to convert
{{BlockInfoUnderConstruction}} as an interface and
{{BlockInfoContiguousUnderConstruction}} as its implementation. The HDFS-7285
branch will add {{BlockInfoStripedUnderConstruction}} as another
implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8915) TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins
Anu Engineer created HDFS-8915: -- Summary: TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins Key: HDFS-8915 URL: https://issues.apache.org/jira/browse/HDFS-8915 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Affects Versions: 2.8.0 Reporter: Anu Engineer Assignee: Anu Engineer This test was added as part of HDFS-8883, There is a race condition in the test and it has failed *once* in the Apache Jenkins run. Here is the stack FAILED: org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount Error Message: Expected number of blocked thread not found expected:3 but was:1 Stack Trace: java.lang.AssertionError: Expected number of blocked thread not found expected:3 but was:1 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount(TestFSNamesystem.java:261) From cursory code reading , even though we call into readlock.lock() there is no guarantee that our code is put in the wait queue. A proposed fix could be to check for any thread in the lock queue instead of all 3, or disable the test. It could also indicate an issue with the test infra-structure but any test open to variations in result due to infra-structure issues creates noise in tests so we are better off fixing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8916) add nonDfsUsedSpace back to Name Node UI column
Chang Li created HDFS-8916: -- Summary: add nonDfsUsedSpace back to Name Node UI column Key: HDFS-8916 URL: https://issues.apache.org/jira/browse/HDFS-8916 Project: Hadoop HDFS Issue Type: Bug Reporter: Chang Li Assignee: Chang Li nonDfsUsedSpace was taken out in HDFS-8816. Though currently we can see this info in the pop up but we lost the ability to sort them. So propose to add non dfs usage back to column in namenode UI -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8917) Cleanup BlockInfoUnderConstruction from comments and tests
Zhe Zhang created HDFS-8917:
---
Summary: Cleanup BlockInfoUnderConstruction from comments and tests
Key: HDFS-8917
URL: https://issues.apache.org/jira/browse/HDFS-8917
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.8.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
HDFS-8801 eliminates the {{BlockInfoUnderConstruction}} class. This JIRA is a
follow-on to cleanup comments and tests which refer to the class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Hadoop-Hdfs-trunk-Java8 - Build # 280 - Still Failing
See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/280/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7938 lines...] [INFO] [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-hdfs-project --- [INFO] Not executing Javadoc as the project is not a Java classpath-capable package [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop HDFS Client . SUCCESS [03:01 min] [INFO] Apache Hadoop HDFS FAILURE [ 03:30 h] [INFO] Apache Hadoop HttpFS .. SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED [INFO] Apache Hadoop HDFS-NFS SKIPPED [INFO] Apache Hadoop HDFS Project SUCCESS [ 0.055 s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 03:33 h [INFO] Finished at: 2015-08-18T17:35:07+00:00 [INFO] Final Memory: 63M/616M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-hdfs: There was a timeout or other error in the fork - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-hdfs Build step 'Execute shell' marked build as failure Archiving artifacts Sending artifact delta relative to Hadoop-Hdfs-trunk-Java8 #222 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 4313432 bytes Compression is 0.0% Took 12 sec Recording test results Updating HDFS-6407 Updating HDFS-8862 Updating HDFS-8792 Updating HDFS-8845 Updating HDFS-8801 Updating HDFS-8895 Updating HDFS-8880 Updating HDFS-8713 Updating HDFS-8883 Updating HDFS-8278 Sending e-mails to: hdfs-dev@hadoop.apache.org Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 3 tests failed. REGRESSION: org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure.testReplaceDatanodeOnFailure Error Message: expected:3 but was:2 Stack Trace: java.lang.AssertionError: expected:3 but was:2 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure$SlowWriter.checkReplication(TestReplaceDatanodeOnFailure.java:235) at org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure.testReplaceDatanodeOnFailure(TestReplaceDatanodeOnFailure.java:154) REGRESSION: org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateWithDataNodesRestartImmediately Error Message: inode should complete in ~3 ms. Expected: is true but: was false Stack Trace: java.lang.AssertionError: inode should complete in ~3 ms. Expected: is true but: was false at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.junit.Assert.assertThat(Assert.java:865) at org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.checkBlockRecovery(TestFileTruncate.java:1187) at org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.checkBlockRecovery(TestFileTruncate.java:1171) at org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.checkBlockRecovery(TestFileTruncate.java:1166) at org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateWithDataNodesRestartImmediately(TestFileTruncate.java:794) FAILED: org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes.testCircularLinkedListWrites Error Message: Some
Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #280
See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/280/changes Changes: [xyao] HDFS-8883. NameNode Metrics : Add FSNameSystem lock Queue Length. Contributed by Anu Engineer. [wang] HDFS-8713. Convert DatanodeDescriptor to use SLF4J logging. [wheat9] HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. Contributed by Haohui Mai. [wheat9] HDFS-8801. Convert BlockInfoUnderConstruction as a feature. Contributed by Jing Zhao. [cmccabe] HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) [cmccabe] HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via Colin P. McCabe) [arp] HDFS-8880. NameNode metrics logging. (Arpit Agarwal) [wang] HDFS-8895. Remove deprecated BlockStorageLocation APIs. [szetszwo] HDFS-8278. When computing max-size-to-move in Balancer, count only the storage with remaining = default block size. [yliu] HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) -- [...truncated 7745 lines...] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.772 sec - in org.apache.hadoop.cli.TestCryptoAdminCLI Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.cli.TestAclCLI Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.157 sec - in org.apache.hadoop.cli.TestAclCLI Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.cli.TestXAttrCLI Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.404 sec - in org.apache.hadoop.cli.TestXAttrCLI Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.cli.TestCacheAdminCLI Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.764 sec - in org.apache.hadoop.cli.TestCacheAdminCLI Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.TestRefreshCallQueue Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.036 sec - in org.apache.hadoop.TestRefreshCallQueue Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.TestXAttr Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.085 sec - in org.apache.hadoop.fs.TestXAttr Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.TestWebHdfsFileContextMainOperations Tests run: 60, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.255 sec - in org.apache.hadoop.fs.TestWebHdfsFileContextMainOperations Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.TestFcHdfsSetUMask Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.571 sec - in org.apache.hadoop.fs.TestFcHdfsSetUMask Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.795 sec - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractConcat Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.948 sec - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.941 sec - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRename Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractSetTimes Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.81 sec - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractSetTimes Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.931 sec - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractOpen Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractAppend Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.933 sec - in
[jira] [Created] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
Rui Li created HDFS-8920:
Summary: Erasure Coding: when recovering lost blocks, logs can be
too verbose and hurt performance
Key: HDFS-8920
URL: https://issues.apache.org/jira/browse/HDFS-8920
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rui Li
When we test reading data with datanodes killed,
{{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
effectively blocks the client JVM. This log seems too verbose:
{code}
if (chosenNode == null) {
DFSClient.LOG.warn(No live nodes contain block + block.getBlock() +
after checking nodes = + Arrays.toString(nodes) +
, ignoredNodes = + ignoredNodes);
return null;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8921) Add an option to Balancer so tha it only uses the k-most over-utilized DNs or all over-utilized DNs as sources.
Tsz Wo Nicholas Sze created HDFS-8921: - Summary: Add an option to Balancer so tha it only uses the k-most over-utilized DNs or all over-utilized DNs as sources. Key: HDFS-8921 URL: https://issues.apache.org/jira/browse/HDFS-8921 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer mover Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Arpit suggested to add a separate option to source from the most over-utilized DataNodes first so the administrator does not have to pass the source DNs manually; see [this comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576]. The new option could allow specifying the k-most over-utilized DNs or all over-utilized DNs as sources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8919) Use SLF4j in ProtobufRpcEngine class
Xiaoyu Yao created HDFS-8919: Summary: Use SLF4j in ProtobufRpcEngine class Key: HDFS-8919 URL: https://issues.apache.org/jira/browse/HDFS-8919 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor This are considerable amount of debug/trace level logs in this class. This ticket is opened to convert it to use SLF4J for better performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8912) Implement ShrinkableHashMap extends java HashMap and use properly
[
https://issues.apache.org/jira/browse/HDFS-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HDFS-8912.
--
Resolution: Invalid
I forgot one thing: many variables/methods of java {{HashMap}} we need to touch
in {{ShrinkableHashMap}} are {{package-private}}, so we can't simply extend
{{HashMap}}. Implementing a new {{ShrinkableHashMap}} is a bit heavy, close it
as Invalid.
Implement ShrinkableHashMap extends java HashMap and use properly
-
Key: HDFS-8912
URL: https://issues.apache.org/jira/browse/HDFS-8912
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Currently {{LightWeightHashSet}} and {{LightWeightLinkedSet}} are used in
hdfs, there are two advantages compared to java HashSet: one is the entry
requires fewer memory, another is it's shrinkable. In real cluster, hdfs is
a long running service, and {{set}} may become large at some time and may
become small after that, so shrinking the {{set}} when size hits the shrink
threshold is necessary, it can improve the NN memory.
Same situation for {{map}}, some HashMap used in BlockManager (e.g., the
hashmap in CorruptReplicasMap), it's better to be shrinkable.
I think it's worth to implement ShrinkableHashMap extends the java HashMap,
for quick glance, seems few code is needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Build failed in Jenkins: Hadoop-Hdfs-trunk #2218
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2218/changes Changes: [xyao] HDFS-8883. NameNode Metrics : Add FSNameSystem lock Queue Length. Contributed by Anu Engineer. [wang] HDFS-8713. Convert DatanodeDescriptor to use SLF4J logging. [wheat9] HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. Contributed by Haohui Mai. [wheat9] HDFS-8801. Convert BlockInfoUnderConstruction as a feature. Contributed by Jing Zhao. [cmccabe] HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) [cmccabe] HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via Colin P. McCabe) [arp] HDFS-8880. NameNode metrics logging. (Arpit Agarwal) [wang] HDFS-8895. Remove deprecated BlockStorageLocation APIs. [szetszwo] HDFS-8278. When computing max-size-to-move in Balancer, count only the storage with remaining = default block size. [yliu] HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) -- [...truncated 7798 lines...] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.126 sec - in org.apache.hadoop.hdfs.TestBlockReaderLocalLegacy Running org.apache.hadoop.hdfs.security.TestDelegationToken Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.253 sec - in org.apache.hadoop.hdfs.security.TestDelegationToken Running org.apache.hadoop.hdfs.security.token.block.TestBlockToken Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.911 sec - in org.apache.hadoop.hdfs.security.token.block.TestBlockToken Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.332 sec - in org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser Running org.apache.hadoop.hdfs.security.TestClientProtocolWithDelegationToken Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.842 sec - in org.apache.hadoop.hdfs.security.TestClientProtocolWithDelegationToken Running org.apache.hadoop.hdfs.TestFileCorruption Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.711 sec - in org.apache.hadoop.hdfs.TestFileCorruption Running org.apache.hadoop.hdfs.TestDFSInputStream Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.114 sec - in org.apache.hadoop.hdfs.TestDFSInputStream Running org.apache.hadoop.hdfs.TestFileAppend4 Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.059 sec - in org.apache.hadoop.hdfs.TestFileAppend4 Running org.apache.hadoop.hdfs.TestHdfsAdmin Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.118 sec - in org.apache.hadoop.hdfs.TestHdfsAdmin Running org.apache.hadoop.hdfs.client.impl.TestLeaseRenewer Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.47 sec - in org.apache.hadoop.hdfs.client.impl.TestLeaseRenewer Running org.apache.hadoop.hdfs.TestSnapshotCommands Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.621 sec - in org.apache.hadoop.hdfs.TestSnapshotCommands Running org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.551 sec - in org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream Running org.apache.hadoop.hdfs.TestIsMethodSupported Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.311 sec - in org.apache.hadoop.hdfs.TestIsMethodSupported Running org.apache.hadoop.hdfs.TestDFSStorageStateRecovery Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.298 sec - in org.apache.hadoop.hdfs.TestDFSStorageStateRecovery Running org.apache.hadoop.hdfs.TestGetFileChecksum Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.037 sec - in org.apache.hadoop.hdfs.TestGetFileChecksum Running org.apache.hadoop.hdfs.TestFileConcurrentReader Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.012 sec - in org.apache.hadoop.hdfs.TestFileConcurrentReader Running org.apache.hadoop.hdfs.util.TestBestEffortLongFile Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.279 sec - in org.apache.hadoop.hdfs.util.TestBestEffortLongFile Running org.apache.hadoop.hdfs.util.TestByteArrayManager Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.21 sec - in org.apache.hadoop.hdfs.util.TestByteArrayManager Running org.apache.hadoop.hdfs.util.TestLightWeightLinkedSet Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.256 sec - in org.apache.hadoop.hdfs.util.TestLightWeightLinkedSet Running org.apache.hadoop.hdfs.util.TestMD5FileUtils Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.328 sec - in org.apache.hadoop.hdfs.util.TestMD5FileUtils Running org.apache.hadoop.hdfs.util.TestExactSizeInputStream Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec - in org.apache.hadoop.hdfs.util.TestExactSizeInputStream Running
Hadoop-Hdfs-trunk - Build # 2218 - Still Failing
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2218/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7991 lines...] [INFO] [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-hdfs-project --- [INFO] Skipping javadoc generation [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] --- findbugs-maven-plugin:3.0.0:findbugs (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop HDFS Client . SUCCESS [03:05 min] [INFO] Apache Hadoop HDFS FAILURE [ 02:55 h] [INFO] Apache Hadoop HttpFS .. SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED [INFO] Apache Hadoop HDFS-NFS SKIPPED [INFO] Apache Hadoop HDFS Project SUCCESS [ 0.060 s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 02:58 h [INFO] Finished at: 2015-08-18T16:42:45+00:00 [INFO] Final Memory: 64M/715M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-hdfs: There was a timeout or other error in the fork - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-hdfs Build step 'Execute shell' marked build as failure Archiving artifacts Sending artifact delta relative to Hadoop-Hdfs-trunk #2199 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 3829685 bytes Compression is 0.0% Took 5.6 sec Recording test results Updating HDFS-6407 Updating HDFS-8862 Updating HDFS-8792 Updating HDFS-8845 Updating HDFS-8801 Updating HDFS-8895 Updating HDFS-8880 Updating HDFS-8713 Updating HDFS-8883 Updating HDFS-8278 Sending e-mails to: hdfs-dev@hadoop.apache.org Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 5 tests failed. FAILED: org.apache.hadoop.hdfs.TestRenameWhileOpen.initializationError Error Message: org/apache/hadoop/conf/Configuration Stack Trace: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2531) at java.lang.Class.getMethod0(Class.java:2774) at java.lang.Class.getMethod(Class.java:1663) at org.junit.internal.builders.SuiteMethodBuilder.hasSuiteMethod(SuiteMethodBuilder.java:18) at org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:10) at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59) at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26) at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59) at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
