[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614912#comment-14614912
]
Rakesh R commented on HDFS-8399:
[~vinayrpet], [~hitliuyi], Anything else requires for this jira ? Great to see
feedback. Thanks!
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Labels: Test
Attachments: HDFS-8399-HDFS-7285-00.patch,
HDFS-8399-HDFS-7285-01.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573979#comment-14573979
]
Rakesh R commented on HDFS-8399:
As per the previous discussion, I have labelled this as {{Test}} task. Could
you please review the patch which only adds unit test case, it would be great
to see the feedback. Thanks!
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Labels: Test
Attachments: HDFS-8399-HDFS-7285-00.patch,
HDFS-8399-HDFS-7285-01.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567182#comment-14567182
]
Vinayakumar B commented on HDFS-8399:
-
[~rakeshr], do you want to rebase the patch including only tests?
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567412#comment-14567412
]
Rakesh R commented on HDFS-8399:
Thank you [~vinayrpet] for the interest. I've re-based the unit test case and
attached in the jira. Would you be interested in reviewing it.
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch,
HDFS-8399-HDFS-7285-01.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567747#comment-14567747
]
Hadoop QA commented on HDFS-8399:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 5m 30s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 24s | There were no new javac warning
messages. |
| {color:red}-1{color} | release audit | 0m 13s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 36s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 21s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 1m 30s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 173m 57s | Tests failed in hadoop-hdfs. |
| | | 194m 47s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736577/HDFS-8399-HDFS-7285-01.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 1299357 |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11189/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11189/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11189/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11189/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11189/console |
This message was automatically generated.
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Labels: Test
Attachments: HDFS-8399-HDFS-7285-00.patch,
HDFS-8399-HDFS-7285-01.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567787#comment-14567787
]
Rakesh R commented on HDFS-8399:
Note: Test case failures are unrelated to my patch. Newly added test case is
passing,
[jenkins_test_report|https://builds.apache.org/job/PreCommit-HDFS-Build/11189/testReport/org.apache.hadoop.hdfs.server.namenode/TestStripedFile/]
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Labels: Test
Attachments: HDFS-8399-HDFS-7285-00.patch,
HDFS-8399-HDFS-7285-01.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558732#comment-14558732
]
Yi Liu commented on HDFS-8399:
--
Hi [~rakeshr], I just noticed your comments, sorry for late response, I will
take a look at it, thanks.
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555655#comment-14555655
]
Rakesh R commented on HDFS-8399:
ping [~hitliuyi]
It would be great to see your feedback on my previous comment. If you agree,
then I'm happy to contribute the test case alone otherwise will close this
issue as this doesn't exists after branch merge. Thanks!
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547596#comment-14547596
]
Rakesh R commented on HDFS-8399:
Yes, this issue has been resolved after the branch merge. Could you please take
a look at the test case, is it worth adding this to our unit test suite?
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547493#comment-14547493
]
Yi Liu commented on HDFS-8399:
--
Please check the latest branch after merge, it should not exist.
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543659#comment-14543659
]
Hadoop QA commented on HDFS-8399:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 48s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 47s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 14s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 16s | The patch appears to introduce 8
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 189m 1s | Tests failed in hadoop-hdfs. |
| | | 230m 52s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 252] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 913-915] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.createErasureCodingZone(String,
ECSchema):in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.createErasureCodingZone(String,
ECSchema): String.getBytes() At ErasureCodingZoneManager.java:[line 117] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.getECZoneInfo(INodesInPath):in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.getECZoneInfo(INodesInPath):
new String(byte[]) At ErasureCodingZoneManager.java:[line 81] |
| | Dead store to dataBlkNum in
org.apache.hadoop.hdfs.util.StripedBlockUtil.calcualteChunkPositionsInBuf(ECSchema,
LocatedStripedBlock, byte[], int, int, int, int, long, int,
StripedBlockUtil$AlignedStripe[]) At
StripedBlockUtil.java:org.apache.hadoop.hdfs.util.StripedBlockUtil.calcualteChunkPositionsInBuf(ECSchema,
LocatedStripedBlock, byte[], int, int, int, int, long, int,
StripedBlockUtil$AlignedStripe[]) At StripedBlockUtil.java:[line 467] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.planReadPortions(int, int, long,
int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.planReadPortions(int, int, long,
int, int) At StripedBlockUtil.java:[line 206] |
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.util.TestStripedBlockUtil |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.tracing.TestTraceAdmin |
| | hadoop.hdfs.server.datanode.TestTriggerBlockReport |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| Timed out tests | org.apache.hadoop.hdfs.TestDatanodeDeath |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732819/HDFS-8399-HDFS-7285-00.patch
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14544146#comment-14544146
]
Rakesh R commented on HDFS-8399:
It looks like test case failure is unrelated to my patch. Also, the findbugs
will be handled in HDFS-8294.
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543370#comment-14543370
]
Rakesh R commented on HDFS-8399:
IIUC the cause is, {{INodeFile#destroyAndCollectBlocks()}} function is
considering only {{BlockInfoContiguous}} block, instead it should do the delete
logic on the blocks irrespective of the contiguous or striped.
{code}
if (blks != null reclaimContext.collectedBlocks != null) {
for (BlockInfo blk : blocks) {
reclaimContext.collectedBlocks.addDeleteBlock(blk);
blk.setBlockCollection(null);
}
}
{code}
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8399) Erasure Coding: BlockManager is unnecessarily computing recovery work for the deleted blocks
[
https://issues.apache.org/jira/browse/HDFS-8399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543427#comment-14543427
]
Rakesh R commented on HDFS-8399:
I have tried re-producing it through unit test case, please review the scenario
and the proposal to fix the same. Thanks!
Erasure Coding: BlockManager is unnecessarily computing recovery work for the
deleted blocks
Key: HDFS-8399
URL: https://issues.apache.org/jira/browse/HDFS-8399
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8399-HDFS-7285-00.patch
Following exception occurred in the {{ReplicationMonitor}}. As per the
initial analysis, I could see the exception is coming for the blocks of the
deleted file.
{code}
2015-05-14 14:14:40,485 FATAL util.ExitUtil (ExitUtil.java:terminate(127)) -
Terminate called
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
Exception in thread
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1255079
org.apache.hadoop.util.ExitUtil$ExitException: java.lang.AssertionError:
Absolute path required
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:744)
at
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:723)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath(FSDirectory.java:1655)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getECSchemaForPath(FSNamesystem.java:8435)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRecoveryWorkForBlocks(BlockManager.java:1572)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockRecoveryWork(BlockManager.java:1402)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3894)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3846)
at java.lang.Thread.run(Thread.java:722)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3865)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
