[jira] [Created] (HDFS-4875) Add a test for testing snapshot file length
Tsz Wo (Nicholas), SZE created HDFS-4875: Summary: Add a test for testing snapshot file length Key: HDFS-4875 URL: https://issues.apache.org/jira/browse/HDFS-4875 Project: Hadoop HDFS Issue Type: Test Components: snapshots, test Reporter: Tsz Wo (Nicholas), SZE Assignee: Arpit Agarwal Priority: Minor Here is a test idea from Suresh: - A file was length x at the time of snapshot (say s1) creation - Later y bytes get added to the file through append When client gets block locations for the file from snapshot s1, the length it know is x. We should make sure it cannot read beyond length x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
Tsz Wo (Nicholas), SZE created HDFS-4876: Summary: The javadoc of FileWithSnapshot is incorrect Key: HDFS-4876 URL: https://issues.apache.org/jira/browse/HDFS-4876 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor The javadoc said that snapshot files and the original file form a circular linked list. It is no longer true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4875) Add a test for testing snapshot file length
[ https://issues.apache.org/jira/browse/HDFS-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4875: - Issue Type: Sub-task (was: Test) Parent: HDFS-2802 > Add a test for testing snapshot file length > --- > > Key: HDFS-4875 > URL: https://issues.apache.org/jira/browse/HDFS-4875 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots, test >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Agarwal >Priority: Minor > > Here is a test idea from Suresh: > - A file was length x at the time of snapshot (say s1) creation > - Later y bytes get added to the file through append > When client gets block locations for the file from snapshot s1, the length it > know is x. We should make sure it cannot read beyond length x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
[ https://issues.apache.org/jira/browse/HDFS-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4876: - Fix Version/s: (was: 2.1.0-beta) > The javadoc of FileWithSnapshot is incorrect > > > Key: HDFS-4876 > URL: https://issues.apache.org/jira/browse/HDFS-4876 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > > The javadoc said that snapshot files and the original file form a circular > linked list. It is no longer true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
[ https://issues.apache.org/jira/browse/HDFS-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4876: - Attachment: h4876_20130604.patch h4876_20130604.patch: fixes the javadoc and moves FileDiffList to FileWithSnapshot. > The javadoc of FileWithSnapshot is incorrect > > > Key: HDFS-4876 > URL: https://issues.apache.org/jira/browse/HDFS-4876 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h4876_20130604.patch > > > The javadoc said that snapshot files and the original file form a circular > linked list. It is no longer true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[
https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674150#comment-13674150
]
Steve Loughran commented on HDFS-4872:
--
bq. Just mark delete idempotent. A delete retry may delete an object that has
been recreated or replaced between the retries in this case.
This isn't idempotent, so doesn't meet the requirement of the JIRA.
Inode-driven delete could work. You do have to get that inode ID first, but you
can then be confident that there is no change to that file between that
(presumably retrying) operation and the delete call. It's also testable with
ease: create a file, get the inode, overwrite the file, delete by inode, expect
failure.
One funny is what if the file by inode is moved -but you don't want to delete a
file if it has been moved.
that would imply the operation shouldn't be {{delete(inode)}} but instead
{{delete(path, inode)}}.
> Idempotent delete operation.
>
>
> Key: HDFS-4872
> URL: https://issues.apache.org/jira/browse/HDFS-4872
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.0.4-alpha
>Reporter: Konstantin Shvachko
>
> Making delete idempotent is important to provide uninterrupted job execution
> in case of HA failover.
> This is to discuss different approaches to idempotent implementation of
> delete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[
https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674168#comment-13674168
]
Steve Loughran commented on HDFS-4872:
--
BTW this isn't something that other filesystems can be expected to support, so
can't go into the FS APIs -it would have to be hidden behind the normal
{{delete()}} operation. The low-level calls should be {{@VisibleForTesting}}
only.
> Idempotent delete operation.
>
>
> Key: HDFS-4872
> URL: https://issues.apache.org/jira/browse/HDFS-4872
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.0.4-alpha
>Reporter: Konstantin Shvachko
>
> Making delete idempotent is important to provide uninterrupted job execution
> in case of HA failover.
> This is to discuss different approaches to idempotent implementation of
> delete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
[ https://issues.apache.org/jira/browse/HDFS-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4876: - Status: Patch Available (was: Open) > The javadoc of FileWithSnapshot is incorrect > > > Key: HDFS-4876 > URL: https://issues.apache.org/jira/browse/HDFS-4876 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h4876_20130604.patch > > > The javadoc said that snapshot files and the original file form a circular > linked list. It is no longer true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[
https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674254#comment-13674254
]
Hudson commented on HDFS-3934:
--
Integrated in Hadoop-Yarn-trunk #230 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/230/])
Remove extra code that code erroneously committed in HDFS-3934 (cmccabe)
(Revision 1489083)
Add needed file for HDFS-3934 (cmccabe) (Revision 1489068)
HDFS-3934. duplicative dfs_hosts entries handled wrong. (cmccabe) (Revision
1489065)
Result = FAILURE
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489083
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HostFileManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489068
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HostFileManager.java
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489065
Files :
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
> duplicative dfs_hosts entries handled wrong
> ---
>
> Key: HDFS-3934
> URL: https://issues.apache.org/jira/browse/HDFS-3934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Andy Isaacson
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch,
> HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch,
> HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch,
> HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch,
> HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch,
> HDFS-3934.016.patch, HDFS-3934.017.patch
>
>
> A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by
> hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}}
> after the NN restarts because {{getDatanodeListForReport}} does not handle
> such a "pseudo-duplicate" correctly:
> # the "Remove any nodes we know about from the map" loop no longer has the
> knowledge to remove the spurious entries
> # the "The remaining nodes are ones that are referenced by the hosts files"
> loop does not do hostname lookups, so does not know that the IP and hostname
> refer to the same host.
> Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in
> the JSP output: The *Node* column shows ":50010" as the nodename, with HTML
> markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020";
> title="172.29.97.216:50010">:50010}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-4862) SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually
[
https://issues.apache.org/jira/browse/HDFS-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash reassigned HDFS-4862:
--
Assignee: Ravi Prakash
> SafeModeInfo.isManual() returns true when resources are low even if it wasn't
> entered into manually
> ---
>
> Key: HDFS-4862
> URL: https://issues.apache.org/jira/browse/HDFS-4862
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-4862.patch
>
>
> HDFS-1594 changed isManual to this
> {code}
> private boolean isManual() {
> return extension == Integer.MAX_VALUE && !resourcesLow;
> }
> {code}
> One immediate impact of this is that when resources are low, the NN will
> throw away all block reports from DNs. This is undesirable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4862) SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually
[
https://issues.apache.org/jira/browse/HDFS-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-4862:
---
Status: Patch Available (was: Open)
> SafeModeInfo.isManual() returns true when resources are low even if it wasn't
> entered into manually
> ---
>
> Key: HDFS-4862
> URL: https://issues.apache.org/jira/browse/HDFS-4862
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0
>Reporter: Ravi Prakash
> Attachments: HDFS-4862.patch
>
>
> HDFS-1594 changed isManual to this
> {code}
> private boolean isManual() {
> return extension == Integer.MAX_VALUE && !resourcesLow;
> }
> {code}
> One immediate impact of this is that when resources are low, the NN will
> throw away all block reports from DNs. This is undesirable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
[
https://issues.apache.org/jira/browse/HDFS-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674305#comment-13674305
]
Hadoop QA commented on HDFS-4876:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586064/h4876_20130604.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4471//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4471//console
This message is automatically generated.
> The javadoc of FileWithSnapshot is incorrect
>
>
> Key: HDFS-4876
> URL: https://issues.apache.org/jira/browse/HDFS-4876
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Minor
> Attachments: h4876_20130604.patch
>
>
> The javadoc said that snapshot files and the original file form a circular
> linked list. It is no longer true.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[
https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674345#comment-13674345
]
Hudson commented on HDFS-3934:
--
Integrated in Hadoop-Hdfs-trunk #1420 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/])
Remove extra code that code erroneously committed in HDFS-3934 (cmccabe)
(Revision 1489083)
Add needed file for HDFS-3934 (cmccabe) (Revision 1489068)
HDFS-3934. duplicative dfs_hosts entries handled wrong. (cmccabe) (Revision
1489065)
Result = FAILURE
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489083
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HostFileManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489068
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HostFileManager.java
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489065
Files :
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
> duplicative dfs_hosts entries handled wrong
> ---
>
> Key: HDFS-3934
> URL: https://issues.apache.org/jira/browse/HDFS-3934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Andy Isaacson
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch,
> HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch,
> HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch,
> HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch,
> HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch,
> HDFS-3934.016.patch, HDFS-3934.017.patch
>
>
> A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by
> hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}}
> after the NN restarts because {{getDatanodeListForReport}} does not handle
> such a "pseudo-duplicate" correctly:
> # the "Remove any nodes we know about from the map" loop no longer has the
> knowledge to remove the spurious entries
> # the "The remaining nodes are ones that are referenced by the hosts files"
> loop does not do hostname lookups, so does not know that the IP and hostname
> refer to the same host.
> Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in
> the JSP output: The *Node* column shows ":50010" as the nodename, with HTML
> markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020";
> title="172.29.97.216:50010">:50010}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4862) SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually
[
https://issues.apache.org/jira/browse/HDFS-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674356#comment-13674356
]
Hadoop QA commented on HDFS-4862:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585996/HDFS-4862.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestSafeMode
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4472//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4472//console
This message is automatically generated.
> SafeModeInfo.isManual() returns true when resources are low even if it wasn't
> entered into manually
> ---
>
> Key: HDFS-4862
> URL: https://issues.apache.org/jira/browse/HDFS-4862
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-4862.patch
>
>
> HDFS-1594 changed isManual to this
> {code}
> private boolean isManual() {
> return extension == Integer.MAX_VALUE && !resourcesLow;
> }
> {code}
> One immediate impact of this is that when resources are low, the NN will
> throw away all block reports from DNs. This is undesirable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674371#comment-13674371 ] Suresh Srinivas commented on HDFS-4872: --- bq. that would imply the operation shouldn't be delete(inode) but instead delete(path, inode). Good point. I had not thought of this. > Idempotent delete operation. > > > Key: HDFS-4872 > URL: https://issues.apache.org/jira/browse/HDFS-4872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > Making delete idempotent is important to provide uninterrupted job execution > in case of HA failover. > This is to discuss different approaches to idempotent implementation of > delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4874) create with OVERWRITE deletes existing file without checking the lease: feature or a bug.
[ https://issues.apache.org/jira/browse/HDFS-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp resolved HDFS-4874. --- Resolution: Not A Problem Hadoop Flags: Incompatible change I fully agree with Suresh. An overwrite is semantically a delete & create. Since delete isn't influenced by a lease, neither should an overwrite. Having a lease prevent overwrite/delete implies the lease somewhat extends to the directory contents (delete is a directory level operation) which doesn't make since a file lease is a file level lock. > create with OVERWRITE deletes existing file without checking the lease: > feature or a bug. > - > > Key: HDFS-4874 > URL: https://issues.apache.org/jira/browse/HDFS-4874 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > create with OVERWRITE flag will remove a file under construction even if the > issuing client does not hold a lease on the file. > It could be a bug or the feature that applications rely upon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[
https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674413#comment-13674413
]
Hudson commented on HDFS-3934:
--
Integrated in Hadoop-Mapreduce-trunk #1446 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/])
Remove extra code that code erroneously committed in HDFS-3934 (cmccabe)
(Revision 1489083)
Add needed file for HDFS-3934 (cmccabe) (Revision 1489068)
HDFS-3934. duplicative dfs_hosts entries handled wrong. (cmccabe) (Revision
1489065)
Result = SUCCESS
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489083
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HostFileManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489068
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/HostFileManager.java
cmccabe :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489065
Files :
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
> duplicative dfs_hosts entries handled wrong
> ---
>
> Key: HDFS-3934
> URL: https://issues.apache.org/jira/browse/HDFS-3934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.1-alpha
>Reporter: Andy Isaacson
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch,
> HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch,
> HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch,
> HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch,
> HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch,
> HDFS-3934.016.patch, HDFS-3934.017.patch
>
>
> A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by
> hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}}
> after the NN restarts because {{getDatanodeListForReport}} does not handle
> such a "pseudo-duplicate" correctly:
> # the "Remove any nodes we know about from the map" loop no longer has the
> knowledge to remove the spurious entries
> # the "The remaining nodes are ones that are referenced by the hosts files"
> loop does not do hostname lookups, so does not know that the IP and hostname
> refer to the same host.
> Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in
> the JSP output: The *Node* column shows ":50010" as the nodename, with HTML
> markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020";
> title="172.29.97.216:50010">:50010}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674448#comment-13674448 ] Daryn Sharp commented on HDFS-4872: --- I don't think there is a good approach that is portable across filesystems. One cited drawback of rename/delete is the two rpc requests. Won't delete(path, inode) have the same drawback? Ie. 1 rpc to get inode + 1 rpc to issue delete. > Idempotent delete operation. > > > Key: HDFS-4872 > URL: https://issues.apache.org/jira/browse/HDFS-4872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > Making delete idempotent is important to provide uninterrupted job execution > in case of HA failover. > This is to discuss different approaches to idempotent implementation of > delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4862) SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually
[
https://issues.apache.org/jira/browse/HDFS-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HDFS-4862:
---
Attachment: HDFS-4862.patch
Reverted changes in getTurnOffTip(). That madness needs to be fixed in another
JIRA.
> SafeModeInfo.isManual() returns true when resources are low even if it wasn't
> entered into manually
> ---
>
> Key: HDFS-4862
> URL: https://issues.apache.org/jira/browse/HDFS-4862
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-4862.patch, HDFS-4862.patch
>
>
> HDFS-1594 changed isManual to this
> {code}
> private boolean isManual() {
> return extension == Integer.MAX_VALUE && !resourcesLow;
> }
> {code}
> One immediate impact of this is that when resources are low, the NN will
> throw away all block reports from DNs. This is undesirable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4869) FsShell get command does not support writing to stdout
[
https://issues.apache.org/jira/browse/HDFS-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cristina L. Abad updated HDFS-4869:
---
Attachment: 4869.branch-0.23.patch
>From the jiras it wasn't clear that the change in the behavior of -get was
>intentional, so I thought it was a bug. In any case, I have a patch (for 0.23)
>that would re-implement the functionality without bloating the code too much.
>Daryn, can you take a look at the patch? If you think this is a good way to do
>it, then I can post patches for trunk and branch 2. I did try this patch with
>many different scenarios and it worked fine in all my tests.
> FsShell get command does not support writing to stdout
> --
>
> Key: HDFS-4869
> URL: https://issues.apache.org/jira/browse/HDFS-4869
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta, 0.23.8
>Reporter: Cristina L. Abad
>Assignee: Cristina L. Abad
>Priority: Minor
> Attachments: 4869.branch-0.23.patch
>
>
> In FsShell the put command supports using "\-" in place of stdin, but this
> functionality ("\-" in place of stdout) is broken in the get command.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674517#comment-13674517 ] Suresh Srinivas commented on HDFS-4872: --- bq. One cited drawback of rename/delete is the two rpc requests. Won't delete(path, inode) have the same drawback? Ie. 1 rpc to get inode + 1 rpc to issue delete. Yes. What I meant In my previous comment is, if rename + delete is a solution that is being considered, which requires two requests anyway, get inode + delete combination, which also requires two requests is a better solution. Among many pluses, one important one is, it doesn't need to change file system state twice. Thus the boundary conditions such as after rename, if the client dies, no cleanup on namenode is needed. > Idempotent delete operation. > > > Key: HDFS-4872 > URL: https://issues.apache.org/jira/browse/HDFS-4872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > Making delete idempotent is important to provide uninterrupted job execution > in case of HA failover. > This is to discuss different approaches to idempotent implementation of > delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
[ https://issues.apache.org/jira/browse/HDFS-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674563#comment-13674563 ] Jing Zhao commented on HDFS-4876: - +1 for the patch. > The javadoc of FileWithSnapshot is incorrect > > > Key: HDFS-4876 > URL: https://issues.apache.org/jira/browse/HDFS-4876 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Attachments: h4876_20130604.patch > > > The javadoc said that snapshot files and the original file form a circular > linked list. It is no longer true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4862) SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually
[
https://issues.apache.org/jira/browse/HDFS-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674562#comment-13674562
]
Hadoop QA commented on HDFS-4862:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586114/HDFS-4862.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4473//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4473//console
This message is automatically generated.
> SafeModeInfo.isManual() returns true when resources are low even if it wasn't
> entered into manually
> ---
>
> Key: HDFS-4862
> URL: https://issues.apache.org/jira/browse/HDFS-4862
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-4862.patch, HDFS-4862.patch
>
>
> HDFS-1594 changed isManual to this
> {code}
> private boolean isManual() {
> return extension == Integer.MAX_VALUE && !resourcesLow;
> }
> {code}
> One immediate impact of this is that when resources are low, the NN will
> throw away all block reports from DNs. This is undesirable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4871) Skip failing commons tests on Windows
[ https://issues.apache.org/jira/browse/HDFS-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674633#comment-13674633 ] Chuan Liu commented on HDFS-4871: - +1 on the approach to enable CI. [~arpitagarwal], do you know if we need to do something similar for Yarn, mapreduce tests as well? Or is this limited to common tests? Thanks! > Skip failing commons tests on Windows > - > > Key: HDFS-4871 > URL: https://issues.apache.org/jira/browse/HDFS-4871 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.1.0-beta > > > This is a temporary fix proposed to get CI working. We will skip the > following failing tests on Windows: > # TestChRootedFs > # TestFSMainOperationsLocalFileSystem > # TestFcCreateMkdirLocalFs > # TestFcMainOperationsLocalFs > # TestFcPermissionsLocalFs > # TestLocalFSFileContextSymlink - HADOOP-9527 > # TestLocalFileSystem > # TestShellCommandFencer - HADOOP-9526 > # TestSocketIOWithTimeout - HADOOP-8982 > # TestViewFsLocalFs > # TestViewFsTrash > # TestViewFsWithAuthorityLocalFs > The tests will be re-enabled as we fix each. JIRAs for remaining failing > tests to follow soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4871) Skip failing commons tests on Windows
[ https://issues.apache.org/jira/browse/HDFS-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674642#comment-13674642 ] Arpit Agarwal commented on HDFS-4871: - Thanks Chuan. No we'd need to do this for HDFS, MR and Yarn tests too. Once we get past the commons build failure we can address the remaining ones. However I am running into some problems with using assumptions to skip tests in branch-2, I am not sure if this is due to a different version of jUnit. I am looking into it. > Skip failing commons tests on Windows > - > > Key: HDFS-4871 > URL: https://issues.apache.org/jira/browse/HDFS-4871 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.1.0-beta > > > This is a temporary fix proposed to get CI working. We will skip the > following failing tests on Windows: > # TestChRootedFs > # TestFSMainOperationsLocalFileSystem > # TestFcCreateMkdirLocalFs > # TestFcMainOperationsLocalFs > # TestFcPermissionsLocalFs > # TestLocalFSFileContextSymlink - HADOOP-9527 > # TestLocalFileSystem > # TestShellCommandFencer - HADOOP-9526 > # TestSocketIOWithTimeout - HADOOP-8982 > # TestViewFsLocalFs > # TestViewFsTrash > # TestViewFsWithAuthorityLocalFs > The tests will be re-enabled as we fix each. JIRAs for remaining failing > tests to follow soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
Jing Zhao created HDFS-4877: --- Summary: Snapshot: fix the scenario where a directory is renamed under its prior descendant Key: HDFS-4877 URL: https://issues.apache.org/jira/browse/HDFS-4877 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Critical Suppose we have the following operations: 1. rename /dir1/foo/bar to /dir2/bar2 2. rename /dir1/foo to /dir2/bar2/foo2 I.e., we rename a directory (foo) to a child of its prior descendant (bar). If we have taken snapshots on root before the above 2 rename operations, a circle can be generated consisting of nodes with the types INodeReference.WithName and INodeReference.DstReference: WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) When deleting a snapshot before the rename operations, the current code may hit an infinite loop when cleaning the snapshot data or updating the quota usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[ https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674648#comment-13674648 ] Plamen Jeliazkov commented on HDFS-4867: New patch with Konstantin's comments taken up. > metaSave NPEs when there are invalid blocks in repl queue. > -- > > Key: HDFS-4867 > URL: https://issues.apache.org/jira/browse/HDFS-4867 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Plamen Jeliazkov > Fix For: 3.0.0 > > Attachments: HDFS-4867.trunk.patch > > > Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs > and stops generating further report. Normally ReplicationMonitor removes them > quickly, but if the queue is huge, it takes very long time. Also in safe > mode, they stay. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[ https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-4867: --- Attachment: HDFS-4867.trunk.patch > metaSave NPEs when there are invalid blocks in repl queue. > -- > > Key: HDFS-4867 > URL: https://issues.apache.org/jira/browse/HDFS-4867 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Plamen Jeliazkov > Fix For: 3.0.0 > > Attachments: HDFS-4867.trunk.patch, HDFS-4867.trunk.patch > > > Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs > and stops generating further report. Normally ReplicationMonitor removes them > quickly, but if the queue is huge, it takes very long time. Also in safe > mode, they stay. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4871) Skip failing commons tests on Windows
[ https://issues.apache.org/jira/browse/HDFS-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674661#comment-13674661 ] Ivan Mitic commented on HDFS-4871: -- Thanks Arpit, +1 on the proposal. Getting CI working on Windows is a great step forward for Hadoop on Windows! bq. JIRAs for remaining failing tests to follow soon. My personal preference would be to file a single parent task Jira, and as we fix individual tests we can link their Jiras to the parent Jira. bq. However I am running into some problems with using assumptions to skip tests in branch-2, I am not sure if this is due to a different version of jUnit. I am looking into it. Can't we do this in a less intrusive manner, for example in test patch (or Windows equivalent), or some other script external to test code? > Skip failing commons tests on Windows > - > > Key: HDFS-4871 > URL: https://issues.apache.org/jira/browse/HDFS-4871 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.1.0-beta > > > This is a temporary fix proposed to get CI working. We will skip the > following failing tests on Windows: > # TestChRootedFs > # TestFSMainOperationsLocalFileSystem > # TestFcCreateMkdirLocalFs > # TestFcMainOperationsLocalFs > # TestFcPermissionsLocalFs > # TestLocalFSFileContextSymlink - HADOOP-9527 > # TestLocalFileSystem > # TestShellCommandFencer - HADOOP-9526 > # TestSocketIOWithTimeout - HADOOP-8982 > # TestViewFsLocalFs > # TestViewFsTrash > # TestViewFsWithAuthorityLocalFs > The tests will be re-enabled as we fix each. JIRAs for remaining failing > tests to follow soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[ https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674670#comment-13674670 ] Jing Zhao commented on HDFS-4877: - The main idea to fix is based on the observation that the nodes in the circle do not co-exist in the same snapshot. For example, if we took snapshot s1 on root before the rename operations, foo and bar's WithName nodes are included in s1, while the new foo2's DstReference node is not included in s1. Therefore we can break the circle while snapshot deletion if we only clean those nodes that are included in the snapshot. > Snapshot: fix the scenario where a directory is renamed under its prior > descendant > -- > > Key: HDFS-4877 > URL: https://issues.apache.org/jira/browse/HDFS-4877 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Critical > > Suppose we have the following operations: > 1. rename /dir1/foo/bar to /dir2/bar2 > 2. rename /dir1/foo to /dir2/bar2/foo2 > I.e., we rename a directory (foo) to a child of its prior descendant (bar). > If we have taken snapshots on root before the above 2 rename operations, a > circle can be generated consisting of nodes with the types > INodeReference.WithName and INodeReference.DstReference: > WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> > WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) > When deleting a snapshot before the rename operations, the current code may > hit an infinite loop when cleaning the snapshot data or updating the quota > usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
Tao Luo created HDFS-4878: - Summary: On Remove Block, Block is not Removed from neededReplications queue Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1825) Remove thriftfs contrib
[ https://issues.apache.org/jira/browse/HDFS-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674685#comment-13674685 ] Chu Tong commented on HDFS-1825: After removing hdfs thrift server, is there any alternative to access hdfs via thrift like service? > Remove thriftfs contrib > --- > > Key: HDFS-1825 > URL: https://issues.apache.org/jira/browse/HDFS-1825 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Nigel Daley >Assignee: Nigel Daley > Fix For: 0.22.0 > > Attachments: HDFS-1825.patch > > > As per vote on general@ > (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3cef44cfe2-692f-4956-8b33-d125d05e2...@mac.com%3E) > thriftfs can be removed: > svn remove hdfs/trunk/src/contrib/thriftfs > and wiki updated: > http://wiki.apache.org/hadoop/Attic -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[ https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4877: Status: Patch Available (was: Open) > Snapshot: fix the scenario where a directory is renamed under its prior > descendant > -- > > Key: HDFS-4877 > URL: https://issues.apache.org/jira/browse/HDFS-4877 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Critical > Attachments: HDFS-4877.001.patch > > > Suppose we have the following operations: > 1. rename /dir1/foo/bar to /dir2/bar2 > 2. rename /dir1/foo to /dir2/bar2/foo2 > I.e., we rename a directory (foo) to a child of its prior descendant (bar). > If we have taken snapshots on root before the above 2 rename operations, a > circle can be generated consisting of nodes with the types > INodeReference.WithName and INodeReference.DstReference: > WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> > WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) > When deleting a snapshot before the rename operations, the current code may > hit an infinite loop when cleaning the snapshot data or updating the quota > usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4491) Parallel testing HDFS
[ https://issues.apache.org/jira/browse/HDFS-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated HDFS-4491: -- Status: Patch Available (was: Open) > Parallel testing HDFS > - > > Key: HDFS-4491 > URL: https://issues.apache.org/jira/browse/HDFS-4491 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 3.0.0 >Reporter: Tsuyoshi OZAWA >Assignee: Andrey Klochkov > Attachments: HDFS-4491--n2.patch, HDFS-4491--n3.patch, > HDFS-4491--n4.patch, HDFS-4491--n5.patch, HDFS-4491--n6.patch, > HDFS-4491--n7.patch, HDFS-4491--n8.patch, HDFS-4491--n9.patch, HDFS-4491.patch > > > Parallel execution of HDFS tests in multiple forks. See HADOOP-9287 for > details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[ https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4877: Attachment: HDFS-4877.001.patch Initial patch for review. > Snapshot: fix the scenario where a directory is renamed under its prior > descendant > -- > > Key: HDFS-4877 > URL: https://issues.apache.org/jira/browse/HDFS-4877 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Critical > Attachments: HDFS-4877.001.patch > > > Suppose we have the following operations: > 1. rename /dir1/foo/bar to /dir2/bar2 > 2. rename /dir1/foo to /dir2/bar2/foo2 > I.e., we rename a directory (foo) to a child of its prior descendant (bar). > If we have taken snapshots on root before the above 2 rename operations, a > circle can be generated consisting of nodes with the types > INodeReference.WithName and INodeReference.DstReference: > WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> > WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) > When deleting a snapshot before the rename operations, the current code may > hit an infinite loop when cleaning the snapshot data or updating the quota > usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675139#comment-13675139 ] Ravi Prakash commented on HDFS-1172: I am able to consistently reproduce this issue with the following command on an 80 node cluster: hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar SliveTest -baseDir /user/someUser/slive -duration 120 -dirSize 122500 -files 122500 -maps 560 -reduces 1 -seed 1 -ops 100 -readSize 1048576,1048576 -writeSize 1048576,1048576 -appendSize 1048576,1048576 -replication 1,1 -blockSize 1024,1024 -delete 0,uniform -create 100,uniform -mkdir 0,uniform -rename 0,uniform -append 0,uniform -ls 0,uniform -read 0,uniform This litters the task logs with the NotReplicatedYetException at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1268) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:469) > Blocks in newly completed files are considered under-replicated too quickly > --- > > Key: HDFS-1172 > URL: https://issues.apache.org/jira/browse/HDFS-1172 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.21.0 >Reporter: Todd Lipcon > Fix For: 0.24.0 > > Attachments: HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt, > replicateBlocksFUC1.patch, replicateBlocksFUC1.patch, replicateBlocksFUC.patch > > > I've seen this for a long time, and imagine it's a known issue, but couldn't > find an existing JIRA. It often happens that we see the NN schedule > replication on the last block of files very quickly after they're completed, > before the other DNs in the pipeline have a chance to report the new block. > This results in a lot of extra replication work on the cluster, as we > replicate the block and then end up with multiple excess replicas which are > very quickly deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4879) Add "blocked ArrayList" collection to avoid CMS full GCs
Todd Lipcon created HDFS-4879:
-
Summary: Add "blocked ArrayList" collection to avoid CMS full GCs
Key: HDFS-4879
URL: https://issues.apache.org/jira/browse/HDFS-4879
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.0.4-alpha, 3.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
We recently saw an issue where a large deletion was issued which caused 25M
blocks to be collected during {{deleteInternal}}. Currently, the list of
collected blocks is an ArrayList, meaning that we had to allocate a contiguous
25M-entry array (~400MB). After a NN has been running for a long amount of
time, the old generation may become fragmented such that it's hard to find a
400MB contiguous chunk of heap.
In general, we should try to design the NN such that the only large objects are
long-lived and created at startup time. We can improve this particular case
(and perhaps some others) by introducing a new List implementation which is
made of a linked list of arrays, each of which is size-limited (eg to 1MB).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[
https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675172#comment-13675172
]
Hadoop QA commented on HDFS-4867:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586139/HDFS-4867.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4474//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4474//console
This message is automatically generated.
> metaSave NPEs when there are invalid blocks in repl queue.
> --
>
> Key: HDFS-4867
> URL: https://issues.apache.org/jira/browse/HDFS-4867
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.7, 2.0.4-alpha
>Reporter: Kihwal Lee
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0
>
> Attachments: HDFS-4867.trunk.patch, HDFS-4867.trunk.patch
>
>
> Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs
> and stops generating further report. Normally ReplicationMonitor removes them
> quickly, but if the queue is huge, it takes very long time. Also in safe
> mode, they stay.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675183#comment-13675183 ] Arpit Agarwal commented on HDFS-4872: - I like the get INode+Delete approach for reasons already mentioned. timestamps are unreliable without synced clocks and RequestID would require server overhead. Rename+Delete is inferior to get INode+Delete because it requires two write operations to the filesystem and introduces more failure modes that need to be handled. e.g. if the rename succeeds but the delete call is lost before being processed by the NameNode, how do we handle retries? There was some discussion about return codes in HDFS-4849 and here are the possibilities: # INode exists, path matched - delete and return true. # INode exists, path does not match - return false. # INode ID not found - presumably the NameNode should assume that the INode has already been deleted and return true? > Idempotent delete operation. > > > Key: HDFS-4872 > URL: https://issues.apache.org/jira/browse/HDFS-4872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > Making delete idempotent is important to provide uninterrupted job execution > in case of HA failover. > This is to discuss different approaches to idempotent implementation of > delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[ https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4877: Priority: Blocker (was: Critical) > Snapshot: fix the scenario where a directory is renamed under its prior > descendant > -- > > Key: HDFS-4877 > URL: https://issues.apache.org/jira/browse/HDFS-4877 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Blocker > Attachments: HDFS-4877.001.patch > > > Suppose we have the following operations: > 1. rename /dir1/foo/bar to /dir2/bar2 > 2. rename /dir1/foo to /dir2/bar2/foo2 > I.e., we rename a directory (foo) to a child of its prior descendant (bar). > If we have taken snapshots on root before the above 2 rename operations, a > circle can be generated consisting of nodes with the types > INodeReference.WithName and INodeReference.DstReference: > WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> > WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) > When deleting a snapshot before the rename operations, the current code may > hit an infinite loop when cleaning the snapshot data or updating the quota > usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[ https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4877: Affects Version/s: 2.1.0-beta > Snapshot: fix the scenario where a directory is renamed under its prior > descendant > -- > > Key: HDFS-4877 > URL: https://issues.apache.org/jira/browse/HDFS-4877 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Critical > Attachments: HDFS-4877.001.patch > > > Suppose we have the following operations: > 1. rename /dir1/foo/bar to /dir2/bar2 > 2. rename /dir1/foo to /dir2/bar2/foo2 > I.e., we rename a directory (foo) to a child of its prior descendant (bar). > If we have taken snapshots on root before the above 2 rename operations, a > circle can be generated consisting of nodes with the types > INodeReference.WithName and INodeReference.DstReference: > WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> > WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) > When deleting a snapshot before the rename operations, the current code may > hit an infinite loop when cleaning the snapshot data or updating the quota > usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4491) Parallel testing HDFS
[
https://issues.apache.org/jira/browse/HDFS-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675210#comment-13675210
]
Hadoop QA commented on HDFS-4491:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584742/HDFS-4491--n9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 92 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1160 javac
compiler warnings (more than the trunk's current 1155 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4475//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4475//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4475//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4475//console
This message is automatically generated.
> Parallel testing HDFS
> -
>
> Key: HDFS-4491
> URL: https://issues.apache.org/jira/browse/HDFS-4491
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0
>Reporter: Tsuyoshi OZAWA
>Assignee: Andrey Klochkov
> Attachments: HDFS-4491--n2.patch, HDFS-4491--n3.patch,
> HDFS-4491--n4.patch, HDFS-4491--n5.patch, HDFS-4491--n6.patch,
> HDFS-4491--n7.patch, HDFS-4491--n8.patch, HDFS-4491--n9.patch, HDFS-4491.patch
>
>
> Parallel execution of HDFS tests in multiple forks. See HADOOP-9287 for
> details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4879) Add "blocked ArrayList" collection to avoid CMS full GCs
[
https://issues.apache.org/jira/browse/HDFS-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-4879:
--
Attachment: hdfs-4879.txt
Attached patch implements the simple data structure and integrates it in the
delete and deleteSnapshot cases. There may be a few other cases where this
would be helpful, but nothing sprang to mind immediately.
The patch looks large mostly because I had to change a bunch of "List"
to "ChunkedArrayList". I decided this was better than making the new
data structure implement the whole List interface, given that we only need very
few of the list operations (add, iterate, and clear).
The patch has a unit test for the data structure which includes a simple
performance test - performance is comparable to ArrayList as expected for our
use cases.
The changes to the NN code itself are covered by existing tests. I added some
log info to the new data structure code and ran TestLargeDirectoryDelete and
verified that it was creating multiple "chunks" in the data structure.
> Add "blocked ArrayList" collection to avoid CMS full GCs
>
>
> Key: HDFS-4879
> URL: https://issues.apache.org/jira/browse/HDFS-4879
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.0.4-alpha
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: hdfs-4879.txt
>
>
> We recently saw an issue where a large deletion was issued which caused 25M
> blocks to be collected during {{deleteInternal}}. Currently, the list of
> collected blocks is an ArrayList, meaning that we had to allocate a
> contiguous 25M-entry array (~400MB). After a NN has been running for a long
> amount of time, the old generation may become fragmented such that it's hard
> to find a 400MB contiguous chunk of heap.
> In general, we should try to design the NN such that the only large objects
> are long-lived and created at startup time. We can improve this particular
> case (and perhaps some others) by introducing a new List implementation which
> is made of a linked list of arrays, each of which is size-limited (eg to 1MB).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675234#comment-13675234 ] Konstantin Shvachko commented on HDFS-4872: --- I missed to include Suresh's proposal of deleting by inode id, sorry. Will edit the summary now to include it. > Idempotent delete operation. > > > Key: HDFS-4872 > URL: https://issues.apache.org/jira/browse/HDFS-4872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > Making delete idempotent is important to provide uninterrupted job execution > in case of HA failover. > This is to discuss different approaches to idempotent implementation of > delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HDFS-4872) Idempotent delete operation.
[ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13673589#comment-13673589 ] Konstantin Shvachko edited comment on HDFS-4872 at 6/4/13 9:08 PM: --- This discussion started in HDFS-4849. Where we discussed -four- five approaches so far # Just mark delete idempotent. [A delete retry may delete an object that has been recreated or replaced between the retries in this case.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670142&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670142] # Design ["duplicate request cache"|http://www.freesoft.org/CIE/RFC/1813/47.htm]. This is applicable to almost any operation and semantics. Drawbacks: complexity, performance, and RAM overhead. [See last two paragraphs.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538] # Add modTime parameter to delete operation with the meaning that the object is deleted only if its modification time is <= than modTime parameter. # Replace delete with idempotent rename to a temporary object, then delete the latter with non-idempotent delete. [See the beginning of this comment.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538] # [Delete by inodeID.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13669018&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13669018] This is a two step delete: first obtain inodeID, then call delete(path, inodeID) with a meaning that the path is deleted if the object referred still has the same inodeID. was (Author: shv): This discussion started in HDFS-4849. Where we discussed four approaches so far # Just mark delete idempotent. [A delete retry may delete an object that has been recreated or replaced between the retries in this case.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670142&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670142] # Design ["duplicate request cache"|http://www.freesoft.org/CIE/RFC/1813/47.htm]. This is applicable to almost any operation and semantics. Drawbacks: complexity, performance, and RAM overhead. [See last two paragraphs.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538] # Add modTime parameter to delete operation with the meaning that the object is deleted only if its modification time is <= than modTime parameter. # Replace delete with idempotent rename to a temporary object, then delete the latter with non-idempotent delete. [See the beginning of this comment.|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538] > Idempotent delete operation. > > > Key: HDFS-4872 > URL: https://issues.apache.org/jira/browse/HDFS-4872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > Making delete idempotent is important to provide uninterrupted job execution > in case of HA failover. > This is to discuss different approaches to idempotent implementation of > delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4879) Add "blocked ArrayList" collection to avoid CMS full GCs
[
https://issues.apache.org/jira/browse/HDFS-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-4879:
--
Attachment: hdfs-4879.txt
New rev with slight improvements I noticed during further testing:
- made default chunk size 8K elements instead of 4K - should be slightly
faster, and still small enough chunks to fit in a fragmented heap
- previously was allowing even the first chunk to realloc itself up to
maxChunkSize, but now just allocates a new chunk whenever any chunk fills its
allocated capacity.
- cache the last chunk, which sped up add() a bit
- change the perf test to alternate between the two implementations to be a
little more fair in terms of jit, gc, etc
This should be ready for review now.
> Add "blocked ArrayList" collection to avoid CMS full GCs
>
>
> Key: HDFS-4879
> URL: https://issues.apache.org/jira/browse/HDFS-4879
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.0.4-alpha
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: hdfs-4879.txt, hdfs-4879.txt
>
>
> We recently saw an issue where a large deletion was issued which caused 25M
> blocks to be collected during {{deleteInternal}}. Currently, the list of
> collected blocks is an ArrayList, meaning that we had to allocate a
> contiguous 25M-entry array (~400MB). After a NN has been running for a long
> amount of time, the old generation may become fragmented such that it's hard
> to find a 400MB contiguous chunk of heap.
> In general, we should try to design the NN such that the only large objects
> are long-lived and created at startup time. We can improve this particular
> case (and perhaps some others) by introducing a new List implementation which
> is made of a linked list of arrays, each of which is size-limited (eg to 1MB).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: HDFS-4878.patch Removing block from neededReplications queue when removeBlock() is called > On Remove Block, Block is not Removed from neededReplications queue > --- > > Key: HDFS-4878 > URL: https://issues.apache.org/jira/browse/HDFS-4878 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Tao Luo > Attachments: HDFS-4878.patch > > > Remove block removes the specified block from pendingReplications, but not > from neededReplications queue. > The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Patch Available (was: Open) > On Remove Block, Block is not Removed from neededReplications queue > --- > > Key: HDFS-4878 > URL: https://issues.apache.org/jira/browse/HDFS-4878 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Tao Luo > Attachments: HDFS-4878.patch > > > Remove block removes the specified block from pendingReplications, but not > from neededReplications queue. > The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4815) TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN: Double call countReplicas() to fetch corruptReplicas and liveReplicas is not needed
[
https://issues.apache.org/jira/browse/HDFS-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675278#comment-13675278
]
Aaron T. Myers commented on HDFS-4815:
--
+1, patch looks good to me as well.
Chris, mind committing this? Or would you like me to do it?
> TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN:
> Double call countReplicas() to fetch corruptReplicas and liveReplicas is not
> needed
> ---
>
> Key: HDFS-4815
> URL: https://issues.apache.org/jira/browse/HDFS-4815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Tian Hong Wang
>Assignee: Tian Hong Wang
> Labels: patch
> Attachments: HDFS-4815.patch
>
>
> In TestRBWBlockInvalidation, the original code is:
> while (!isCorruptReported) {
> if (countReplicas(namesystem, blk).corruptReplicas() > 0) {
> isCorruptReported = true;
> }
> Thread.sleep(100);
> }
> assertEquals("There should be 1 replica in the corruptReplicasMap", 1,
> countReplicas(namesystem, blk).corruptReplicas());
> Once the program detects there exists one corruptReplica, it will break the
> while loop. After that, it call countReplicas() again in assertEquals(). But
> sometimes I met the following error:
> java.lang.AssertionError: There should be 1 replica in the corruptReplicasMap
> expected:<1> but was:<0>
> It's obviously that second function call countReplicas() in assertEquals(),
> the corruptReplicas value has been changed since program go to sleep and
> BlockManger recovered the corrupt block during this sleep time.
> So what I do is:
> 1) once detecting there exists one corruptReplica, break the loop and don't
> call sleep(), the same as liveReplicas
> 2) don't double check the countReplicas & liveReplicas in assertEquals()
> 3) sometimes I meet the problem of testcase timeout, so I speed up the block
> report interval
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-4874) create with OVERWRITE deletes existing file without checking the lease: feature or a bug.
[ https://issues.apache.org/jira/browse/HDFS-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-4874: --- If you don't mind, Daryn, I'll reopen it while we are still discussing. > create with OVERWRITE deletes existing file without checking the lease: > feature or a bug. > - > > Key: HDFS-4874 > URL: https://issues.apache.org/jira/browse/HDFS-4874 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > create with OVERWRITE flag will remove a file under construction even if the > issuing client does not hold a lease on the file. > It could be a bug or the feature that applications rely upon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4874) create with OVERWRITE deletes existing file without checking the lease: feature or a bug.
[ https://issues.apache.org/jira/browse/HDFS-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675293#comment-13675293 ] Andrew Wang commented on HDFS-4874: --- Just throwing in my 2c, I've seen some some client applications that do creates with overwrite and expect this behavior. So, I side with Suresh and Daryn on this one. > create with OVERWRITE deletes existing file without checking the lease: > feature or a bug. > - > > Key: HDFS-4874 > URL: https://issues.apache.org/jira/browse/HDFS-4874 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > create with OVERWRITE flag will remove a file under construction even if the > issuing client does not hold a lease on the file. > It could be a bug or the feature that applications rely upon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4840) ReplicationMonitor gets NPE during shutdown
[ https://issues.apache.org/jira/browse/HDFS-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4840: - Resolution: Fixed Fix Version/s: 2.1.0-beta 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Jing for the review. I've committed this to branch-2, branch-2.1.0-beta and trunk. > ReplicationMonitor gets NPE during shutdown > --- > > Key: HDFS-4840 > URL: https://issues.apache.org/jira/browse/HDFS-4840 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HDFS-4840.patch.txt > > > TestBlocksWithNotEnoughRacks occasionally fails during test tear down because > RaplicationMonitor gets NPE. > Seen at https://builds.apache.org/job/Hadoop-Hdfs-trunk/1406/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4840) ReplicationMonitor gets NPE during shutdown
[ https://issues.apache.org/jira/browse/HDFS-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675314#comment-13675314 ] Hudson commented on HDFS-4840: -- Integrated in Hadoop-trunk-Commit #3860 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3860/]) HDFS-4840. ReplicationMonitor gets NPE during shutdown. Contributed by Kihwal Lee. (Revision 1489634) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489634 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > ReplicationMonitor gets NPE during shutdown > --- > > Key: HDFS-4840 > URL: https://issues.apache.org/jira/browse/HDFS-4840 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HDFS-4840.patch.txt > > > TestBlocksWithNotEnoughRacks occasionally fails during test tear down because > RaplicationMonitor gets NPE. > Seen at https://builds.apache.org/job/Hadoop-Hdfs-trunk/1406/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[
https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675324#comment-13675324
]
Hadoop QA commented on HDFS-4877:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586148/HDFS-4877.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4476//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4476//console
This message is automatically generated.
> Snapshot: fix the scenario where a directory is renamed under its prior
> descendant
> --
>
> Key: HDFS-4877
> URL: https://issues.apache.org/jira/browse/HDFS-4877
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>Priority: Blocker
> Attachments: HDFS-4877.001.patch
>
>
> Suppose we have the following operations:
> 1. rename /dir1/foo/bar to /dir2/bar2
> 2. rename /dir1/foo to /dir2/bar2/foo2
> I.e., we rename a directory (foo) to a child of its prior descendant (bar).
> If we have taken snapshots on root before the above 2 rename operations, a
> circle can be generated consisting of nodes with the types
> INodeReference.WithName and INodeReference.DstReference:
> WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) -->
> WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo)
> When deleting a snapshot before the rename operations, the current code may
> hit an infinite loop when cleaning the snapshot data or updating the quota
> usage for the nodes in the above circle. This jira will fix the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[
https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675357#comment-13675357
]
Ravi Prakash commented on HDFS-4867:
Patch looks good to me. Thanks Plamen!
{quote} metaSave is probably a casualty here. Should we take a look at why
orphaned / missing blocks are kept in replication queues in the first place?
It seems that when we delete a file blocks can also be removed from replication
queue, because what is the point of replicating them if they don't belong to
any files.
{quote}
+1 for Konstantin's suggestion. Plamen, could you please open another JIRA for
it?
> metaSave NPEs when there are invalid blocks in repl queue.
> --
>
> Key: HDFS-4867
> URL: https://issues.apache.org/jira/browse/HDFS-4867
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.7, 2.0.4-alpha
>Reporter: Kihwal Lee
>Assignee: Plamen Jeliazkov
> Fix For: 3.0.0
>
> Attachments: HDFS-4867.trunk.patch, HDFS-4867.trunk.patch
>
>
> Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs
> and stops generating further report. Normally ReplicationMonitor removes them
> quickly, but if the queue is huge, it takes very long time. Also in safe
> mode, they stay.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[ https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675359#comment-13675359 ] Ravi Prakash commented on HDFS-4867: bq. Plamen, could you please open another JIRA for it? Seems like Tao did already. Thanks Tao! > metaSave NPEs when there are invalid blocks in repl queue. > -- > > Key: HDFS-4867 > URL: https://issues.apache.org/jira/browse/HDFS-4867 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Plamen Jeliazkov > Fix For: 3.0.0 > > Attachments: HDFS-4867.trunk.patch, HDFS-4867.trunk.patch > > > Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs > and stops generating further report. Normally ReplicationMonitor removes them > quickly, but if the queue is huge, it takes very long time. Also in safe > mode, they stay. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4874) create with OVERWRITE deletes existing file without checking the lease: feature or a bug.
[ https://issues.apache.org/jira/browse/HDFS-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675376#comment-13675376 ] Konstantin Shvachko commented on HDFS-4874: --- This sounds like an acceptable behaviour. Is it the same in other versions? > create with OVERWRITE deletes existing file without checking the lease: > feature or a bug. > - > > Key: HDFS-4874 > URL: https://issues.apache.org/jira/browse/HDFS-4874 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > create with OVERWRITE flag will remove a file under construction even if the > issuing client does not hold a lease on the file. > It could be a bug or the feature that applications rely upon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4815) TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN: Double call countReplicas() to fetch corruptReplicas and liveReplicas is not needed
[
https://issues.apache.org/jira/browse/HDFS-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675379#comment-13675379
]
Chris Nauroth commented on HDFS-4815:
-
Sorry, Aaron. I'll be away for at least a week, possibly longer, so would you
please commit this one? Thanks!
> TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN:
> Double call countReplicas() to fetch corruptReplicas and liveReplicas is not
> needed
> ---
>
> Key: HDFS-4815
> URL: https://issues.apache.org/jira/browse/HDFS-4815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Tian Hong Wang
>Assignee: Tian Hong Wang
> Labels: patch
> Attachments: HDFS-4815.patch
>
>
> In TestRBWBlockInvalidation, the original code is:
> while (!isCorruptReported) {
> if (countReplicas(namesystem, blk).corruptReplicas() > 0) {
> isCorruptReported = true;
> }
> Thread.sleep(100);
> }
> assertEquals("There should be 1 replica in the corruptReplicasMap", 1,
> countReplicas(namesystem, blk).corruptReplicas());
> Once the program detects there exists one corruptReplica, it will break the
> while loop. After that, it call countReplicas() again in assertEquals(). But
> sometimes I met the following error:
> java.lang.AssertionError: There should be 1 replica in the corruptReplicasMap
> expected:<1> but was:<0>
> It's obviously that second function call countReplicas() in assertEquals(),
> the corruptReplicas value has been changed since program go to sleep and
> BlockManger recovered the corrupt block during this sleep time.
> So what I do is:
> 1) once detecting there exists one corruptReplica, break the loop and don't
> call sleep(), the same as liveReplicas
> 2) don't double check the countReplicas & liveReplicas in assertEquals()
> 3) sometimes I meet the problem of testcase timeout, so I speed up the block
> report interval
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4874) create with OVERWRITE deletes existing file without checking the lease: feature or a bug.
[ https://issues.apache.org/jira/browse/HDFS-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675389#comment-13675389 ] Konstantin Shvachko commented on HDFS-4874: --- Also I have seen on 2.0 that the other client (the one who was overwritten) was failing with "Cannot allocate block ... passed 'previous' block " instead of a lease exception. Which would mean either that the lease is not removed by OVERWRITE or not checked somewhere in addBlock(). Anybody seen this? Couldn't reproduce it in lab environment. > create with OVERWRITE deletes existing file without checking the lease: > feature or a bug. > - > > Key: HDFS-4874 > URL: https://issues.apache.org/jira/browse/HDFS-4874 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > create with OVERWRITE flag will remove a file under construction even if the > issuing client does not hold a lease on the file. > It could be a bug or the feature that applications rely upon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4815) TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN: Double call countReplicas() to fetch corruptReplicas and liveReplicas is not needed
[
https://issues.apache.org/jira/browse/HDFS-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-4815:
-
Resolution: Fixed
Fix Version/s: 2.1.0-beta
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
OK, I've just committed this to trunk and branch-2.
Thanks a lot for the contribution, Tian.
> TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN:
> Double call countReplicas() to fetch corruptReplicas and liveReplicas is not
> needed
> ---
>
> Key: HDFS-4815
> URL: https://issues.apache.org/jira/browse/HDFS-4815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Tian Hong Wang
>Assignee: Tian Hong Wang
> Labels: patch
> Fix For: 2.1.0-beta
>
> Attachments: HDFS-4815.patch
>
>
> In TestRBWBlockInvalidation, the original code is:
> while (!isCorruptReported) {
> if (countReplicas(namesystem, blk).corruptReplicas() > 0) {
> isCorruptReported = true;
> }
> Thread.sleep(100);
> }
> assertEquals("There should be 1 replica in the corruptReplicasMap", 1,
> countReplicas(namesystem, blk).corruptReplicas());
> Once the program detects there exists one corruptReplica, it will break the
> while loop. After that, it call countReplicas() again in assertEquals(). But
> sometimes I met the following error:
> java.lang.AssertionError: There should be 1 replica in the corruptReplicasMap
> expected:<1> but was:<0>
> It's obviously that second function call countReplicas() in assertEquals(),
> the corruptReplicas value has been changed since program go to sleep and
> BlockManger recovered the corrupt block during this sleep time.
> So what I do is:
> 1) once detecting there exists one corruptReplica, break the loop and don't
> call sleep(), the same as liveReplicas
> 2) don't double check the countReplicas & liveReplicas in assertEquals()
> 3) sometimes I meet the problem of testcase timeout, so I speed up the block
> report interval
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-4878: --- Fix Version/s: 3.0.0 > On Remove Block, Block is not Removed from neededReplications queue > --- > > Key: HDFS-4878 > URL: https://issues.apache.org/jira/browse/HDFS-4878 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Tao Luo > Fix For: 3.0.0 > > Attachments: HDFS-4878.patch > > > Remove block removes the specified block from pendingReplications, but not > from neededReplications queue. > The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Open (was: Patch Available) > On Remove Block, Block is not Removed from neededReplications queue > --- > > Key: HDFS-4878 > URL: https://issues.apache.org/jira/browse/HDFS-4878 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Tao Luo > Fix For: 3.0.0 > > Attachments: HDFS-4878.patch > > > Remove block removes the specified block from pendingReplications, but not > from neededReplications queue. > The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Patch Available (was: Open) > On Remove Block, Block is not Removed from neededReplications queue > --- > > Key: HDFS-4878 > URL: https://issues.apache.org/jira/browse/HDFS-4878 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Tao Luo > Fix For: 3.0.0 > > Attachments: HDFS-4878.patch > > > Remove block removes the specified block from pendingReplications, but not > from neededReplications queue. > The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4815) TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN: Double call countReplicas() to fetch corruptReplicas and liveReplicas is not needed
[
https://issues.apache.org/jira/browse/HDFS-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675413#comment-13675413
]
Hudson commented on HDFS-4815:
--
Integrated in Hadoop-trunk-Commit #3863 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3863/])
HDFS-4815. TestRBWBlockInvalidation: Double call countReplicas() to fetch
corruptReplicas and liveReplicas is not needed. Contributed by Tian Hong Wang.
(Revision 1489668)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489668
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
> TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN:
> Double call countReplicas() to fetch corruptReplicas and liveReplicas is not
> needed
> ---
>
> Key: HDFS-4815
> URL: https://issues.apache.org/jira/browse/HDFS-4815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Reporter: Tian Hong Wang
>Assignee: Tian Hong Wang
> Labels: patch
> Fix For: 2.1.0-beta
>
> Attachments: HDFS-4815.patch
>
>
> In TestRBWBlockInvalidation, the original code is:
> while (!isCorruptReported) {
> if (countReplicas(namesystem, blk).corruptReplicas() > 0) {
> isCorruptReported = true;
> }
> Thread.sleep(100);
> }
> assertEquals("There should be 1 replica in the corruptReplicasMap", 1,
> countReplicas(namesystem, blk).corruptReplicas());
> Once the program detects there exists one corruptReplica, it will break the
> while loop. After that, it call countReplicas() again in assertEquals(). But
> sometimes I met the following error:
> java.lang.AssertionError: There should be 1 replica in the corruptReplicasMap
> expected:<1> but was:<0>
> It's obviously that second function call countReplicas() in assertEquals(),
> the corruptReplicas value has been changed since program go to sleep and
> BlockManger recovered the corrupt block during this sleep time.
> So what I do is:
> 1) once detecting there exists one corruptReplica, break the loop and don't
> call sleep(), the same as liveReplicas
> 2) don't double check the countReplicas & liveReplicas in assertEquals()
> 3) sometimes I meet the problem of testcase timeout, so I speed up the block
> report interval
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[
https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675414#comment-13675414
]
Hadoop QA commented on HDFS-4878:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586178/HDFS-4878.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4477//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4477//console
This message is automatically generated.
> On Remove Block, Block is not Removed from neededReplications queue
> ---
>
> Key: HDFS-4878
> URL: https://issues.apache.org/jira/browse/HDFS-4878
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Tao Luo
> Fix For: 3.0.0
>
> Attachments: HDFS-4878.patch
>
>
> Remove block removes the specified block from pendingReplications, but not
> from neededReplications queue.
> The fix would be to remove from neededReplications as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4879) Add "blocked ArrayList" collection to avoid CMS full GCs
[
https://issues.apache.org/jira/browse/HDFS-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675424#comment-13675424
]
Colin Patrick McCabe commented on HDFS-4879:
This is a good improvement; it should be really helpful in keeping those GCs
down.
Since you're not actually implementing {{java.util.List}}, why not name it
something like {{BlockedArrayContainer}}?
As long as we're merely implementing {{Iterable}}, do we have to implement
{{get}} at all? it seems to be fairly inefficient, so it might encourage the
wrong kind of programmer behavior.
> Add "blocked ArrayList" collection to avoid CMS full GCs
>
>
> Key: HDFS-4879
> URL: https://issues.apache.org/jira/browse/HDFS-4879
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.0.4-alpha
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: hdfs-4879.txt, hdfs-4879.txt
>
>
> We recently saw an issue where a large deletion was issued which caused 25M
> blocks to be collected during {{deleteInternal}}. Currently, the list of
> collected blocks is an ArrayList, meaning that we had to allocate a
> contiguous 25M-entry array (~400MB). After a NN has been running for a long
> amount of time, the old generation may become fragmented such that it's hard
> to find a 400MB contiguous chunk of heap.
> In general, we should try to design the NN such that the only large objects
> are long-lived and created at startup time. We can improve this particular
> case (and perhaps some others) by introducing a new List implementation which
> is made of a linked list of arrays, each of which is size-limited (eg to 1MB).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[ https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675427#comment-13675427 ] Konstantin Shvachko commented on HDFS-4872: --- > A delete retry may delete an object that has been recreated or replaced > between the retries Isn't that what Zookeeper does? (I really don't know, please educate if somebody does.) > timestamps are unreliable without synced clocks > How is time synchronization between client and server done? We don't synchronize clocks. modTime is taken from the file status, therefore it is the NameNode time. So in a sense it is the same as inodeID approach except it can be used with versions of HDFS that don't have inodeIDs yet. Also it is a bit more generic, because you can use any timestamp as modTime that you have seen reported by NN. Like, if I know the modTime of a file that I just created I can use it to delete files created before. This is if we really want to avoid delete retries removing "future" files. Todd, I think your rename() strategy is right. Should we implement it to make rename idempotent? This is the last operation that prevents us from providing uninterrupted job execution through HA failovers. I agree with others that resolving delete through a rename + delete is somewhat like delaying the problem for later. What if two clients rename to the same temp file, then the delete will not know what it is deleting. > Idempotent delete operation. > > > Key: HDFS-4872 > URL: https://issues.apache.org/jira/browse/HDFS-4872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Konstantin Shvachko > > Making delete idempotent is important to provide uninterrupted job execution > in case of HA failover. > This is to discuss different approaches to idempotent implementation of > delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[ https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675447#comment-13675447 ] Konstantin Shvachko commented on HDFS-4867: --- +1 for the patch. > metaSave NPEs when there are invalid blocks in repl queue. > -- > > Key: HDFS-4867 > URL: https://issues.apache.org/jira/browse/HDFS-4867 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Plamen Jeliazkov > Fix For: 3.0.0 > > Attachments: HDFS-4867.trunk.patch, HDFS-4867.trunk.patch > > > Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs > and stops generating further report. Normally ReplicationMonitor removes them > quickly, but if the queue is huge, it takes very long time. Also in safe > mode, they stay. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4872) Idempotent delete operation.
[
https://issues.apache.org/jira/browse/HDFS-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675458#comment-13675458
]
Aaron T. Myers commented on HDFS-4872:
--
I think we should again seriously consider implementing the duplicate request
cache to address all of these problems with one solution. The stated drawbacks
of this were "complexity, performance, and RAM overhead."
Regarding performance, I don't see how a constant-time lookup in an in-memory
hash map could be slower or more taxing on the NN than the alternative
suggestions of the client making two totally separate RPCs.
As for RAM overhead, the math that Todd did in [this
comment|https://issues.apache.org/jira/browse/HDFS-4849?focusedCommentId=13670538&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13670538]
suggests that RAM usage should be basically negligible versus the rest of the
NN heap. We should also probably store the results of the calls in the
duplicate request cache, so we can return those to the client on a retried
call, but from a quick review of the non-idempotent operations in HDFS, all of
those return values are constant in size and none of them should be very large:
{noformat}
// Commonly-used non-idempotent client operations.
HdfsFileStatus create
void abandonBlock
boolean rename
void rename2
boolean delete
void updatePipeline
// Less commonly-used non-idempotent client operations.
void createSymlink
void concat
LocatedBlock append
void cancelDelegationToken
DataEncryptionKey getDataEncryptionKey
// Mostly administrative commands, unlikely most clients ever use.
void saveNamespace
boolean restoreFailedStorage
void refreshNodes
void finalizeUpgrade
void metaSave
{noformat}
As for complexity, I'm honestly not sure what complexity folks are concerned
about. It's well established how to generate GUIDs on clients, we already log
all FS metadata mutations to disk so adding in those GUIDs shouldn't be too
tough, and adding an in-memory hash lookup on the NN should be pretty
straightforward. We can likely reuse the lightweight hash sets we already use
elsewhere in HDFS for the cache itself.
The stated reason for preferring the rename-to-tmp-then-delete scheme versus
the get-inode-id-then-delete scheme is to support versions of HDFS which do not
yet have INode ID support. A duplicate request cache should work in either case.
> Idempotent delete operation.
>
>
> Key: HDFS-4872
> URL: https://issues.apache.org/jira/browse/HDFS-4872
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.0.4-alpha
>Reporter: Konstantin Shvachko
>
> Making delete idempotent is important to provide uninterrupted job execution
> in case of HA failover.
> This is to discuss different approaches to idempotent implementation of
> delete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[ https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675477#comment-13675477 ] Plamen Jeliazkov commented on HDFS-4867: The patch for trunk is applicable to branch-2. > metaSave NPEs when there are invalid blocks in repl queue. > -- > > Key: HDFS-4867 > URL: https://issues.apache.org/jira/browse/HDFS-4867 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.5-alpha > > Attachments: HDFS-4867.trunk.patch, HDFS-4867.trunk.patch > > > Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs > and stops generating further report. Normally ReplicationMonitor removes them > quickly, but if the queue is huge, it takes very long time. Also in safe > mode, they stay. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4867) metaSave NPEs when there are invalid blocks in repl queue.
[ https://issues.apache.org/jira/browse/HDFS-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-4867: --- Fix Version/s: 2.0.5-alpha > metaSave NPEs when there are invalid blocks in repl queue. > -- > > Key: HDFS-4867 > URL: https://issues.apache.org/jira/browse/HDFS-4867 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Kihwal Lee >Assignee: Plamen Jeliazkov > Fix For: 3.0.0, 2.0.5-alpha > > Attachments: HDFS-4867.trunk.patch, HDFS-4867.trunk.patch > > > Since metaSave cannot get the inode holding a orphaned/invalid block, it NPEs > and stops generating further report. Normally ReplicationMonitor removes them > quickly, but if the queue is huge, it takes very long time. Also in safe > mode, they stay. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[ https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4873: Affects Version/s: 2.1.0-beta Status: Patch Available (was: Open) > callGetBlockLocations returns incorrect number of blocks for snapshotted files > -- > > Key: HDFS-4873 > URL: https://issues.apache.org/jira/browse/HDFS-4873 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Hari Mankude >Assignee: Jing Zhao > Attachments: HDFS-4873.001.patch > > > callGetBlockLocations() returns all the blocks of a file even when they are > not present in the snap version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[ https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4873: Attachment: HDFS-4873.001.patch Thanks a lot for the good catch, Hari! Upload the initial patch for review, in which I just followed your suggestions to truncate the length to fileSize before calling createLocatedBlocks. Besides, I just identify the last block that the fileSize hits as the last block in the returned locatedBlocks. > callGetBlockLocations returns incorrect number of blocks for snapshotted files > -- > > Key: HDFS-4873 > URL: https://issues.apache.org/jira/browse/HDFS-4873 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Hari Mankude >Assignee: Jing Zhao > Attachments: HDFS-4873.001.patch > > > callGetBlockLocations() returns all the blocks of a file even when they are > not present in the snap version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[ https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4877: Labels: snapshots (was: ) > Snapshot: fix the scenario where a directory is renamed under its prior > descendant > -- > > Key: HDFS-4877 > URL: https://issues.apache.org/jira/browse/HDFS-4877 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Blocker > Labels: snapshots > Attachments: HDFS-4877.001.patch > > > Suppose we have the following operations: > 1. rename /dir1/foo/bar to /dir2/bar2 > 2. rename /dir1/foo to /dir2/bar2/foo2 > I.e., we rename a directory (foo) to a child of its prior descendant (bar). > If we have taken snapshots on root before the above 2 rename operations, a > circle can be generated consisting of nodes with the types > INodeReference.WithName and INodeReference.DstReference: > WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> > WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) > When deleting a snapshot before the rename operations, the current code may > hit an infinite loop when cleaning the snapshot data or updating the quota > usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4877) Snapshot: fix the scenario where a directory is renamed under its prior descendant
[ https://issues.apache.org/jira/browse/HDFS-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4877: Component/s: snapshots > Snapshot: fix the scenario where a directory is renamed under its prior > descendant > -- > > Key: HDFS-4877 > URL: https://issues.apache.org/jira/browse/HDFS-4877 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Blocker > Attachments: HDFS-4877.001.patch > > > Suppose we have the following operations: > 1. rename /dir1/foo/bar to /dir2/bar2 > 2. rename /dir1/foo to /dir2/bar2/foo2 > I.e., we rename a directory (foo) to a child of its prior descendant (bar). > If we have taken snapshots on root before the above 2 rename operations, a > circle can be generated consisting of nodes with the types > INodeReference.WithName and INodeReference.DstReference: > WithName (foo) --> WithCount (for foo) --> foo2 --> WithName (bar) --> > WithCount (for bar) --> bar2 --> DstReference (foo) --> WithCount (for foo) > When deleting a snapshot before the rename operations, the current code may > hit an infinite loop when cleaning the snapshot data or updating the quota > usage for the nodes in the above circle. This jira will fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[
https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675543#comment-13675543
]
Hadoop QA commented on HDFS-4873:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586234/HDFS-4873.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4478//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4478//console
This message is automatically generated.
> callGetBlockLocations returns incorrect number of blocks for snapshotted files
> --
>
> Key: HDFS-4873
> URL: https://issues.apache.org/jira/browse/HDFS-4873
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Hari Mankude
>Assignee: Jing Zhao
> Attachments: HDFS-4873.001.patch
>
>
> callGetBlockLocations() returns all the blocks of a file even when they are
> not present in the snap version
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4880) Diagnostic logging while loading name/edits files
Arpit Agarwal created HDFS-4880: --- Summary: Diagnostic logging while loading name/edits files Key: HDFS-4880 URL: https://issues.apache.org/jira/browse/HDFS-4880 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0, 1.3.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Add some minimal diagnostic logging to help determine location of the files being loaded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4871) Skip failing commons tests on Windows
[ https://issues.apache.org/jira/browse/HDFS-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675591#comment-13675591 ] Chuan Liu commented on HDFS-4871: - >My personal preference would be to file a single parent task Jira, and as we >fix individual tests we can link their Jiras to the parent Jira. This is a really good suggestion! I have create HADOOP-9620 to track the unit test issues in branch-2 on Windows. > Skip failing commons tests on Windows > - > > Key: HDFS-4871 > URL: https://issues.apache.org/jira/browse/HDFS-4871 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Fix For: 2.1.0-beta > > > This is a temporary fix proposed to get CI working. We will skip the > following failing tests on Windows: > # TestChRootedFs > # TestFSMainOperationsLocalFileSystem > # TestFcCreateMkdirLocalFs > # TestFcMainOperationsLocalFs > # TestFcPermissionsLocalFs > # TestLocalFSFileContextSymlink - HADOOP-9527 > # TestLocalFileSystem > # TestShellCommandFencer - HADOOP-9526 > # TestSocketIOWithTimeout - HADOOP-8982 > # TestViewFsLocalFs > # TestViewFsTrash > # TestViewFsWithAuthorityLocalFs > The tests will be re-enabled as we fix each. JIRAs for remaining failing > tests to follow soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
[ https://issues.apache.org/jira/browse/HDFS-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4876: - Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Jing for reviewing the patch. I have committed this. > The javadoc of FileWithSnapshot is incorrect > > > Key: HDFS-4876 > URL: https://issues.apache.org/jira/browse/HDFS-4876 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: h4876_20130604.patch > > > The javadoc said that snapshot files and the original file form a circular > linked list. It is no longer true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4876) The javadoc of FileWithSnapshot is incorrect
[ https://issues.apache.org/jira/browse/HDFS-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675622#comment-13675622 ] Hudson commented on HDFS-4876: -- Integrated in Hadoop-trunk-Commit #3864 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3864/]) HDFS-4876. Fix the javadoc of FileWithSnapshot and move FileDiffList to FileWithSnapshot. (Revision 1489708) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489708 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java > The javadoc of FileWithSnapshot is incorrect > > > Key: HDFS-4876 > URL: https://issues.apache.org/jira/browse/HDFS-4876 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: h4876_20130604.patch > > > The javadoc said that snapshot files and the original file form a circular > linked list. It is no longer true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4880) Diagnostic logging while loading name/edits files
[ https://issues.apache.org/jira/browse/HDFS-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4880: Attachment: HDFS-4880.branch-1.patch > Diagnostic logging while loading name/edits files > - > > Key: HDFS-4880 > URL: https://issues.apache.org/jira/browse/HDFS-4880 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 1.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-4880.branch-1.patch > > > Add some minimal diagnostic logging to help determine location of the files > being loaded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4880) Diagnostic logging while loading name/edits files
[ https://issues.apache.org/jira/browse/HDFS-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675639#comment-13675639 ] Arpit Agarwal commented on HDFS-4880: - edits file location is already logged on branch-1, start logging image file location. > Diagnostic logging while loading name/edits files > - > > Key: HDFS-4880 > URL: https://issues.apache.org/jira/browse/HDFS-4880 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 1.3.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-4880.branch-1.patch > > > Add some minimal diagnostic logging to help determine location of the files > being loaded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
