[jira] [Commented] (HDFS-4372) Track NameNode startup progress
[ https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680253#comment-13680253 ] Jing Zhao commented on HDFS-4372: - Thanks for the responses Chris! bq. I wrote all of this code before snapshots, so I'll take a look to see if there is something more reasonable to do now that snapshots are in trunk. The main challenge here is that after renaming with snapshots, for the same subtree, we count in the number of snapshot diffs for namespace quota usage when the subtree is referenced by a DstReference node (which means the subtree is included in the current tree); but we do not count the snapshot diffs when we come from a WithName node. In such case, it is difficult to decide whether we need to increment the counter (and the value of the increment) after loading an inode from the fsimage. bq. I added the check for containsKey to limit the impact of that. Maybe it's just a minor optimization, but considering that these methods could be called in tight loops during startup, I wanted to keep it under control. This makes sense to me. Thanks a lot for the explanation. bq. By switching to primitive long with -1 reserved for uninitialized, is the intent to reduce memory footprint of a boxed Long? If so, then I definitely can make that change. Yes. Thanks! Track NameNode startup progress --- Key: HDFS-4372 URL: https://issues.apache.org/jira/browse/HDFS-4372 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HDFS-4372.1.patch Track detailed progress information about the steps of NameNode startup to enable display to users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[ https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4873: - Hadoop Flags: Reviewed +1 patch looks good. Thanks Jing. callGetBlockLocations returns incorrect number of blocks for snapshotted files -- Key: HDFS-4873 URL: https://issues.apache.org/jira/browse/HDFS-4873 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hari Mankude Assignee: Jing Zhao Attachments: HDFS-4873.001.patch, HDFS-4873.002.patch callGetBlockLocations() returns all the blocks of a file even when they are not present in the snap version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680358#comment-13680358 ] Hudson commented on HDFS-4878: -- Integrated in Hadoop-Yarn-trunk #237 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/237/]) HDFS-4878. On Remove Block, block is not removed from neededReplications queue. Contributed by Tao Luo. (Revision 1491671) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1491671 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 2.1.0-beta Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4752) TestRBWBlockInvalidation fails on Windows due to file locking
[ https://issues.apache.org/jira/browse/HDFS-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680359#comment-13680359 ] Hudson commented on HDFS-4752: -- Integrated in Hadoop-Yarn-trunk #237 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/237/]) HDFS-4752. TestRBWBlockInvalidation fails on Windows due to file locking. Contributed by Chris Nauroth. (Revision 1491631) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1491631 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java TestRBWBlockInvalidation fails on Windows due to file locking - Key: HDFS-4752 URL: https://issues.apache.org/jira/browse/HDFS-4752 Project: Hadoop HDFS Issue Type: Bug Components: datanode, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HDFS-4752.1.patch, HDFS-4752.2.patch The test attempts to invalidate a block by deleting its block file and meta file. This happens while a datanode thread holds the files open for write. On Windows, this causes a locking conflict, and the test fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4898) BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack
Eric Sirianni created HDFS-4898:
---
Summary: BlockPlacementPolicyWithNodeGroup.chooseRemoteRack()
fails to properly fallback to local rack
Key: HDFS-4898
URL: https://issues.apache.org/jira/browse/HDFS-4898
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.0.4-alpha, 1.2.0
Reporter: Eric Sirianni
Priority: Minor
As currently implemented, {{BlockPlacementPolicyWithNodeGroup}} does not
properly fallback to local rack when no nodes are available in remote racks,
resulting in an improper {{NotEnoughReplicasException}}.
{code:title=BlockPlacementPolicyWithNodeGroup.java}
@Override
protected void chooseRemoteRack(int numOfReplicas,
DatanodeDescriptor localMachine, HashMapNode, Node excludedNodes,
long blocksize, int maxReplicasPerRack, ListDatanodeDescriptor results,
boolean avoidStaleNodes) throws NotEnoughReplicasException {
int oldNumOfReplicas = results.size();
// randomly choose one node from remote racks
try {
chooseRandom(
numOfReplicas,
~ + NetworkTopology.getFirstHalf(localMachine.getNetworkLocation()),
excludedNodes, blocksize, maxReplicasPerRack, results,
avoidStaleNodes);
} catch (NotEnoughReplicasException e) {
chooseRandom(numOfReplicas - (results.size() - oldNumOfReplicas),
localMachine.getNetworkLocation(), excludedNodes, blocksize,
maxReplicasPerRack, results, avoidStaleNodes);
}
}
{code}
As currently coded the {{chooseRandom()}} call in the {{catch}} block will
never succeed as the set of nodes within the passed in node path (e.g.
{{/rack1/nodegroup1}}) is entirely contained within the set of excluded nodes
(both are the set of nodes within the same nodegroup as the node chosen first
replica).
The bug is that the fallback {{chooseRandom()}} call in the catch block should
be passing in the _complement_ of the node path used in the initial
{{chooseRandom()}} call in the try block (e.g. {{/rack1}}) - namely:
{code}
NetworkTopology.getFirstHalf(localMachine.getNetworkLocation())
{code}
This will yield the proper fallback behavior of choosing a random node from
_within the same rack_, but still excluding those nodes _in the same nodegroup_
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680396#comment-13680396 ] Hudson commented on HDFS-4878: -- Integrated in Hadoop-Mapreduce-trunk #1454 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1454/]) HDFS-4878. On Remove Block, block is not removed from neededReplications queue. Contributed by Tao Luo. (Revision 1491671) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1491671 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 2.1.0-beta Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4898) BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack
[
https://issues.apache.org/jira/browse/HDFS-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sirianni updated HDFS-4898:
Status: Patch Available (was: Open)
BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly
fallback to local rack
-
Key: HDFS-4898
URL: https://issues.apache.org/jira/browse/HDFS-4898
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.0.4-alpha, 1.2.0
Reporter: Eric Sirianni
Priority: Minor
As currently implemented, {{BlockPlacementPolicyWithNodeGroup}} does not
properly fallback to local rack when no nodes are available in remote racks,
resulting in an improper {{NotEnoughReplicasException}}.
{code:title=BlockPlacementPolicyWithNodeGroup.java}
@Override
protected void chooseRemoteRack(int numOfReplicas,
DatanodeDescriptor localMachine, HashMapNode, Node excludedNodes,
long blocksize, int maxReplicasPerRack, ListDatanodeDescriptor
results,
boolean avoidStaleNodes) throws NotEnoughReplicasException {
int oldNumOfReplicas = results.size();
// randomly choose one node from remote racks
try {
chooseRandom(
numOfReplicas,
~ +
NetworkTopology.getFirstHalf(localMachine.getNetworkLocation()),
excludedNodes, blocksize, maxReplicasPerRack, results,
avoidStaleNodes);
} catch (NotEnoughReplicasException e) {
chooseRandom(numOfReplicas - (results.size() - oldNumOfReplicas),
localMachine.getNetworkLocation(), excludedNodes, blocksize,
maxReplicasPerRack, results, avoidStaleNodes);
}
}
{code}
As currently coded the {{chooseRandom()}} call in the {{catch}} block will
never succeed as the set of nodes within the passed in node path (e.g.
{{/rack1/nodegroup1}}) is entirely contained within the set of excluded nodes
(both are the set of nodes within the same nodegroup as the node chosen first
replica).
The bug is that the fallback {{chooseRandom()}} call in the catch block
should be passing in the _complement_ of the node path used in the initial
{{chooseRandom()}} call in the try block (e.g. {{/rack1}}) - namely:
{code}
NetworkTopology.getFirstHalf(localMachine.getNetworkLocation())
{code}
This will yield the proper fallback behavior of choosing a random node from
_within the same rack_, but still excluding those nodes _in the same
nodegroup_
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4819) Update Snapshot doc for HDFS-4758
[ https://issues.apache.org/jira/browse/HDFS-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4819: - Issue Type: Sub-task (was: Improvement) Parent: HDFS-2802 Update Snapshot doc for HDFS-4758 - Key: HDFS-4819 URL: https://issues.apache.org/jira/browse/HDFS-4819 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4819_20130611.patch Update Snapshot doc to clarify that nested snapshots are not allowed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4819) Update Snapshot doc for HDFS-4758
[ https://issues.apache.org/jira/browse/HDFS-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4819: - Status: Patch Available (was: Open) Update Snapshot doc for HDFS-4758 - Key: HDFS-4819 URL: https://issues.apache.org/jira/browse/HDFS-4819 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4819_20130611.patch Update Snapshot doc to clarify that nested snapshots are not allowed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4819) Update Snapshot doc for HDFS-4758
[ https://issues.apache.org/jira/browse/HDFS-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4819: - Attachment: h4819_20130611.patch h4819_20130611.patch: 1st patch. Update Snapshot doc for HDFS-4758 - Key: HDFS-4819 URL: https://issues.apache.org/jira/browse/HDFS-4819 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4819_20130611.patch Update Snapshot doc to clarify that nested snapshots are not allowed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4752) TestRBWBlockInvalidation fails on Windows due to file locking
[ https://issues.apache.org/jira/browse/HDFS-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680397#comment-13680397 ] Hudson commented on HDFS-4752: -- Integrated in Hadoop-Mapreduce-trunk #1454 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1454/]) HDFS-4752. TestRBWBlockInvalidation fails on Windows due to file locking. Contributed by Chris Nauroth. (Revision 1491631) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1491631 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java TestRBWBlockInvalidation fails on Windows due to file locking - Key: HDFS-4752 URL: https://issues.apache.org/jira/browse/HDFS-4752 Project: Hadoop HDFS Issue Type: Bug Components: datanode, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HDFS-4752.1.patch, HDFS-4752.2.patch The test attempts to invalidate a block by deleting its block file and meta file. This happens while a datanode thread holds the files open for write. On Windows, this causes a locking conflict, and the test fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4898) BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack
[
https://issues.apache.org/jira/browse/HDFS-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sirianni updated HDFS-4898:
Status: Open (was: Patch Available)
BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly
fallback to local rack
-
Key: HDFS-4898
URL: https://issues.apache.org/jira/browse/HDFS-4898
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.0.4-alpha, 1.2.0
Reporter: Eric Sirianni
Priority: Minor
As currently implemented, {{BlockPlacementPolicyWithNodeGroup}} does not
properly fallback to local rack when no nodes are available in remote racks,
resulting in an improper {{NotEnoughReplicasException}}.
{code:title=BlockPlacementPolicyWithNodeGroup.java}
@Override
protected void chooseRemoteRack(int numOfReplicas,
DatanodeDescriptor localMachine, HashMapNode, Node excludedNodes,
long blocksize, int maxReplicasPerRack, ListDatanodeDescriptor
results,
boolean avoidStaleNodes) throws NotEnoughReplicasException {
int oldNumOfReplicas = results.size();
// randomly choose one node from remote racks
try {
chooseRandom(
numOfReplicas,
~ +
NetworkTopology.getFirstHalf(localMachine.getNetworkLocation()),
excludedNodes, blocksize, maxReplicasPerRack, results,
avoidStaleNodes);
} catch (NotEnoughReplicasException e) {
chooseRandom(numOfReplicas - (results.size() - oldNumOfReplicas),
localMachine.getNetworkLocation(), excludedNodes, blocksize,
maxReplicasPerRack, results, avoidStaleNodes);
}
}
{code}
As currently coded the {{chooseRandom()}} call in the {{catch}} block will
never succeed as the set of nodes within the passed in node path (e.g.
{{/rack1/nodegroup1}}) is entirely contained within the set of excluded nodes
(both are the set of nodes within the same nodegroup as the node chosen first
replica).
The bug is that the fallback {{chooseRandom()}} call in the catch block
should be passing in the _complement_ of the node path used in the initial
{{chooseRandom()}} call in the try block (e.g. {{/rack1}}) - namely:
{code}
NetworkTopology.getFirstHalf(localMachine.getNetworkLocation())
{code}
This will yield the proper fallback behavior of choosing a random node from
_within the same rack_, but still excluding those nodes _in the same
nodegroup_
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4898) BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack
[
https://issues.apache.org/jira/browse/HDFS-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sirianni updated HDFS-4898:
Status: Patch Available (was: Open)
{noformat}
index 8cb072b..302981f 100644
---
a/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyWithNodeGroup.java
+++
b/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyWithNodeGroup.java
@@ -178,7 +178,7 @@ public class BlockPlacementPolicyWithNodeGroup extends
BlockPlacementPolicyDefau
avoidStaleNodes);
} catch (NotEnoughReplicasException e) {
chooseRandom(numOfReplicas - (results.size() - oldNumOfReplicas),
- localMachine.getNetworkLocation(), excludedNodes, blocksize,
+ NetworkTopology.getFirstHalf(localMachine.getNetworkLocation()),
excludedNodes, blocksize,
maxReplicasPerRack, results, avoidStaleNodes);
}
}
{noformat}
BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly
fallback to local rack
-
Key: HDFS-4898
URL: https://issues.apache.org/jira/browse/HDFS-4898
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.0.4-alpha, 1.2.0
Reporter: Eric Sirianni
Priority: Minor
As currently implemented, {{BlockPlacementPolicyWithNodeGroup}} does not
properly fallback to local rack when no nodes are available in remote racks,
resulting in an improper {{NotEnoughReplicasException}}.
{code:title=BlockPlacementPolicyWithNodeGroup.java}
@Override
protected void chooseRemoteRack(int numOfReplicas,
DatanodeDescriptor localMachine, HashMapNode, Node excludedNodes,
long blocksize, int maxReplicasPerRack, ListDatanodeDescriptor
results,
boolean avoidStaleNodes) throws NotEnoughReplicasException {
int oldNumOfReplicas = results.size();
// randomly choose one node from remote racks
try {
chooseRandom(
numOfReplicas,
~ +
NetworkTopology.getFirstHalf(localMachine.getNetworkLocation()),
excludedNodes, blocksize, maxReplicasPerRack, results,
avoidStaleNodes);
} catch (NotEnoughReplicasException e) {
chooseRandom(numOfReplicas - (results.size() - oldNumOfReplicas),
localMachine.getNetworkLocation(), excludedNodes, blocksize,
maxReplicasPerRack, results, avoidStaleNodes);
}
}
{code}
As currently coded the {{chooseRandom()}} call in the {{catch}} block will
never succeed as the set of nodes within the passed in node path (e.g.
{{/rack1/nodegroup1}}) is entirely contained within the set of excluded nodes
(both are the set of nodes within the same nodegroup as the node chosen first
replica).
The bug is that the fallback {{chooseRandom()}} call in the catch block
should be passing in the _complement_ of the node path used in the initial
{{chooseRandom()}} call in the try block (e.g. {{/rack1}}) - namely:
{code}
NetworkTopology.getFirstHalf(localMachine.getNetworkLocation())
{code}
This will yield the proper fallback behavior of choosing a random node from
_within the same rack_, but still excluding those nodes _in the same
nodegroup_
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4869) FsShell get command does not support writing to stdout
[ https://issues.apache.org/jira/browse/HDFS-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cristina L. Abad resolved HDFS-4869. Resolution: Won't Fix FsShell get command does not support writing to stdout -- Key: HDFS-4869 URL: https://issues.apache.org/jira/browse/HDFS-4869 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta, 0.23.8 Reporter: Cristina L. Abad Assignee: Cristina L. Abad Priority: Minor Attachments: 4869.branch-0.23.patch In FsShell the put command supports using \- in place of stdin, but this functionality (\- in place of stdout) is broken in the get command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[
https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680667#comment-13680667
]
Colin Patrick McCabe commented on HDFS-3934:
Just a quick reminder, if there are no comments on this we'll commit to
branch-2 / branch-2.1 in a day or two.
duplicative dfs_hosts entries handled wrong
---
Key: HDFS-3934
URL: https://issues.apache.org/jira/browse/HDFS-3934
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Andy Isaacson
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch,
HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch,
HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch,
HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch,
HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch,
HDFS-3934.016.patch, HDFS-3934.017.patch
A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by
hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}}
after the NN restarts because {{getDatanodeListForReport}} does not handle
such a pseudo-duplicate correctly:
# the Remove any nodes we know about from the map loop no longer has the
knowledge to remove the spurious entries
# the The remaining nodes are ones that are referenced by the hosts files
loop does not do hostname lookups, so does not know that the IP and hostname
refer to the same host.
Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in
the JSP output: The *Node* column shows :50010 as the nodename, with HTML
markup {{a
href=http://:50075/browseDirectory.jsp?namenodeInfoPort=50070amp;dir=%2Famp;nnaddr=172.29.97.196:8020;
title=172.29.97.216:50010:50010/a}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4819) Update Snapshot doc for HDFS-4758
[ https://issues.apache.org/jira/browse/HDFS-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680693#comment-13680693 ] Jing Zhao commented on HDFS-4819: - Thanks for the fix Nicholas! +1 pending Jenkins. Update Snapshot doc for HDFS-4758 - Key: HDFS-4819 URL: https://issues.apache.org/jira/browse/HDFS-4819 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h4819_20130611.patch Update Snapshot doc to clarify that nested snapshots are not allowed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[ https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4873: Resolution: Fixed Fix Version/s: 2.1.0-beta Status: Resolved (was: Patch Available) Thanks for the review Nicholas! I've committed this to trunk, branch-2 and branch-2.1.0-beta. callGetBlockLocations returns incorrect number of blocks for snapshotted files -- Key: HDFS-4873 URL: https://issues.apache.org/jira/browse/HDFS-4873 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hari Mankude Assignee: Jing Zhao Fix For: 2.1.0-beta Attachments: HDFS-4873.001.patch, HDFS-4873.002.patch callGetBlockLocations() returns all the blocks of a file even when they are not present in the snap version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[ https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680700#comment-13680700 ] Hudson commented on HDFS-4873: -- Integrated in Hadoop-trunk-Commit #3899 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3899/]) HDFS-4873. callGetBlockLocations returns incorrect number of blocks for snapshotted files. Contributed by Jing Zhao. (Revision 1491957) Result = SUCCESS jing9 : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1491957 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java callGetBlockLocations returns incorrect number of blocks for snapshotted files -- Key: HDFS-4873 URL: https://issues.apache.org/jira/browse/HDFS-4873 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hari Mankude Assignee: Jing Zhao Fix For: 2.1.0-beta Attachments: HDFS-4873.001.patch, HDFS-4873.002.patch callGetBlockLocations() returns all the blocks of a file even when they are not present in the snap version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4819) Update Snapshot doc for HDFS-4758
[
https://issues.apache.org/jira/browse/HDFS-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680848#comment-13680848
]
Tsz Wo (Nicholas), SZE commented on HDFS-4819:
--
Jenkins has +1'ed but somehow has not posted the result.
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587239/h4819_20130611.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4507//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4507//console
This message is automatically generated.
Update Snapshot doc for HDFS-4758
-
Key: HDFS-4819
URL: https://issues.apache.org/jira/browse/HDFS-4819
Project: Hadoop HDFS
Issue Type: Sub-task
Components: documentation
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
Attachments: h4819_20130611.patch
Update Snapshot doc to clarify that nested snapshots are not allowed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4849) Idempotent create and append operations.
[ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-4849: -- Attachment: idempotentCreate.patch The patch changes startFileInternal() to enable idempotent behaviour for create and append. The change is fairly simple, but there is a pretty comprehensive test case checking different retry and non-retry conditions. Please review. Idempotent create and append operations. Key: HDFS-4849 URL: https://issues.apache.org/jira/browse/HDFS-4849 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.0.4-alpha Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: idempotentCreate.patch create, append and delete operations can be made idempotent. This will reduce chances for a job or other app failures when NN fails over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4849) Idempotent create and append operations.
[ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-4849: -- Status: Patch Available (was: Open) Idempotent create and append operations. Key: HDFS-4849 URL: https://issues.apache.org/jira/browse/HDFS-4849 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.0.4-alpha Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: idempotentCreate.patch create, append and delete operations can be made idempotent. This will reduce chances for a job or other app failures when NN fails over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4849) Idempotent create and append operations.
[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680910#comment-13680910
]
Hadoop QA commented on HDFS-4849:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587352/idempotentCreate.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4508//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4508//console
This message is automatically generated.
Idempotent create and append operations.
Key: HDFS-4849
URL: https://issues.apache.org/jira/browse/HDFS-4849
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.0.4-alpha
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Attachments: idempotentCreate.patch
create, append and delete operations can be made idempotent. This will reduce
chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4819) Update Snapshot doc for HDFS-4758
[ https://issues.apache.org/jira/browse/HDFS-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4819: - Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Jing for reviewing the patch. I have committed this. Update Snapshot doc for HDFS-4758 - Key: HDFS-4819 URL: https://issues.apache.org/jira/browse/HDFS-4819 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Fix For: 2.1.0-beta Attachments: h4819_20130611.patch Update Snapshot doc to clarify that nested snapshots are not allowed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
