[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679542#comment-13679542 ] Ravi Prakash commented on HDFS-4878: Hi Plamen! Thanks for your review. bq. Because our condition includes .hasNext() we will stop before trying to go outside our iterator. So we are safe with this change. True! We won't throw an exception, but we would have skipped over a block. We won't get to this block again until the next iteration and a block with lower priority may be scheduled before a higher one. Its rather simple to fix this, so I would much rather that we did. (Although I noticed that there are already several instances in which we don't). Should we handle this in this JIRA / another? Could we refactor UnderReplicatedBlocks so that decrementReplicationIndex is called from within remove()? I would be a +1 for that change. On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4869) FsShell get command does not support writing to stdout
[ https://issues.apache.org/jira/browse/HDFS-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679572#comment-13679572 ] Daryn Sharp commented on HDFS-4869: --- I don't think this will work when one of the sources is a directory, which illustrates issues associated with allowing a target of – to mean stdout. The command either has to add a restriction barring directories for output to stdout which unbalances the command, ie. can't simply swap in – for the dest with any arguments. Or further modified/hacked to allow recursing, skip the construction of a local path for the dest, checking existence and type of dest, creating dirs that don't exist, etc. It starts to get complicated to fuse the existing behavior of cat into get/copyToLocal. Unless there is a compelling need for this change, I'd still prefer won't fix. FsShell get command does not support writing to stdout -- Key: HDFS-4869 URL: https://issues.apache.org/jira/browse/HDFS-4869 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta, 0.23.8 Reporter: Cristina L. Abad Assignee: Cristina L. Abad Priority: Minor Attachments: 4869.branch-0.23.patch In FsShell the put command supports using \- in place of stdin, but this functionality (\- in place of stdout) is broken in the get command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4869) FsShell get command does not support writing to stdout
[ https://issues.apache.org/jira/browse/HDFS-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679574#comment-13679574 ] Daryn Sharp commented on HDFS-4869: --- It's also deviating from the posix behavior to which FsShell tries to adhere. FsShell get command does not support writing to stdout -- Key: HDFS-4869 URL: https://issues.apache.org/jira/browse/HDFS-4869 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta, 0.23.8 Reporter: Cristina L. Abad Assignee: Cristina L. Abad Priority: Minor Attachments: 4869.branch-0.23.patch In FsShell the put command supports using \- in place of stdin, but this functionality (\- in place of stdout) is broken in the get command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4869) FsShell get command does not support writing to stdout
[ https://issues.apache.org/jira/browse/HDFS-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679585#comment-13679585 ] Cristina L. Abad commented on HDFS-4869: I tried these things and they all worked with the current patch: hdfs dfs -get /path/to/aFile - Result: file is printed to stdout hdfs dfs -get /path/to/aDirWithTwoFiles/* - Result: content of both files is printed to stdout hdfs dfs -get /path/to/aFileThatDoesNotExist - Result: error message No such file or directory hdfs dfs -get /path/to/aFile* - Result: prints the two files that match aFile* to stdout hdfs dfs -get /path/to/aDirWithTwoFiles - Result: error message: Is a directory I guess you were referring to the last case in your comment. I thought this was the same behavior of the prior -get aDir - implementation, but I could be wrong about this. In any case, I am OK with leaving it as won't fix; at least this jira now documents the fact that -get aDir - does not work in the current versions of Hadoop. FsShell get command does not support writing to stdout -- Key: HDFS-4869 URL: https://issues.apache.org/jira/browse/HDFS-4869 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta, 0.23.8 Reporter: Cristina L. Abad Assignee: Cristina L. Abad Priority: Minor Attachments: 4869.branch-0.23.patch In FsShell the put command supports using \- in place of stdin, but this functionality (\- in place of stdout) is broken in the get command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4897) Moving/copying a directory does not include the directory's quota settings
Stephen Chu created HDFS-4897:
-
Summary: Moving/copying a directory does not include the
directory's quota settings
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
[~atm] and I found that when a directory is copied or moved, its quotas
settings aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop` fs
-cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user takes
snapshots and backs up a subtree using snapshots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[ https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4873: Attachment: HDFS-4873.002.patch Thanks Nicholas! Update the patch to address your comments. callGetBlockLocations returns incorrect number of blocks for snapshotted files -- Key: HDFS-4873 URL: https://issues.apache.org/jira/browse/HDFS-4873 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Affects Versions: 3.0.0, 2.1.0-beta Reporter: Hari Mankude Assignee: Jing Zhao Attachments: HDFS-4873.001.patch, HDFS-4873.002.patch callGetBlockLocations() returns all the blocks of a file even when they are not present in the snap version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4581) DataNode#checkDiskError should not be called on network errors
[ https://issues.apache.org/jira/browse/HDFS-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4581: -- Target Version/s: 1.2.1 Fix Version/s: (was: 1.3.0) 1.2.1 I merged this change to branch 1.2 to be picked up for 1.2.1 DataNode#checkDiskError should not be called on network errors -- Key: HDFS-4581 URL: https://issues.apache.org/jira/browse/HDFS-4581 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 1.1.1, 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Rohit Kochar Assignee: Rohit Kochar Fix For: 0.23.7, 2.0.4-alpha, 1.2.1 Attachments: HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-4699:
--
Target Version/s: 1.2.1 (was: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0)
Affects Version/s: (was: 1.3.0)
1.2.0
I merged this patch to branch-1.2 to be picked up for 1.2.1.
TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
---
Key: HDFS-4699
URL: https://issues.apache.org/jira/browse/HDFS-4699
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 1.2.0, 3.0.0, 0.23.7, 2.0.4-alpha
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.0.0, 2.1.0-beta, 0.23.8, 1.3.0
Attachments: HDFS-4699.1.patch, HDFS-4699.branch-0.23.1.patch,
HDFS-4699.branch-1.1.patch
I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
sporadically due to timeout during {{loopRecoverLease}}, which waits for up
to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Moving/copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679680#comment-13679680
]
Daryn Sharp commented on HDFS-4897:
---
I wouldn't expect a move to drop the quotas, but it does stand to reason that
copy shouldn't transfer the quota - at least by default. Perhaps the –p option
could be extended, but one difficulty is that neither FileSystem nor
AbstractFileSystem expose quota operations.
Moving/copying a directory does not include the directory's quota settings
--
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied or moved, its quotas
settings aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and backs up a subtree using snapshots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4880) Diagnostic logging while loading name/edits files
[ https://issues.apache.org/jira/browse/HDFS-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4880: -- Target Version/s: 1.2.1 Affects Version/s: (was: 1.3.0) 1.0.0 Fix Version/s: (was: 1.3.0) 1.2.1 I merged this to branch-1.2 for release 1.2.1 Diagnostic logging while loading name/edits files - Key: HDFS-4880 URL: https://issues.apache.org/jira/browse/HDFS-4880 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 1.0.0, 3.0.0 Reporter: Arpit Agarwal Assignee: Suresh Srinivas Fix For: 2.1.0-beta, 1.2.1 Attachments: HDFS-4880.branch-1.patch, HDFS-4880.branch-1.patch, HDFS-4880.trunk.patch Add some minimal diagnostic logging to help determine location of the files being loaded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Moving/copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679689#comment-13679689
]
Aaron T. Myers commented on HDFS-4897:
--
bq. Perhaps the –p option could be extended,
I think that's probably the right approach.
bq. but one difficulty is that neither FileSystem nor AbstractFileSystem expose
quota operations.
Perhaps we should add a preserveAttributes(Path src, Path dst) call to the
FileSystem/AbstractFileSystem interfaces with default implementations that do
nothing. That way implementing file systems with FS-specific attributes can do
their own thing.
Moving/copying a directory does not include the directory's quota settings
--
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied or moved, its quotas
settings aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and backs up a subtree using snapshots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Chu updated HDFS-4897:
--
Description:
[~atm] and I found that when a directory is copied, its quotas settings aren't
included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop` fs
-cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user takes
snapshots and restores a subtree using snapshots because we use copy (not
allowed to move snapshots).
was:
[~atm] and I found that when a directory is copied or moved, its quotas
settings aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop` fs
-cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user takes
snapshots and backs up a subtree using snapshots.
Summary: Copying a directory does not include the directory's quota
settings (was: Moving/copying a directory does not include the directory's
quota settings)
Sorry, I made a mistake in the initial filing. _hdfs dfs -mv_ will retain the
quotas settings, but not _-cp_. This still means that when we want to restore a
subtree using snapshots, we won't be able to retain quotas settings because we
can't use move.
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4261: -- Target Version/s: 1.2.1, 2.0.5-alpha (was: 1.3.0, 2.0.5-alpha) Fix Version/s: (was: 1.3.0) (was: 1-win) 1.2.1 I merged this to branch-1.2 for release 1.2.1 TestBalancerWithNodeGroup times out --- Key: HDFS-4261 URL: https://issues.apache.org/jira/browse/HDFS-4261 Project: Hadoop HDFS Issue Type: Bug Components: balancer Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha Reporter: Tsz Wo (Nicholas), SZE Assignee: Junping Du Fix For: 2.1.0-beta, 1.2.1 Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, HDFS-4261-branch-2.patch, HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, test-balancer-with-node-group-timeout.txt When I manually ran TestBalancerWithNodeGroup, it always timed out in my machine. Looking at the Jerkins report [build #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], TestBalancerWithNodeGroup somehow was skipped so that the problem was not detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679717#comment-13679717
]
Jing Zhao commented on HDFS-4897:
-
Hi Stephen, I tried hadoop fs -mv and in my test the same quota setting were
included in the new directory:
{code}
$ hadoop dfsadmin -setQuota 100 /user/jing
$ hadoop fs -count -q /user/jing
100 99none inf1
0 0 /user/jing
$ hadoop fs -mv /user/jing /user/jing-copy2
$ hadoop fs -count -q /user/jing-copy2
100 99none inf1
0 0 /user/jing-copy2
{code}
So is this issue only about copying a directory?
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679723#comment-13679723
]
Suresh Srinivas commented on HDFS-4897:
---
bq. Sorry, I made a mistake in the initial filing. hdfs dfs -mv will retain the
quotas settings, but not -cp. This still means that when we want to restore a
subtree using snapshots, we won't be able to retain quotas settings because we
can't use move.
Can you explain what you mean by restore a subtree using snapshots? Currently
there is no restore supported. If you want to restore some data, you need to
copy, which should follow the current copy semantics.
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679724#comment-13679724
]
Jing Zhao commented on HDFS-4897:
-
bq. Sorry, I made a mistake in the initial filing. hdfs dfs -mv will retain the
quotas settings, but not -cp.
Yeah, I should refresh the page before posting my last comment :)
bq. This still means that when we want to restore a subtree using snapshots, we
won't be able to retain quotas settings because we can't use move.
We can either extend the existing command (as suggested by Daryn and Aaron) or
we can provide a separate snapshot tool which can retain the quota setting when
restoring.
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679740#comment-13679740
]
Stephen Chu commented on HDFS-4897:
---
bq. Can you explain what you mean by restore a subtree using snapshots?
Currently there is no restore supported. If you want to restore some data, you
need to copy, which should follow the current copy semantics.
I meant using copy to restore some data. Sorry for being unclear.
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679745#comment-13679745
]
Suresh Srinivas commented on HDFS-4897:
---
bq. I meant using copy to restore some data. Sorry for being unclear.
If that is the case, the suggest of cp -p, to enable the semantics you are
looking for, makes sense.
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679745#comment-13679745
]
Suresh Srinivas edited comment on HDFS-4897 at 6/10/13 6:39 PM:
bq. I meant using copy to restore some data. Sorry for being unclear.
If that is the case, the suggestion of cp -p, to enable the semantics you are
looking for, makes sense.
was (Author: sureshms):
bq. I meant using copy to restore some data. Sorry for being unclear.
If that is the case, the suggest of cp -p, to enable the semantics you are
looking for, makes sense.
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4883) complete() should verify fileId
[ https://issues.apache.org/jira/browse/HDFS-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679764#comment-13679764 ] Brandon Li commented on HDFS-4883: -- Hi Konstantin, it makes sense to use fileId in complete(). Adding fileId can do things right in two cases: 1. after renaming an opened file, the client should still be able to close the file. 2. after renaming an opened file, the client should not close the wrong file (it could if using only path in complete()). Here is an example of case2. 1. Client opens file A and B. 2. Delete B, rename A to B. 3. Close B. Without fileId in complete(), original file A is closed. complete() should verify fileId --- Key: HDFS-4883 URL: https://issues.apache.org/jira/browse/HDFS-4883 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.1.0-beta Reporter: Konstantin Shvachko It seems inconsistent that complete() avoids verifying real fileId, while addBlock() does it meticulously. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4883) complete() should verify fileId
[ https://issues.apache.org/jira/browse/HDFS-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679766#comment-13679766 ] Brandon Li commented on HDFS-4883: -- BTW, adding fileId to complete() is not an incompatible change. complete() should verify fileId --- Key: HDFS-4883 URL: https://issues.apache.org/jira/browse/HDFS-4883 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.1.0-beta Reporter: Konstantin Shvachko It seems inconsistent that complete() avoids verifying real fileId, while addBlock() does it meticulously. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4888) Refactor and fix FSNamesystem.getTurnOffTip to sanity
[ https://issues.apache.org/jira/browse/HDFS-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679775#comment-13679775 ] Kousuke Saruta commented on HDFS-4888: -- [~raviprak], could you give us more details? Refactor and fix FSNamesystem.getTurnOffTip to sanity - Key: HDFS-4888 URL: https://issues.apache.org/jira/browse/HDFS-4888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8 Reporter: Ravi Prakash Assignee: Ravi Prakash e.g. When resources are low, the command to leave safe mode is not printed. This method is unnecessarily complex -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4873) callGetBlockLocations returns incorrect number of blocks for snapshotted files
[
https://issues.apache.org/jira/browse/HDFS-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679800#comment-13679800
]
Hadoop QA commented on HDFS-4873:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587106/HDFS-4873.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4502//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4502//console
This message is automatically generated.
callGetBlockLocations returns incorrect number of blocks for snapshotted files
--
Key: HDFS-4873
URL: https://issues.apache.org/jira/browse/HDFS-4873
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Hari Mankude
Assignee: Jing Zhao
Attachments: HDFS-4873.001.patch, HDFS-4873.002.patch
callGetBlockLocations() returns all the blocks of a file even when they are
not present in the snap version
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4897) Copying a directory does not include the directory's quota settings
[
https://issues.apache.org/jira/browse/HDFS-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679834#comment-13679834
]
Kousuke Saruta commented on HDFS-4897:
--
In 3.0.0 implementation, cp -p command preserves only mtime, atime, owner,
group, permisson of files.
I think that as well as quota, other attributes should be preserved by cp -p
command. So, I think that adding preserveAttributes(Path src, Path dst) is good
idea.
{code}
protected void copyFileToTarget(PathData src, PathData target) throws
IOException {
src.fs.setVerifyChecksum(verifyChecksum);
InputStream in = null;
try {
in = src.fs.open(src.path);
copyStreamToTarget(in, target);
if(preserve) {
target.fs.setTimes(
target.path,
src.stat.getModificationTime(),
src.stat.getAccessTime());
target.fs.setOwner(
target.path,
src.stat.getOwner(),
src.stat.getGroup());
target.fs.setPermission(
target.path,
src.stat.getPermission());
}
} finally {
IOUtils.closeStream(in);
}
}
{code}
Copying a directory does not include the directory's quota settings
---
Key: HDFS-4897
URL: https://issues.apache.org/jira/browse/HDFS-4897
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Stephen Chu
Labels: quota
[~atm] and I found that when a directory is copied, its quotas settings
aren't included.
{code}
[04:21:33] atm@simon:~/src/apache/hadoop.git$ hadoop fs -ls /user
Found 2 items
drwxr-xr-x - atm atm 0 2013-06-07 16:17 /user/atm
drwx-- - hdfs supergroup 0 2013-06-07 16:21 /user/hdfs
[04:21:44] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q /user/atm
100 91none inf7
2 3338 /user/atm
[04:21:51] atm@simon:~/src/apache/hadoop.git$ sudo -u hdfs -E `which hadoop`
fs -cp /user/atm /user/atm-copy
[04:22:00] atm@simon:~/src/apache/hadoop.git$ hadoop fs -count -q
/user/atm-copy
none infnone inf6
1 3338 /user/atm-copy
{code}
This also means that a user will not retain quotas settings when the user
takes snapshots and restores a subtree using snapshots because we use copy
(not allowed to move snapshots).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4883) complete() should verify fileId
[ https://issues.apache.org/jira/browse/HDFS-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-4883: - Hadoop Flags: (was: Incompatible change) complete() should verify fileId --- Key: HDFS-4883 URL: https://issues.apache.org/jira/browse/HDFS-4883 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.1.0-beta Reporter: Konstantin Shvachko It seems inconsistent that complete() avoids verifying real fileId, while addBlock() does it meticulously. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878_branch2.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: HDFS-4878.patch On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: HDFS-4878_branch2.patch On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Patch Available (was: Open) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4752) TestRBWBlockInvalidation fails on Windows due to file locking
[ https://issues.apache.org/jira/browse/HDFS-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679998#comment-13679998 ] Suresh Srinivas commented on HDFS-4752: --- Chris, I agree with your analys about disable the test on Windows, instead of adding complexity of config hooks. +1 for the patch. TestRBWBlockInvalidation fails on Windows due to file locking - Key: HDFS-4752 URL: https://issues.apache.org/jira/browse/HDFS-4752 Project: Hadoop HDFS Issue Type: Bug Components: datanode, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HDFS-4752.1.patch, HDFS-4752.2.patch The test attempts to invalidate a block by deleting its block file and meta file. This happens while a datanode thread holds the files open for write. On Windows, this causes a locking conflict, and the test fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Open (was: Patch Available) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4752) TestRBWBlockInvalidation fails on Windows due to file locking
[ https://issues.apache.org/jira/browse/HDFS-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680008#comment-13680008 ] Hudson commented on HDFS-4752: -- Integrated in Hadoop-trunk-Commit #3892 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3892/]) HDFS-4752. TestRBWBlockInvalidation fails on Windows due to file locking. Contributed by Chris Nauroth. (Revision 1491631) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1491631 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java TestRBWBlockInvalidation fails on Windows due to file locking - Key: HDFS-4752 URL: https://issues.apache.org/jira/browse/HDFS-4752 Project: Hadoop HDFS Issue Type: Bug Components: datanode, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HDFS-4752.1.patch, HDFS-4752.2.patch The test attempts to invalidate a block by deleting its block file and meta file. This happens while a datanode thread holds the files open for write. On Windows, this causes a locking conflict, and the test fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878_branch2.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878_branch2.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: HDFS-4878_branch2.patch HDFS-4878_branch2.patch On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680018#comment-13680018 ] Tao Luo commented on HDFS-4878: --- Hi Ravi. Thanks for your suggestion. I was thinking along the same lines, and have uploaded the patch. On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[
https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680023#comment-13680023
]
Hadoop QA commented on HDFS-4878:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587154/HDFS-4878_branch2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4504//console
This message is automatically generated.
On Remove Block, Block is not Removed from neededReplications queue
---
Key: HDFS-4878
URL: https://issues.apache.org/jira/browse/HDFS-4878
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0
Reporter: Tao Luo
Assignee: Tao Luo
Fix For: 3.0.0
Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch
Remove block removes the specified block from pendingReplications, but not
from neededReplications queue.
The fix would be to remove from neededReplications as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Patch Available (was: Open) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: HDFS-4878.patch On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Open (was: Patch Available) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[
https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680093#comment-13680093
]
Hadoop QA commented on HDFS-4878:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587148/HDFS-4878.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4503//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4503//console
This message is automatically generated.
On Remove Block, Block is not Removed from neededReplications queue
---
Key: HDFS-4878
URL: https://issues.apache.org/jira/browse/HDFS-4878
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0
Reporter: Tao Luo
Assignee: Tao Luo
Fix For: 3.0.0
Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch
Remove block removes the specified block from pendingReplications, but not
from neededReplications queue.
The fix would be to remove from neededReplications as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680094#comment-13680094 ] Junping Du commented on HDFS-4261: -- Ok. Thanks Suresh! TestBalancerWithNodeGroup times out --- Key: HDFS-4261 URL: https://issues.apache.org/jira/browse/HDFS-4261 Project: Hadoop HDFS Issue Type: Bug Components: balancer Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha Reporter: Tsz Wo (Nicholas), SZE Assignee: Junping Du Fix For: 2.1.0-beta, 1.2.1 Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, HDFS-4261-branch-2.patch, HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, test-balancer-with-node-group-timeout.txt When I manually ran TestBalancerWithNodeGroup, it always timed out in my machine. Looking at the Jerkins report [build #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], TestBalancerWithNodeGroup somehow was skipped so that the problem was not detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680096#comment-13680096 ] Konstantin Shvachko commented on HDFS-4878: --- Strictly speaking we should decrementReplicationIndex() if we remove the block at a position before the replicationIndex, and should not decrement if the block position is after the replicationIndex. In general we don't know the position of the block in the queue, so we cannot know whether the index should be decremented. It should be targeted in a different jira if desired. The worst thing that happens if we don't change replicationIndex is that some blocks will be replicated later, which happens with some blocks one way or another. Eventually everything will be processed and that is what's important. I'd rather commit the original patches. On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4372) Track NameNode startup progress
[
https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680112#comment-13680112
]
Jing Zhao commented on HDFS-4372:
-
Hi Chris,
The patch looks very good to me. Currently the patch needs rebase (mainly
for the changes in FSImageFormat.java). Some thoughts about the patch:
1. FSImageFormat.java
After snapshot is supported, the number of files/directories that is read in
the head of the fsimage (denoted as N_total), may no longer be equal to the
number of times of invocation of loadINode(...). This is because when we
compute the total namespace quota usage, some nodes may be counted multiple
times (in case that rename happened with existence of snapshots). Thus when
using a counter to track the number of inodes loaded from the fsimage, it is
challenging to keep the final value of the counter equal to N_total.
So instead of adding extra complex counting logic to fsimage loading, do you
think we can simply set the counter of inodes to N_total after finishing
loading the inodes from a fsimage?
2. FSNamesystem.java, in loadFSImage(...),
{code}
+ } else {
+// No need to save, so mark the phase done.
+StartupProgress prog = NameNode.getStartupProgress();
+prog.beginPhase(Phase.SAVING_CHECKPOINT);
+prog.endPhase(Phase.SAVING_CHECKPOINT);
{code}
Here can we present that we have not saved the namespace in the WebUI? (Users
may also get the same information when they see a very small elapsed time for
the phase).
Some minor issues:
3. FSEditLogLoader.java
Why do we need to create a step in both loadFSEdits(...) and
loadEditRecords(...)?
4. StartupProgress.java
4.1 There are some unused imports.
4.2 In lazyInitStep(...), maybe the code
{code}
if (!steps.containsKey(step)) {
steps.putIfAbsent(step, new StepTracking());
}
return steps.get(step);
{code}
can be simplified to
{code}
steps.putIfAbsent(step, new StepTracking());
return steps.get(step);
{code}
or something like
{code}
StepTracking st = new StepTracking();
StepTracking oldValue = steps.putIfAbsent(step, st);
return oldValue != null ? oldValue : st;
{code}
5. StepTracking.java and PhaseTracking.java
Can we use long instead of Long for beginTime/endTime/total? We can initialize
them with -1.
Track NameNode startup progress
---
Key: HDFS-4372
URL: https://issues.apache.org/jira/browse/HDFS-4372
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-4372.1.patch
Track detailed progress information about the steps of NameNode startup to
enable display to users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[
https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680111#comment-13680111
]
Hadoop QA commented on HDFS-4878:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587159/HDFS-4878.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4505//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4505//console
This message is automatically generated.
On Remove Block, Block is not Removed from neededReplications queue
---
Key: HDFS-4878
URL: https://issues.apache.org/jira/browse/HDFS-4878
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.0.0
Reporter: Tao Luo
Assignee: Tao Luo
Fix For: 3.0.0
Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch, HDFS-4878.patch
Remove block removes the specified block from pendingReplications, but not
from neededReplications queue.
The fix would be to remove from neededReplications as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878_branch2.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: (was: HDFS-4878.patch) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Open (was: Patch Available) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: HDFS-4878.patch On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Attachment: HDFS-4878_branch2.patch On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Luo updated HDFS-4878: -- Status: Patch Available (was: Open) On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 3.0.0 Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-4878: -- Resolution: Fixed Fix Version/s: (was: 3.0.0) 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thank you Tao. On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 2.1.0-beta Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4878) On Remove Block, Block is not Removed from neededReplications queue
[ https://issues.apache.org/jira/browse/HDFS-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680143#comment-13680143 ] Hudson commented on HDFS-4878: -- Integrated in Hadoop-trunk-Commit #3893 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3893/]) HDFS-4878. On Remove Block, block is not removed from neededReplications queue. Contributed by Tao Luo. (Revision 1491671) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1491671 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java On Remove Block, Block is not Removed from neededReplications queue --- Key: HDFS-4878 URL: https://issues.apache.org/jira/browse/HDFS-4878 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Tao Luo Assignee: Tao Luo Fix For: 2.1.0-beta Attachments: HDFS-4878_branch2.patch, HDFS-4878.patch Remove block removes the specified block from pendingReplications, but not from neededReplications queue. The fix would be to remove from neededReplications as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4372) Track NameNode startup progress
[
https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680190#comment-13680190
]
Chris Nauroth commented on HDFS-4372:
-
Jing, thanks very much for reviewing the patch! I'll work on a rebase.
Meanwhile, here are responses to your feedback.
{quote}
So instead of adding extra complex counting logic to fsimage loading, do you
think we can simply set the counter of inodes to N_total after finishing
loading the inodes from a fsimage?
{quote}
The desire was to have a running total of inodes loaded so far, to give a
reasonable indicator of progress up to this point. Unfortunately, setting the
counter to the total after finishing loading would give us a very coarse
progress update, which would appear to suddenly jump from 0% done to 100% done.
I wrote all of this code before snapshots, so I'll take a look to see if there
is something more reasonable to do now that snapshots are in trunk.
{quote}
Here can we present that we have not saved the namespace in the WebUI?
{quote}
I'll try to find a way to display save not required or similar for this case.
{quote}
Why do we need to create a step in both loadFSEdits(...) and
loadEditRecords(...)?
{quote}
Since the 2 different methods can be called in isolation, I found it necessary
to create a step independently in both. When I rebase, I'll check again to see
if this is truly necessary.
{quote}
In lazyInitStep(...), maybe the code...
{quote}
I started with something similar to your suggestion, but I wanted to be
conscious of instantiating a lot of small objects that would just get discarded
immediately and cause garbage collection churn. The problem with the
alternative you suggested is that we would always call {{new StepTracking()}}
to create a new object, even if one already existed in the map. I added the
check for {{containsKey}} to limit the impact of that. Maybe it's just a minor
optimization, but considering that these methods could be called in tight loops
during startup, I wanted to keep it under control.
{quote}
Can we use long instead of Long for beginTime/endTime/total? We can initialize
them with -1.
{quote}
I used nullable Long just because logically the values might not be initialized
yet. By switching to primitive long with -1 reserved for uninitialized, is
the intent to reduce memory footprint of a boxed Long? If so, then I
definitely can make that change.
Track NameNode startup progress
---
Key: HDFS-4372
URL: https://issues.apache.org/jira/browse/HDFS-4372
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-4372.1.patch
Track detailed progress information about the steps of NameNode startup to
enable display to users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
