[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975154#comment-14975154
]
Yongjun Zhang commented on HDFS-9231:
-
Thanks Xiao for the new revs. +1 on 009 pending jenkins.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch,
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975559#comment-14975559
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 23m 17s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 9m 9s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 19s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 35s | The applied patch generated 2
new checkstyle issues (total was 427, now 427). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 48s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 33s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 65m 46s | Tests failed in hadoop-hdfs. |
| | | 120m 14s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768830/HDFS-9231.009.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 56e4f62 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13205/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/13205/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13205/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13205/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13205/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13205/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch,
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972793#comment-14972793
]
Xiao Chen commented on HDFS-9231:
-
Patch 08 addresses checkstyle errors.
The test failure is unrelate and passed locally. The findbugs warning is not
visible from the link...
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972834#comment-14972834
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 22m 1s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 9m 25s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 30s | The applied patch generated 3
new checkstyle issues (total was 427, now 428). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 37s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 17s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 51m 11s | Tests failed in hadoop-hdfs. |
| | | 104m 12s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768546/HDFS-9231.008.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 446212a |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13182/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/13182/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13182/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13182/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13182/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13182/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971469#comment-14971469
]
Xiao Chen commented on HDFS-9231:
-
Thanks a lot for the review [~yzhangal]!
{quote}
1. The description is not quite accurate per our discussion, suggest to modify.
Especially the patch actually does change (and fix) the behavior when without
-includeSnapshots.
{quote}
It was great to talk to you. I have updated the description. Modified patch
summary in the end of this comment.
{quote}
2. A possible optimization in FSDirSnapshotOp#getSnapshotFiles. It seems that
the sf variable could be calculated in caller for once before the loop in the
caller, and pass to this method.
{quote}
My apologies for the confusion, I added some comments in this method. But
getting sf for each snapshottable dir is needed, since /d1 and /d2 have
different snapshotlist.
{quote}
3. final INodesInPath iip = fsd.getINodesInPath4Write(snap, false); maybe
substituted with call to getINodesInPath
{quote}
Good catch! I updated the code to call {{getINode}} which invokes
{{getINodesInPath}}.
{quote}
4. The check if (!corruptFileBlocks.isEmpty()) in
listCorruptFileBlocksWithSnapshot is not needed
{quote}
Good call. Fixed.
{quote}
5. Add comment in listCorruptFileBlocks() before the call
namenode.getNamesystem().listCorruptFileBlocksWithSnapshot, to indicate that
snapshottableDirs is only relevant when -includeSnapshots is specified.
{quote}
Added a link to {{FSNamesystem#listCorruptFileBlocksWithSnapshot}} which
explains that parameter in javadoc.
{quote}
6. In listCorruptFileBlocksWithSnapshot, we can add
{code}
if (snapshottableDirs == null) {
continue;
}
{code}
to avoid the call to getSnapshotFiles.
{quote}
I'm not sure this is necessary. On one hand, it definitely saves 1 call stack.
On the other hand, with the existence of all those loops and checks, I think
the performance gain of saving 1 call stack would be trivial. And the nullity
check of snapshottableDirs is already performed as a first step in
{{getSnapshotFiles}}.
Attached patch 005 with the above modifications. Updated summary below:
- {{fsck -list-corruptfileblocks -includeSnapshots}} will also show full dir of
snapshots
- {{fsck -list-corruptfileblocks}} without -includeSnapshots will not show
corrupt blocks that only have snapshot files
- NameNode WebUI's way of showing corrupted files/blocks unchanged.
- Added a sentence in NN WebUI to hint the admin to run fsck with
-includeSnapshots, if there're snapshots present in the system.
- Some refactoring to reuse existing code in new methods getSnapshottableDirs
and ListCorruptFileBlocksWithSnapshot
- The reasoning of keep minimal change to NN WebUI and fsck without
-includeSnapshots is that getting all possible snapshots may be slow.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971662#comment-14971662
]
Yongjun Zhang commented on HDFS-9231:
-
Hi [~xiaochen],
Thanks for the new rev. Some further comments:
1.
{quote}
My apologies for the confusion, I added some comments in this method. But
getting sf for each snapshottable dir is needed, since /d1 and /d2 have
different snapshotlist.
{quote}
I meant to precalcuate a list of "sf"s, and pass the list to
FSDirSnapshotOp#getSnapshotFiles so to avoid doing it once for each file. Sorry
for not having been clear.
2.
{quote}
Added a link to FSNamesystem#listCorruptFileBlocksWithSnapshot which explains
that parameter in javadoc.
{quote}
I think it's ok to not have this link since it looks a bit redundant. So
consider I took back my earlier comment since you have javadoc in
FSNamesystem#listCorruptFileBlocksWithSnapshot. :-) sorry about that.
3.
{quote}
Good catch! I updated the code to call getINode which invokes getINodesInPath.
{quote}
Good change to use {{getINode}}.
4. In {{testFsckListCorruptSnapshotFiles() }}
Suggest to introduce two variables, numFiles, numSnapshotFiles, initially
numFile = 3 and numSnapshotFiles = 0, then when you create /corruptData/file,
increment numFiles by 1, when you create snapshot, set numSnapshotFiles to 4.
when deleting the file, decrement numFile by 1. When gets to assertion part,
the first one check against (numFiles + numSnapshotFiles) (in place of the
"7"), the second one check against numFiles.
Thanks.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972259#comment-14972259
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 23m 51s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 10m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 13m 58s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 31s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 58s | The applied patch generated 5
new checkstyle issues (total was 427, now 430). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 2m 1s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 46s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 30s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 4m 19s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 63m 51s | Tests failed in hadoop-hdfs. |
| | | 125m 24s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768427/HDFS-9231.007.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 15eb84b |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13172/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/13172/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13172/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13172/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13172/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13172/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch, HDFS-9231.007.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971947#comment-14971947
]
Xiao Chen commented on HDFS-9231:
-
Per the offline talk, we can make {{FSDirSnapshotOp#getSnapshotFiles}} more
elegant by adding an interface on {{Snapshot}} to get the full snapshot root
path, hence eliminating the need of pass in snapshottableDir. Attached patch 07
to do this. Thanks Yongjun.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch, HDFS-9231.007.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971862#comment-14971862
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 35s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 17s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 33s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 22s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 51m 37s | Tests failed in hadoop-hdfs. |
| | | 96m 22s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768348/HDFS-9231.005.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 600ad7b |
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13161/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13161/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13161/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13161/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971855#comment-14971855
]
Xiao Chen commented on HDFS-9231:
-
Hi Yongjun,
Thanks again for the quick review! Your comments makes sense, I attached patch
6 to address them. Sorry I missed the point of #1 earlier.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch,
> HDFS-9231.006.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt
> blocks with the original file dir instead of the snapshot dir, and {{fsck
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck
> will still show that deleted file as corrupted, although what's actually
> corrupted is the snapshot.
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970046#comment-14970046
]
Yongjun Zhang commented on HDFS-9231:
-
Hi [~xiaochen],
Thanks for reporting the issue and the patch. Good to have discussed in person.
The patch looks good in general. I have some comments here:
1. The description is not quite accurate per our discussion, suggest to modify.
Especially the patch actually does change (and fix) the behavior when without
-includeSnapshots.
2. A possible optimization in FSDirSnapshotOp#getSnapshotFiles. It seems that
the {{sf}} variable could be calculated in caller for once before the loop in
the caller, and pass to this method.
3. {{final INodesInPath iip = fsd.getINodesInPath4Write(snap, false);}} maybe
substituted with call to {{getINodesInPath}}
4. The check {{if (!corruptFileBlocks.isEmpty())}} in
{{listCorruptFileBlocksWithSnapshot}} is not needed
5. Add comment in {{listCorruptFileBlocks()}} before the call
{{namenode.getNamesystem().listCorruptFileBlocksWithSnapshot}}, to indicate
that {{snapshottableDirs}} is only relevant when -includeSnapshots is
specified.
6. In {{listCorruptFileBlocksWithSnapshot}}, we can add
{code}
if (snapshottableDirs == null) {
continue;
}
{code}
to avoid the call to {{getSnapshotFiles}}.
Thanks.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967525#comment-14967525
]
Xiao Chen commented on HDFS-9231:
-
patch 004 is attached. I think I should re-summarize it below:
- fsck from command line with {{-includeSnapshots}} will also show full dir of
snapshots
- fsck from command line without {{-includeSnapshots}} behavior unchanged
- NameNode WebUI's way of showing corrupted files/blocks unchanged.
- Added a sentence in NN WebUI to hint the admin to run fsck with
{{-includeSnapshots}} if there're snapshots present in the system.
- Some refactoring to reuse existing code in new methods
{{getSnapshottableDirs}} and {{ListCorruptFileBlocksWithSnapshot}}
- The reasoning of keep minimal change to NN WebUI and fsck without
{{-includeSnapshots}} is that getting all possible snapshots may be slow, since
it's user configured.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967815#comment-14967815
]
Xiao Chen commented on HDFS-9231:
-
The test failures look unrelated and passed locally.
Findbugs seems to be having problem.
{noformat}
Exception in thread "main" java.io.FileNotFoundException:
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/trunkFindbugsWarningshadoop-hdfs.xml
(No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:146)
at
edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231)
at
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308)
at
edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295)
at edu.umd.cs.findbugs.workflow.Filter.main(Filter.java:712)
Pre-patch trunk findbugs is broken?
{noformat}
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14967726#comment-14967726
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 10s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 30s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 35s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 34s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 51m 8s | Tests failed in hadoop-hdfs. |
| | | 94m 47s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.tools.TestDFSHAAdminMiniCluster |
| | hadoop.hdfs.tools.TestStoragePolicyCommands |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| Timed out tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767823/HDFS-9231.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e27c2ae |
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13110/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13110/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13110/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13110/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch, HDFS-9231.004.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957369#comment-14957369
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 11s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 54s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 21s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 22s | The applied patch generated 4
new checkstyle issues (total was 370, now 372). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 35s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 49m 52s | Tests failed in hadoop-hdfs. |
| | | 96m 2s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.fs.TestGlobPaths |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766570/HDFS-9231.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0d77e85 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12981/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12981/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12981/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12981/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12981/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956329#comment-14956329
]
Xiao Chen commented on HDFS-9231:
-
Attached patch 001. Description below:
* fsck from command line with -includeSnapshots will also show full dir of
snapshots
* fsck from command line without -includeSnapshots behavior unchanged
* NameNode WebUI always show full dir of snapshots
* Some refactoring to reuse {{getSnapshottableDirs}} and
{{ListCorruptFileBlocksWithSnapshot}}
* Getting all possible snapshots is not so efficient, but considering fsck
should not be performed frequently and nothing is added into
{{listCorruptFileBlocks}} where fslock is used, the impact should be minimal.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956456#comment-14956456
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 36s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 4s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 56s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 40s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 27s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 50m 18s | Tests failed in hadoop-hdfs. |
| | | 97m 10s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFSNamesystemMBean |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766475/HDFS-9231.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2a98724 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12971/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12971/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12971/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12971/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957961#comment-14957961
]
Hadoop QA commented on HDFS-9231:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 11s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 6s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 27s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 26s | The applied patch generated 3
new checkstyle issues (total was 370, now 371). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 30s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 25s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 50m 13s | Tests failed in hadoop-hdfs. |
| | | 96m 48s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.fs.TestGlobPaths |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766629/HDFS-9231.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3d50855 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12990/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12990/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12990/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12990/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12990/console |
This message was automatically generated.
> fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
> ---
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch,
> HDFS-9231.003.patch
>
>
> For snapshot files, fsck shows corrupt blocks with the original file dir
> instead of the snapshot dir.
> This can be confusing since even when the original file is deleted, a new
> fsck run will still show that file as corrupted although what's actually
> corrupted is the snapshot.
> This is true even when given the -includeSnapshots option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9231) fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot
[ https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958010#comment-14958010 ] Xiao Chen commented on HDFS-9231: - The pre-patch Findbugs warning is unrelated. (see HDFS-9242) The checkstyle issues are not introduced by this patch. The test failures are unrelated, and passed locally. > fsck doesn't explicitly list when Bad Replicas/Blocks are in a snapshot > --- > > Key: HDFS-9231 > URL: https://issues.apache.org/jira/browse/HDFS-9231 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, > HDFS-9231.003.patch > > > For snapshot files, fsck shows corrupt blocks with the original file dir > instead of the snapshot dir. > This can be confusing since even when the original file is deleted, a new > fsck run will still show that file as corrupted although what's actually > corrupted is the snapshot. > This is true even when given the -includeSnapshots option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
