[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685382#comment-15685382
]
Brahma Reddy Battula commented on HDFS-10941:
-
[~zhz] looks chnages.txt not update for this jira in branch-2.7..can you please
update .?
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10941.001.patch, HDFS-10941.002.patch,
> HDFS-10941.002.patch, HDFS-10941.003.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658441#comment-15658441
]
Hudson commented on HDFS-10941:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10825 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10825/])
HDFS-10941. Improve BlockManager#processMisReplicatesAsync log. (xyao: rev
4484b48498b2ab2a40a404c487c7a4e875df10dc)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10941.001.patch, HDFS-10941.002.patch,
> HDFS-10941.002.patch, HDFS-10941.003.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658193#comment-15658193
]
Xiaoyu Yao commented on HDFS-10941:
---
Thanks [~vagarychen] for the update. The latest patch LGTM. +1 and I will
commit it shortly.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-10941.001.patch, HDFS-10941.002.patch,
> HDFS-10941.002.patch, HDFS-10941.003.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657997#comment-15657997
]
Hadoop QA commented on HDFS-10941:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-10941 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838588/HDFS-10941.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0bb9a235a3bc 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 503e73e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17529/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17529/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17529/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
>
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657612#comment-15657612
]
Xiaoyu Yao commented on HDFS-10941:
---
Thanks [~vagarychen] for the update. The v3 patch has a potential perf issue
with the wrapper approach. The toString() and string concat cost will always be
there even with the {{if (LOG.isTraceEnabled())}} guard inside the wrapper.
I would suggest we leverage the slf4j parameterized logging like below to avoid
it without the wrapper.
More detail about sl4fj logging performance can be found here:
http://www.slf4j.org/faq.html#logging_performance.
{code}
case UNDER_REPLICATED:
LOG.trace("under replicated block: {} result: {}", block, res);
nrUnderReplicated++;
{code}
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-10941.001.patch, HDFS-10941.002.patch,
> HDFS-10941.002.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637538#comment-15637538
]
Chen Liang commented on HDFS-10941:
---
The failed tests seem unrelated. Local tests never had
{{TestEncryptionZones.testStartFileRetry}} failed. And the other three tests
randomly fail either with or without the patch so it appears the tests
themselves are flaky.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-10941.001.patch, HDFS-10941.002.patch,
> HDFS-10941.002.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634707#comment-15634707
]
Hadoop QA commented on HDFS-10941:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10941 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836985/HDFS-10941.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 258639f113f8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7534aee |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17414/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17414/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key:
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626451#comment-15626451
]
Xiaobing Zhou commented on HDFS-10941:
--
+1 pending on Jenkins. Thanks.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-10941.001.patch, HDFS-10941.002.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613043#comment-15613043
]
Hadoop QA commented on HDFS-10941:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.security.TestPermission |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10941 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835631/HDFS-10941.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e2f50c04aef8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ac35ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17331/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17331/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17331/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
>
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612862#comment-15612862
]
Xiaobing Zhou commented on HDFS-10941:
--
Thanks for the patch [~vagarychen], can you add a shared function to move the
logs into it? It will keep code clean.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-10941.001.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612731#comment-15612731
]
Xiaoyu Yao commented on HDFS-10941:
---
Thanks [~vagarychen] for working on this. Patch LGTM. +1 pending Jenkins.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Attachments: HDFS-10941.001.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
