[jira] [Commented] (HDFS-9839) Reduce verbosity of processReport logging
[
https://issues.apache.org/jira/browse/HDFS-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155935#comment-15155935
]
Hudson commented on HDFS-9839:
--
FAILURE: Integrated in Hadoop-trunk-Commit #9335 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9335/])
HDFS-9839. Reduce verbosity of processReport logging. (Contributed by (arp: rev
d5abd293a890a8a1da48a166a291ae1c5644ad57)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Reduce verbosity of processReport logging
> -
>
> Key: HDFS-9839
> URL: https://issues.apache.org/jira/browse/HDFS-9839
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-9839.01.patch
>
>
> {{BlockManager#processReport}} logs one line for each invalidated block at
> INFO. HDFS-7503 moved this logging outside the NameSystem write lock but we
> still see the NameNode being slowed down when the number of block
> invalidations is very large e.g. just after a large amount of data is deleted.
> {code}
> for (Block b : invalidatedBlocks) {
> blockLog.info("BLOCK* processReport: {} on node {} size {} does not "
> +
> "belong to any file", b, node, b.getNumBytes());
> }
> {code}
> We can change this statement to DEBUG and just log the number of block
> invalidations at INFO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9839) Reduce verbosity of processReport logging
[
https://issues.apache.org/jira/browse/HDFS-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155925#comment-15155925
]
ASF GitHub Bot commented on HDFS-9839:
--
Github user asfgit closed the pull request at:
https://github.com/apache/hadoop/pull/78
> Reduce verbosity of processReport logging
> -
>
> Key: HDFS-9839
> URL: https://issues.apache.org/jira/browse/HDFS-9839
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-9839.01.patch
>
>
> {{BlockManager#processReport}} logs one line for each invalidated block at
> INFO. HDFS-7503 moved this logging outside the NameSystem write lock but we
> still see the NameNode being slowed down when the number of block
> invalidations is very large e.g. just after a large amount of data is deleted.
> {code}
> for (Block b : invalidatedBlocks) {
> blockLog.info("BLOCK* processReport: {} on node {} size {} does not "
> +
> "belong to any file", b, node, b.getNumBytes());
> }
> {code}
> We can change this statement to DEBUG and just log the number of block
> invalidations at INFO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9839) Reduce verbosity of processReport logging
[
https://issues.apache.org/jira/browse/HDFS-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155858#comment-15155858
]
Xiaoyu Yao commented on HDFS-9839:
--
Patch LGTM. +1.
> Reduce verbosity of processReport logging
> -
>
> Key: HDFS-9839
> URL: https://issues.apache.org/jira/browse/HDFS-9839
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-9839.01.patch
>
>
> {{BlockManager#processReport}} logs one line for each invalidated block at
> INFO. HDFS-7503 moved this logging outside the NameSystem write lock but we
> still see the NameNode being slowed down when the number of block
> invalidations is very large e.g. just after a large amount of data is deleted.
> {code}
> for (Block b : invalidatedBlocks) {
> blockLog.info("BLOCK* processReport: {} on node {} size {} does not "
> +
> "belong to any file", b, node, b.getNumBytes());
> }
> {code}
> We can change this statement to DEBUG and just log the number of block
> invalidations at INFO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9839) Reduce verbosity of processReport logging
[
https://issues.apache.org/jira/browse/HDFS-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155803#comment-15155803
]
Hadoop QA commented on HDFS-9839:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: patch generated 0
new + 151 unchanged - 1 fixed = 151 total (was 152) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 8s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 59s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 175m 21s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.tracing.TestTracing |
| | hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.mover.TestStorageMover |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.metrics2.sink.TestRollin
[jira] [Commented] (HDFS-9839) Reduce verbosity of processReport logging
[
https://issues.apache.org/jira/browse/HDFS-9839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155732#comment-15155732
]
ASF GitHub Bot commented on HDFS-9839:
--
GitHub user arp7 opened a pull request:
https://github.com/apache/hadoop/pull/78
HDFS-9839. Reduce verbosity of processReport logging
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arp7/hadoop HDFS-9839
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop/pull/78.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #78
commit 8b23b41ada23168fe2cb71f4a3b920c68e66ee74
Author: Arpit Agarwal
Date: 2016-02-20T18:43:14Z
HDFS-9839. Reduce verbosity of processReport logging
> Reduce verbosity of processReport logging
> -
>
> Key: HDFS-9839
> URL: https://issues.apache.org/jira/browse/HDFS-9839
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> {{BlockManager#processReport}} logs one line for each invalidated block at
> INFO. HDFS-7503 moved this logging outside the NameSystem write lock but we
> still see the NameNode being slowed down when the number of block
> invalidations is very large e.g. just after a large amount of data is deleted.
> {code}
> for (Block b : invalidatedBlocks) {
> blockLog.info("BLOCK* processReport: {} on node {} size {} does not "
> +
> "belong to any file", b, node, b.getNumBytes());
> }
> {code}
> We can change this statement to DEBUG and just log the number of block
> invalidations at INFO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
