[
https://issues.apache.org/jira/browse/HDFS-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649564#comment-17649564
]
ASF GitHub Bot commented on HDFS-16872:
---------------------------------------
ChengbingLiu opened a new pull request, #5246:
URL: https://github.com/apache/hadoop/pull/5246
### Description of PR
In our production cluster with Observer NameNode enabled, we have plenty of
logs printed by `FSEditLogLoader` and `RedundantEditLogInputStream`. The
`LogThrottlingHelper` doesn't seem to work.
```
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Start loading edits file ByteStringEditLog[17686250688, 17686250688],
ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688,
17686250688] maxTxnsToRead = 92233720368547758072022-10-25 09:26:50,380 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688],
ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688,
17686250688]' to transaction ID 17686250688
2022-10-25 09:26:50,380 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688]' to
transaction ID 17686250688
2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250688,
17686250688], ByteStringEditLog[17686250688, 17686250688],
ByteStringEditLog[17686250688, 17686250688]) of total size 527.0, total edits
1.0, total load time 0.0 ms
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Start loading edits file ByteStringEditLog[17686250689, 17686250693],
ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689,
17686250693] maxTxnsToRead = 9223372036854775807
2022-10-25 09:26:50,387 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693],
ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689,
17686250693]' to transaction ID 17686250689
2022-10-25 09:26:50,387 INFO
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693]' to
transaction ID 17686250689
2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250689,
17686250693], ByteStringEditLog[17686250689, 17686250693],
ByteStringEditLog[17686250689, 17686250693]) of total size 890.0, total edits
5.0, total load time 1.0 ms
```
After some digging, I found the cause is that `LogThrottlingHelper`'s are
declared as instance variables of all the enclosing classes, including
`FSImage`, `FSEditLogLoader` and `RedundantEditLogInputStream`. Therefore the
logging frequency will not be limited across different instances. For classes
with only limited number of instances, such as `FSImage`, this is fine. For
others whose instances are created frequently, such as `FSEditLogLoader` and
`RedundantEditLogInputStream`, it will result in plenty of logs.
This can be fixed by declaring `LogThrottlingHelper`'s as static members.
### How was this patch tested?
Through a test case.
> Fix log throttling by declaring LogThrottlingHelper as static members
> ---------------------------------------------------------------------
>
> Key: HDFS-16872
> URL: https://issues.apache.org/jira/browse/HDFS-16872
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.3.4
> Reporter: Chengbing Liu
> Priority: Major
>
> In our production cluster with Observer NameNode enabled, we have plenty of
> logs printed by {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}. The
> {{LogThrottlingHelper}} doesn't seem to work.
> {noformat}
> 2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
> Start loading edits file ByteStringEditLog[17686250688, 17686250688],
> ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688,
> 17686250688] maxTxnsToRead = 9223372036854775807
> 2022-10-25 09:26:50,380 INFO
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
> Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688],
> ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688,
> 17686250688]' to transaction ID 17686250688
> 2022-10-25 09:26:50,380 INFO
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
> Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688]' to
> transaction ID 17686250688
> 2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
> Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250688,
> 17686250688], ByteStringEditLog[17686250688, 17686250688],
> ByteStringEditLog[17686250688, 17686250688]) of total size 527.0, total edits
> 1.0, total load time 0.0 ms
> 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
> Start loading edits file ByteStringEditLog[17686250689, 17686250693],
> ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689,
> 17686250693] maxTxnsToRead = 9223372036854775807
> 2022-10-25 09:26:50,387 INFO
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
> Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693],
> ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689,
> 17686250693]' to transaction ID 17686250689
> 2022-10-25 09:26:50,387 INFO
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream:
> Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693]' to
> transaction ID 17686250689
> 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
> Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250689,
> 17686250693], ByteStringEditLog[17686250689, 17686250693],
> ByteStringEditLog[17686250689, 17686250693]) of total size 890.0, total edits
> 5.0, total load time 1.0 ms
> {noformat}
> After some digging, I found the cause is that {{LogThrottlingHelper}}'s are
> declared as instance variables of all the enclosing classes, including
> {{FSImage}}, {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}.
> Therefore the logging frequency will not be limited across different
> instances. For classes with only limited number of instances, such as
> {{FSImage}}, this is fine. For others whose instances are created frequently,
> such as {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}, it will
> result in plenty of logs.
> This can be fixed by declaring {{LogThrottlingHelper}}'s as static members.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
