[ https://issues.apache.org/jira/browse/HDFS-16872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17650099#comment-17650099 ]
ASF GitHub Bot commented on HDFS-16872: --------------------------------------- ChengbingLiu commented on code in PR #5246: URL: https://github.com/apache/hadoop/pull/5246#discussion_r1054026139 ########## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogThrottlingHelper.java: ########## @@ -259,6 +259,10 @@ public LogAction record(String recorderName, long currentTimeMs, currentLogs.put(recorderName, currentLog); } currentLog.recordValues(values); + if (currentTimeMs < lastLogTimestampMs) { + // Reset lastLogTimestampMs: this should only happen in tests + lastLogTimestampMs = Long.MIN_VALUE; + } Review Comment: I thought about this solution, but the problem is that we have to add `reset()` methods in `LogThrottlingHelper` as well as `FSEditLogLoader`. Will this be OK? > Fix log throttling by declaring LogThrottlingHelper as static members > --------------------------------------------------------------------- > > Key: HDFS-16872 > URL: https://issues.apache.org/jira/browse/HDFS-16872 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 3.3.4 > Reporter: Chengbing Liu > Priority: Major > Labels: pull-request-available > > In our production cluster with Observer NameNode enabled, we have plenty of > logs printed by {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}. The > {{LogThrottlingHelper}} doesn't seem to work. > {noformat} > 2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: > Start loading edits file ByteStringEditLog[17686250688, 17686250688], > ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, > 17686250688] maxTxnsToRead = 9223372036854775807 > 2022-10-25 09:26:50,380 INFO > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: > Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688], > ByteStringEditLog[17686250688, 17686250688], ByteStringEditLog[17686250688, > 17686250688]' to transaction ID 17686250688 > 2022-10-25 09:26:50,380 INFO > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: > Fast-forwarding stream 'ByteStringEditLog[17686250688, 17686250688]' to > transaction ID 17686250688 > 2022-10-25 09:26:50,380 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: > Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250688, > 17686250688], ByteStringEditLog[17686250688, 17686250688], > ByteStringEditLog[17686250688, 17686250688]) of total size 527.0, total edits > 1.0, total load time 0.0 ms > 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: > Start loading edits file ByteStringEditLog[17686250689, 17686250693], > ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, > 17686250693] maxTxnsToRead = 9223372036854775807 > 2022-10-25 09:26:50,387 INFO > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: > Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693], > ByteStringEditLog[17686250689, 17686250693], ByteStringEditLog[17686250689, > 17686250693]' to transaction ID 17686250689 > 2022-10-25 09:26:50,387 INFO > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream: > Fast-forwarding stream 'ByteStringEditLog[17686250689, 17686250693]' to > transaction ID 17686250689 > 2022-10-25 09:26:50,387 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: > Loaded 1 edits file(s) (the last named ByteStringEditLog[17686250689, > 17686250693], ByteStringEditLog[17686250689, 17686250693], > ByteStringEditLog[17686250689, 17686250693]) of total size 890.0, total edits > 5.0, total load time 1.0 ms > {noformat} > After some digging, I found the cause is that {{LogThrottlingHelper}}'s are > declared as instance variables of all the enclosing classes, including > {{FSImage}}, {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}. > Therefore the logging frequency will not be limited across different > instances. For classes with only limited number of instances, such as > {{FSImage}}, this is fine. For others whose instances are created frequently, > such as {{FSEditLogLoader}} and {{RedundantEditLogInputStream}}, it will > result in plenty of logs. > This can be fixed by declaring {{LogThrottlingHelper}}'s as static members. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org