[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317645#comment-17317645 ]
Wei-Chiu Chuang commented on HDFS-15887: ---------------------------------------- Not an expert here, but makes sense to me. > Make LogRoll and TailEdits execute in parallel > ---------------------------------------------- > > Key: HDFS-15887 > URL: https://issues.apache.org/jira/browse/HDFS-15887 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: JiangHua Zhu > Assignee: JiangHua Zhu > Priority: Major > Labels: pull-request-available > Attachments: edit_files.jpg > > Time Spent: 20m > Remaining Estimate: 0h > > In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, > and when a checkpoint occurs, it will compete with TailEdits for lock > (FSNamesystem#cpLock). > Usually, it takes a long time to execute checkpoint, which will cause the > size of the generated edit log file to be relatively large. > For example, here is an actual effect: > The StandbyCheckpointer log is triggered as follows : edit_files.jpg > 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State > Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering > checkpoint because there have been 5142154 txns since the last checkpoint, > which exceeds the configured threshold 1000000 > When loading an edit log with a large amount of data, the processing time > will be longer. We should make the edit log size as even as possible, which > is good for the operation of the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org