[jira] [Updated] (HDFS-4080) Add an option to disable block-level state change logging
[ https://issues.apache.org/jira/browse/HDFS-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4080: - Attachment: hdfs-4080-trunk.patch hdfs-4080-branch-0.23.patch Add an option to disable block-level state change logging - Key: HDFS-4080 URL: https://issues.apache.org/jira/browse/HDFS-4080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: hdfs-4080.1.patch, hdfs-4080-branch-0.23.patch, hdfs-4080.patch, hdfs-4080.patch, hdfs-4080-trunk.patch Although the block-level logging in namenode is useful for debugging, it can add a significant overhead to busy hdfs clusters since they are done while the namespace write lock is held. One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes while logging 11 million log messages for 5.5 million block invalidation events. It will be useful if we have an option to disable these block-level log messages and keep other state change messages going. If others feel that they can turned into DEBUG (with addition of isDebugEnabled() checks), that may also work too, but there might be people depending on the messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4080) Add an option to disable block-level state change logging
[ https://issues.apache.org/jira/browse/HDFS-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4080: - Attachment: hdfs-4080.patch The new patch addresses the review comment. The scope of the change is limited to the block state changes= messages from BlockManager friends and the calls through DatanodeProtocol. Add an option to disable block-level state change logging - Key: HDFS-4080 URL: https://issues.apache.org/jira/browse/HDFS-4080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: hdfs-4080.1.patch, hdfs-4080.patch, hdfs-4080.patch Although the block-level logging in namenode is useful for debugging, it can add a significant overhead to busy hdfs clusters since they are done while the namespace write lock is held. One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes while logging 11 million log messages for 5.5 million block invalidation events. It will be useful if we have an option to disable these block-level log messages and keep other state change messages going. If others feel that they can turned into DEBUG (with addition of isDebugEnabled() checks), that may also work too, but there might be people depending on the messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4080) Add an option to disable block-level state change logging
[ https://issues.apache.org/jira/browse/HDFS-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4080: - Attachment: hdfs-4080.patch Add an option to disable block-level state change logging - Key: HDFS-4080 URL: https://issues.apache.org/jira/browse/HDFS-4080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: hdfs-4080.patch Although the block-level logging in namenode is useful for debugging, it can add a significant overhead to busy hdfs clusters since they are done while the namespace write lock is held. One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes while logging 11 million log messages for 5.5 million block invalidation events. It will be useful if we have an option to disable these block-level log messages and keep other state change messages going. If others feel that they can turned into DEBUG (with addition of isDebugEnabled() checks), that may also work too, but there might be people depending on the messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4080) Add an option to disable block-level state change logging
[ https://issues.apache.org/jira/browse/HDFS-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4080: - Status: Patch Available (was: Open) Add an option to disable block-level state change logging - Key: HDFS-4080 URL: https://issues.apache.org/jira/browse/HDFS-4080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: hdfs-4080.patch Although the block-level logging in namenode is useful for debugging, it can add a significant overhead to busy hdfs clusters since they are done while the namespace write lock is held. One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes while logging 11 million log messages for 5.5 million block invalidation events. It will be useful if we have an option to disable these block-level log messages and keep other state change messages going. If others feel that they can turned into DEBUG (with addition of isDebugEnabled() checks), that may also work too, but there might be people depending on the messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4080) Add an option to disable block-level state change logging
[ https://issues.apache.org/jira/browse/HDFS-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4080: - Attachment: hdfs-4080.1.patch Updated the patch to the latest trunk. Add an option to disable block-level state change logging - Key: HDFS-4080 URL: https://issues.apache.org/jira/browse/HDFS-4080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: hdfs-4080.1.patch, hdfs-4080.patch Although the block-level logging in namenode is useful for debugging, it can add a significant overhead to busy hdfs clusters since they are done while the namespace write lock is held. One example is shown in HDFS-4075. In this example, the write lock was held for 5 minutes while logging 11 million log messages for 5.5 million block invalidation events. It will be useful if we have an option to disable these block-level log messages and keep other state change messages going. If others feel that they can turned into DEBUG (with addition of isDebugEnabled() checks), that may also work too, but there might be people depending on the messages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
