[ https://issues.apache.org/jira/browse/HDFS-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485649#comment-15485649 ]
Erik Krogen edited comment on HDFS-10475 at 9/12/16 11:47 PM: -------------------------------------------------------------- To get a mapping of operation -> lock time metrics we propose the following: 1. Move the logging/metrics logic into FSNamesystemLock rather than FSNamesystem to centralize logic and tracking. 2. Add new methods, {{(read|write)Unlock(operation)}}, in which you specify a name for the current operation as you unlock (note that for metrics collecting the name is only needed on unlock). If an operation is not specified, a catch-all 'default' or 'other' operation would be used. We would manually add the name of the operation to the unlock call for those operations which we think are likely to contribute significantly to the overall lock hold time. This is a manual process since otherwise we would need to get a stack trace (to find the method name) on each call to {{unlock}} which may be prohibitively expensive. 3. Add a map of OperationName -> MutableRate metrics to FSNamesystemLock, all of which are also contained within a MetricsRegistry. On each time a lock is released we look up the corresponding MutableRate and add a value for the lock hold time. We do not use the map within MetricsRegistry because it is synchronized and we do not want contention on this map to cause slowness around the FSNamesystem lock. The best type of map to use within FSNamesystemLock to hold the MutableRate metrics is tricky. Ideally we would use a Java 8 ConcurrentHashMap, using {{computeIfAbsent}} to create new MutableRate metrics objects and insert them into the registry whenever a new operation is encountered. However this functionality is not available in Java 7 and we would like to support older versions. Thus we propose using a regular HashMap (wrapped within a call to {{Collections.unmodifiableMap}}) which is initialized with all of the different operations at the time the FSNamesystemLock is created. This allows for lock-free access, but requires that we have a list of all the possible operations. So we suggest an Enum, e.g. FSNamesystemLockMetricOp, which lists all of the operations of interest to be supplied to the {{(read|write)Unlock}} calls. This would likely be a list of a few dozen operations of interest which are likely to be relatively expensive lock holders. Operations not listed within this Enum would be regarded as "other"/"default". We believe this is the right tradeoff between granularity of metrics, performance, and developer effort, but it is certainly not ideal in terms of manual effort required. We would be interested to hear any other ideas about how to make the metrics collection require less manual intervention. was (Author: xkrogen): To get a mapping of operation -> lock time metrics we propose the following: 1. Move the logging/metrics logic into FSNamesystemLock rather than FSNamesystem to centralize logic and tracking. 2. Add new methods, {{(read|write)Unlock(operation)}}, in which you specify a name for the current operation as you unlock (note that for metrics collecting the name is only needed on unlock). If an operation is not specified, a catch-all 'default' or 'other' operation would be used. We would manually add the name of the operation to the unlock call for those operations which we think are likely to contribute significantly to the overall lock hold time. This is a manual process since otherwise we would need to get a stack trace (to find the method name) on each call to {{unlock}} which may be prohibitively expensive. 3. FSNamesystemLock contains a map of OperationName -> MutableRate metrics, all of which are also contained within a MetricsRegistry. On each time a lock is released we look up the corresponding MutableRate and add a value for the lock hold time. We do not use the map within MetricsRegistry because it is synchronized and we do not want contention on this map to cause slowness around the FSNamesystem lock. The best type of map to use within FSNamesystemLock to hold the MutableRate metrics is tricky. Ideally we would use a Java 8 ConcurrentHashMap, using {{computeIfAbsent}} to create new MutableRate metrics objects and insert them into the registry whenever a new operation is encountered. However this functionality is not available in Java 7 and we would like to support older versions. Thus we propose using a regular HashMap (wrapped within a call to {{Collections.unmodifiableMap}}) which is initialized with all of the different operations at the time the FSNamesystemLock is created. This allows for lock-free access, but requires that we have a list of all the possible operations. So we suggest an Enum, e.g. FSNamesystemLockMetricOp, which lists all of the operations of interest to be supplied to the {{(read|write)Unlock}} calls. This would likely be a list of a few dozen operations of interest which are likely to be relatively expensive lock holders. Operations not listed within this Enum would be regarded as "other"/"default". We believe this is the right tradeoff between granularity of metrics, performance, and developer effort, but it is certainly not ideal in terms of manual effort required. We would be interested to hear any other ideas about how to make the metrics collection require less manual intervention. > Adding metrics for long FSNamesystem read and write locks > --------------------------------------------------------- > > Key: HDFS-10475 > URL: https://issues.apache.org/jira/browse/HDFS-10475 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Xiaoyu Yao > Assignee: Erik Krogen > > This is a follow up of the comment on HADOOP-12916 and > [here|https://issues.apache.org/jira/browse/HDFS-9924?focusedCommentId=15310837&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15310837] > add more metrics and WARN/DEBUG logs for long FSD/FSN locking operations on > namenode similar to what we have for slow write/network WARN/metrics on > datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org