[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[ https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041372#comment-16041372 ] Erik Krogen commented on HADOOP-14492: -- [~cltlfcjin] I actually think that it is the {{NameNodeActivity}} metric which is named in a misleading way. The RpcDetailedMetric is named after the RPC call, which is in fact {{BlockReport}}. {{NameNodeActivity}} already has the metric {{StorageBlockReportOps}} which has a more clear naming (though it is a duplicate of {{BlockReportNumOps}}). It seems that we should deprecate {{NameNodeActivity#BlockReportNumOps}} and change {{StorageBlockReportOps}} to be a {{MutableRate}} to get both the count and the time, but this would be an incompatible change. Perhaps with the 3.0 release would be a good time. I filed a follow-on JIRA to discuss: HADOOP-14502. I will close this as JIRA as the original purpose (addressing {{MutableRate}} vs {{MutableRatesWithAggregation}}) is not an issue. Feel free to reopen if you disagree. > RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction > cause the Xavgtime confused > - > > Key: HADOOP-14492 > URL: https://issues.apache.org/jira/browse/HADOOP-14492 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.8.0, 2.7.4 >Reporter: Lantao Jin >Assignee: Erik Krogen >Priority: Minor > > For performance purpose, > [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the > metrics behaviour in {{RpcDetailedMetrics}}. > In 2.7.4: > {code} > public class RpcDetailedMetrics { > @Metric MutableRatesWithAggregation rates; > {code} > In old version: > {code} > public class RpcDetailedMetrics { > @Metric MutableRates rates; > {code} > But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old > version: > {code} > public class NameNodeMetrics { > @Metric("Block report") MutableRate blockReport; > {code} > It causes the metrics in JMX very different between them. > {quote} > name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030", > modelerType: "RpcDetailedActivityForPort8030", > tag.port: "8030", > tag.Context: "rpcdetailed", > ... > BlockReportNumOps: 237634, > BlockReportAvgTime: 1382, > ... > name: "Hadoop:service=NameNode,name=NameNodeActivity", > modelerType: "NameNodeActivity", > tag.ProcessName: "NameNode", > ... > BlockReportNumOps: 2592932, > BlockReportAvgTime: 19.258064516129032, > ... > {quote} > In the old version. They are correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[ https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040978#comment-16040978 ] Lantao Jin commented on HADOOP-14492: - Thanks [~xkrogen] to explain. Yes, our DataNode run with multiple storages. Assigning to you is good. I think it better to use a different metrics name like {{WholeDNBlockReportAvgTime}}. > RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction > cause the Xavgtime confused > - > > Key: HADOOP-14492 > URL: https://issues.apache.org/jira/browse/HADOOP-14492 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.8.0, 2.7.4 >Reporter: Lantao Jin >Assignee: Erik Krogen >Priority: Minor > > For performance purpose, > [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the > metrics behaviour in {{RpcDetailedMetrics}}. > In 2.7.4: > {code} > public class RpcDetailedMetrics { > @Metric MutableRatesWithAggregation rates; > {code} > In old version: > {code} > public class RpcDetailedMetrics { > @Metric MutableRates rates; > {code} > But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old > version: > {code} > public class NameNodeMetrics { > @Metric("Block report") MutableRate blockReport; > {code} > It causes the metrics in JMX very different between them. > {quote} > name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030", > modelerType: "RpcDetailedActivityForPort8030", > tag.port: "8030", > tag.Context: "rpcdetailed", > ... > BlockReportNumOps: 237634, > BlockReportAvgTime: 1382, > ... > name: "Hadoop:service=NameNode,name=NameNodeActivity", > modelerType: "NameNodeActivity", > tag.ProcessName: "NameNode", > ... > BlockReportNumOps: 2592932, > BlockReportAvgTime: 19.258064516129032, > ... > {quote} > In the old version. They are correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[ https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039792#comment-16039792 ] Erik Krogen commented on HADOOP-14492: -- Hey [~cltlfcjin] - these metrics can be different since the RpcDetailed metric is the time to complete a DN's entire block report, which may include multiple storages (more than one value for {{dfs.datanode.data.dir}}), each of which contributes to the NameNodeActivity metric. In the cluster this was recorded from, do your datanodes run with multiple storages? > RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction > cause the Xavgtime confused > - > > Key: HADOOP-14492 > URL: https://issues.apache.org/jira/browse/HADOOP-14492 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.8.0, 2.7.4 >Reporter: Lantao Jin >Priority: Minor > > For performance purpose, > [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the > metrics behaviour in {{RpcDetailedMetrics}}. > In 2.7.4: > {code} > public class RpcDetailedMetrics { > @Metric MutableRatesWithAggregation rates; > {code} > In old version: > {code} > public class RpcDetailedMetrics { > @Metric MutableRates rates; > {code} > But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old > version: > {code} > public class NameNodeMetrics { > @Metric("Block report") MutableRate blockReport; > {code} > It causes the metrics in JMX very different between them. > {quote} > name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030", > modelerType: "RpcDetailedActivityForPort8030", > tag.port: "8030", > tag.Context: "rpcdetailed", > ... > BlockReportNumOps: 237634, > BlockReportAvgTime: 1382, > ... > name: "Hadoop:service=NameNode,name=NameNodeActivity", > modelerType: "NameNodeActivity", > tag.ProcessName: "NameNode", > ... > BlockReportNumOps: 2592932, > BlockReportAvgTime: 19.258064516129032, > ... > {quote} > In the old version. They are correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[ https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038202#comment-16038202 ] Lantao Jin commented on HADOOP-14492: - How about changing the {{NameNodeMetrics}} to use {{MutableRatesWithAggregation}} instead of {{MutableRates}}? Any idea? [~steve_l] > RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction > cause the Xavgtime confused > - > > Key: HADOOP-14492 > URL: https://issues.apache.org/jira/browse/HADOOP-14492 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.8.0, 2.7.4 >Reporter: Lantao Jin >Priority: Minor > > For performance purpose, > [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the > metrics behaviour in {{RpcDetailedMetrics}}. > In 2.7.4: > {code} > public class RpcDetailedMetrics { > @Metric MutableRatesWithAggregation rates; > {code} > In old version: > {code} > public class RpcDetailedMetrics { > @Metric MutableRates rates; > {code} > But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old > version: > {code} > public class NameNodeMetrics { > @Metric("Block report") MutableRate blockReport; > {code} > It causes the metrics in JMX very different between them. > {quote} > name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030", > modelerType: "RpcDetailedActivityForPort8030", > tag.port: "8030", > tag.Context: "rpcdetailed", > ... > BlockReportNumOps: 237634, > BlockReportAvgTime: 1382, > ... > name: "Hadoop:service=NameNode,name=NameNodeActivity", > modelerType: "NameNodeActivity", > tag.ProcessName: "NameNode", > ... > BlockReportNumOps: 2592932, > BlockReportAvgTime: 19.258064516129032, > ... > {quote} > In the old version. They are correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org