[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[
https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041372#comment-16041372
]
Erik Krogen commented on HADOOP-14492:
--
[~cltlfcjin] I actually think that it is the {{NameNodeActivity}} metric which
is named in a misleading way. The RpcDetailedMetric is named after the RPC
call, which is in fact {{BlockReport}}. {{NameNodeActivity}} already has the
metric {{StorageBlockReportOps}} which has a more clear naming (though it is a
duplicate of {{BlockReportNumOps}}). It seems that we should deprecate
{{NameNodeActivity#BlockReportNumOps}} and change {{StorageBlockReportOps}} to
be a {{MutableRate}} to get both the count and the time, but this would be an
incompatible change. Perhaps with the 3.0 release would be a good time. I filed
a follow-on JIRA to discuss: HADOOP-14502. I will close this as JIRA as the
original purpose (addressing {{MutableRate}} vs
{{MutableRatesWithAggregation}}) is not an issue. Feel free to reopen if you
disagree.
> RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction
> cause the Xavgtime confused
> -
>
> Key: HADOOP-14492
> URL: https://issues.apache.org/jira/browse/HADOOP-14492
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Lantao Jin
>Assignee: Erik Krogen
>Priority: Minor
>
> For performance purpose,
> [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the
> metrics behaviour in {{RpcDetailedMetrics}}.
> In 2.7.4:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRatesWithAggregation rates;
> {code}
> In old version:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRates rates;
> {code}
> But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old
> version:
> {code}
> public class NameNodeMetrics {
> @Metric("Block report") MutableRate blockReport;
> {code}
> It causes the metrics in JMX very different between them.
> {quote}
> name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030",
> modelerType: "RpcDetailedActivityForPort8030",
> tag.port: "8030",
> tag.Context: "rpcdetailed",
> ...
> BlockReportNumOps: 237634,
> BlockReportAvgTime: 1382,
> ...
> name: "Hadoop:service=NameNode,name=NameNodeActivity",
> modelerType: "NameNodeActivity",
> tag.ProcessName: "NameNode",
> ...
> BlockReportNumOps: 2592932,
> BlockReportAvgTime: 19.258064516129032,
> ...
> {quote}
> In the old version. They are correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[
https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040978#comment-16040978
]
Lantao Jin commented on HADOOP-14492:
-
Thanks [~xkrogen] to explain. Yes, our DataNode run with multiple storages.
Assigning to you is good. I think it better to use a different metrics name
like {{WholeDNBlockReportAvgTime}}.
> RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction
> cause the Xavgtime confused
> -
>
> Key: HADOOP-14492
> URL: https://issues.apache.org/jira/browse/HADOOP-14492
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Lantao Jin
>Assignee: Erik Krogen
>Priority: Minor
>
> For performance purpose,
> [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the
> metrics behaviour in {{RpcDetailedMetrics}}.
> In 2.7.4:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRatesWithAggregation rates;
> {code}
> In old version:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRates rates;
> {code}
> But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old
> version:
> {code}
> public class NameNodeMetrics {
> @Metric("Block report") MutableRate blockReport;
> {code}
> It causes the metrics in JMX very different between them.
> {quote}
> name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030",
> modelerType: "RpcDetailedActivityForPort8030",
> tag.port: "8030",
> tag.Context: "rpcdetailed",
> ...
> BlockReportNumOps: 237634,
> BlockReportAvgTime: 1382,
> ...
> name: "Hadoop:service=NameNode,name=NameNodeActivity",
> modelerType: "NameNodeActivity",
> tag.ProcessName: "NameNode",
> ...
> BlockReportNumOps: 2592932,
> BlockReportAvgTime: 19.258064516129032,
> ...
> {quote}
> In the old version. They are correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[
https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039792#comment-16039792
]
Erik Krogen commented on HADOOP-14492:
--
Hey [~cltlfcjin] - these metrics can be different since the RpcDetailed metric
is the time to complete a DN's entire block report, which may include multiple
storages (more than one value for {{dfs.datanode.data.dir}}), each of which
contributes to the NameNodeActivity metric. In the cluster this was recorded
from, do your datanodes run with multiple storages?
> RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction
> cause the Xavgtime confused
> -
>
> Key: HADOOP-14492
> URL: https://issues.apache.org/jira/browse/HADOOP-14492
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Lantao Jin
>Priority: Minor
>
> For performance purpose,
> [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the
> metrics behaviour in {{RpcDetailedMetrics}}.
> In 2.7.4:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRatesWithAggregation rates;
> {code}
> In old version:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRates rates;
> {code}
> But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old
> version:
> {code}
> public class NameNodeMetrics {
> @Metric("Block report") MutableRate blockReport;
> {code}
> It causes the metrics in JMX very different between them.
> {quote}
> name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030",
> modelerType: "RpcDetailedActivityForPort8030",
> tag.port: "8030",
> tag.Context: "rpcdetailed",
> ...
> BlockReportNumOps: 237634,
> BlockReportAvgTime: 1382,
> ...
> name: "Hadoop:service=NameNode,name=NameNodeActivity",
> modelerType: "NameNodeActivity",
> tag.ProcessName: "NameNode",
> ...
> BlockReportNumOps: 2592932,
> BlockReportAvgTime: 19.258064516129032,
> ...
> {quote}
> In the old version. They are correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14492) RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction cause the Xavgtime confused
[
https://issues.apache.org/jira/browse/HADOOP-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16038202#comment-16038202
]
Lantao Jin commented on HADOOP-14492:
-
How about changing the {{NameNodeMetrics}} to use
{{MutableRatesWithAggregation}} instead of {{MutableRates}}?
Any idea? [~steve_l]
> RpcDetailedMetrics and NameNodeMetrics use different rate metrics abstraction
> cause the Xavgtime confused
> -
>
> Key: HADOOP-14492
> URL: https://issues.apache.org/jira/browse/HADOOP-14492
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Lantao Jin
>Priority: Minor
>
> For performance purpose,
> [HADOOP-13782|https://issues.apache.org/jira/browse/HADOOP-13782] change the
> metrics behaviour in {{RpcDetailedMetrics}}.
> In 2.7.4:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRatesWithAggregation rates;
> {code}
> In old version:
> {code}
> public class RpcDetailedMetrics {
> @Metric MutableRates rates;
> {code}
> But {{NameNodeMetrics}} still use {{MutableRate}} whatever in the new or old
> version:
> {code}
> public class NameNodeMetrics {
> @Metric("Block report") MutableRate blockReport;
> {code}
> It causes the metrics in JMX very different between them.
> {quote}
> name: "Hadoop:service=NameNode,name=RpcDetailedActivityForPort8030",
> modelerType: "RpcDetailedActivityForPort8030",
> tag.port: "8030",
> tag.Context: "rpcdetailed",
> ...
> BlockReportNumOps: 237634,
> BlockReportAvgTime: 1382,
> ...
> name: "Hadoop:service=NameNode,name=NameNodeActivity",
> modelerType: "NameNodeActivity",
> tag.ProcessName: "NameNode",
> ...
> BlockReportNumOps: 2592932,
> BlockReportAvgTime: 19.258064516129032,
> ...
> {quote}
> In the old version. They are correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
