[
https://issues.apache.org/jira/browse/AMBARI-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13723906#comment-13723906
]
Tom Beerbower commented on AMBARI-2762:
---------------------------------------
More Info...
A metric name is mapped to an internal key that is used to find the metric from
the backing source (JMX, Ganglia, ...). These mappings are maintained in
several JSON files. For example, the file jmx_properties.json contains an
entry like ...
{code}
"metrics/dfs/FSNamesystem/CapacityTotal":{
"metric" :
"Hadoop:service=NameNode,name=FSNamesystemMetrics.CapacityTotal",
"pointInTime" : true,
"temporal" : false
}
{code}
A request for the metric named 'metrics/dfs/FSNamesystem/CapacityTotal' will
result in a request for JMX data keyed by
'Hadoop:service=NameNode,name=FSNamesystemMetrics.CapacityTotal'.
This mapping of a static metric name to a static key does not work when we
allow things like queue names and hierarchies as part of the metric name. For
example, from JMX we can get YARN queue metrics like ...
{code}
{
"name" :
"Hadoop:service=ResourceManager,name=QueueMetrics,q0=root,q1=default",
"modelerType" : "QueueMetrics,q0=root,q1=default",
"tag.Queue" : "root.default",
"tag.Context" : "yarn",
"tag.Hostname" : "ip-10-147-212-220.ec2.internal",
"running_0" : 0,
"running_60" : 0,
"running_300" : 0,
"running_1440" : 0,
"AppsSubmitted" : 47,
"AppsRunning" : 0,
"AppsPending" : 0,
"AppsCompleted" : 2,
...
}
{code}
These metrics are for the queue named "default" under the parent "root", or
"/root/default".
In order to support this type of mapping we need to be able to allow template
metric names with argument substitution. Something like ...
{code}
metrics/yarn/Queue/$1/AppsRunning
{code}
Notice the argument '$1' which can be replaced with the queue name. To get the
queue name for the substitution we can allow for regular expression matches in
the key. For example ...
{code}
Hadoop:service=ResourceManager,name=QueueMetrics,(.+).AppsRunning
{code}
The JMX metric keyed by
'Hadoop:service=ResourceManager,name=QueueMetrics,q0=root,q1=default.AppsRunning'
would be a regular expression match with 'q0=root,q1=default' being the queue
name.
This works but makes for strange metric names like
'metrics/yarn/Queue/q0=root,q1=default/AppsRunning'. Ideally we would like to
process the queue name so that the metric name would look like
'metrics/yarn/Queue/root/default/AppsRunning'.
To accomplish this, we could add optional methods to the metric name arguments
to allow for some level of processing at substitution time. For now, we will
only support replaceAll() but we can add more if required in the future. Keep
in mind this is largely an internal feature as the typical user will not be
adding metrics through the default metric providers.
So, an entry in the jmx_properties_2.json file could look like ...
{code}
"metrics/yarn/Queue$1.replaceAll(\",q(\\d+)=\",\"/\")/AppsRunning":{
"metric" :
"Hadoop:service=ResourceManager,name=QueueMetrics(.+).AppsRunning",
"pointInTime" : true,
"temporal" : false
}
{code}
Notice the replaceAll() method attached to the $1 argument. The application of
this replaceAll should result in transforming the value of the argument from
'q0=root,q1=default' to 'root/default' in the above example.
Also note that this changes the existing behavior in that we have been dropping
the queue name for 'q0=root'. For example, the JMX metric keyed by ...
{code}
Hadoop:service=ResourceManager,name=QueueMetrics,q0=root
{code}
... currently maps to the metric named ...
{code}
metrics/yarn/Queue/AggregateContainersAllocated
{code}
With the above changes it would instead map to the metric named ...
{code}
metrics/yarn/Queue/root/AggregateContainersAllocated
{code}
> Additional YARN metrics required in service status call
> -------------------------------------------------------
>
> Key: AMBARI-2762
> URL: https://issues.apache.org/jira/browse/AMBARI-2762
> Project: Ambari
> Issue Type: Task
> Reporter: Tom Beerbower
> Assignee: Tom Beerbower
> Attachments: AMBARI-2762.patch
>
>
> When client calls
> /api/v1/clusters/vm/services?fields=components/host_components/metrics/yarn/Queue,
> we need following additional information:
> Queues defined (hierarchy also if possible)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
