[
https://issues.apache.org/jira/browse/HADOOP-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12497667
]
David Bowen commented on HADOOP-1406:
-------------------------------------
As part of fixing this, I would like to propose two small API changes in
org.apache.hadoop.metrics:
[1] Currently, to remove metrics records from the metrics library's internal
table you have to iterate over them one by one, since removal requires the
record name and all of its tags (name/value pairs). I propose to generalize
this so that you can specify the name and a subset of the tags - saving the
client code the burden of having to remember all the tag/values that it has
used. This doesn't change any API signatures, but it changes the meaning of
MetricsRecord.remove().
[2] Also, as a convenience, I propose to add a method
MetricsRecord.removeTag(String tagName). This method would be added to the
MetricsRecordImpl class in the spi package also. This enables the use pattern:
On module initialization:
create a metrics record named foo
set long-lived tag (e.g. job id)
Then repeatedly:
set short-lived tags (e.g. counter name)
set metric values
call MetricsRecord.update
And finally, on cleanup:
remove the short-lived tags
call MetricsRecord.remove to clean up all the metric data created
Both of these should be binary compatible changes for people using the public
package o.a.h.metrics.
> Metrics based on Map-Reduce Counters are not cleaned up
> -------------------------------------------------------
>
> Key: HADOOP-1406
> URL: https://issues.apache.org/jira/browse/HADOOP-1406
> Project: Hadoop
> Issue Type: Bug
> Reporter: David Bowen
> Assigned To: David Bowen
>
> When map-reduce jobs are finished, the metrics corresponding to their
> counters are not cleaned up. This is a memory leak, but worse it means that
> an ever-increasing amount of metric data is sent to the metrics system (if
> one is enabled).
> The fix is for JobInProgress to clean up the metrics it created when the job
> is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
