[jira] [Created] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
Damien Claveau created HADOOP-14300: --- Summary: GraphiteSink reports metrics containing undesirable whitespaces Key: HADOOP-14300 URL: https://issues.apache.org/jira/browse/HADOOP-14300 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 2.7.3, 2.7.2, 2.7.1, 2.7.0, 2.6.0, 2.5.0 Reporter: Damien Claveau Priority: Trivial According to the Graphite documentation, the data sent must be in the following format: But some tag values that are flattened in the metricpath string sometimes include whitespaces and violate the carbon/graphite format. For example, the dirpath enumeration from the datanode : FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current, /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609 Currently, none of the aggregators available in the Graphite ecosystem can handle these malformed metrics. These whitespaces should be replaced by underscores. I will be happy to attach a small patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[ https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Claveau updated HADOOP-14300: Attachment: HADOOP-14300 Patch attached > GraphiteSink reports metrics containing undesirable whitespaces > --- > > Key: HADOOP-14300 > URL: https://issues.apache.org/jira/browse/HADOOP-14300 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3 >Reporter: Damien Claveau >Priority: Trivial > Attachments: HADOOP-14300 > > > According to the Graphite documentation, the data sent must be in the > following format: > But some tag values that are flattened in the metricpath string sometimes > include whitespaces and violate the carbon/graphite format. > For example, the dirpath enumeration from the datanode : > FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current, > /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609 > Currently, none of the aggregators available in the Graphite ecosystem can > handle these malformed metrics. These whitespaces should be replaced by > underscores. > I will be happy to attach a small patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[ https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967325#comment-15967325 ] Damien Claveau commented on HADOOP-14300: - A couple of things : 1. There is already a unit test class named org.apache.hadoop.metrics2.impl.TestGraphiteMetrics Can I enrich it with my test, or should I rename it TestGraphiteSink to conform to the naming rules, or should I add a new class TestGraphiteSink ? 2. I think I will have 2 more Jira to open to fully achieve the GraphiteSink. The first one is a feature addition to filter out tags from the metric prefix string. That would be useful to avoid the situation I have with the tag "StorageInfo". I will propose it like in https://issues.apache.org/jira/browse/HADOOP-7507 The second one will be a fix for the regression described here https://issues.apache.org/jira/browse/HADOOP-11400?focusedCommentId=14977496&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14977496 Any general advice before I start that work ? > GraphiteSink reports metrics containing undesirable whitespaces > --- > > Key: HADOOP-14300 > URL: https://issues.apache.org/jira/browse/HADOOP-14300 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3 >Reporter: Damien Claveau >Priority: Trivial > Attachments: HADOOP-14300 > > > According to the Graphite documentation, the data sent must be in the > following format: > But some tag values that are flattened in the metricpath string sometimes > include whitespaces and violate the carbon/graphite format. > For example, the dirpath enumeration from the datanode : > FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current, > /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609 > Currently, none of the aggregators available in the Graphite ecosystem can > handle these malformed metrics. These whitespaces should be replaced by > underscores. > I will be happy to attach a small patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[ https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damien Claveau updated HADOOP-14300: Attachment: HADOOP-14300.001.patch New patch attached with full test coverage (and minor refactoring in the unit test). test-patch execution fully succeeded locally. > GraphiteSink reports metrics containing undesirable whitespaces > --- > > Key: HADOOP-14300 > URL: https://issues.apache.org/jira/browse/HADOOP-14300 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3 >Reporter: Damien Claveau >Assignee: Damien Claveau >Priority: Trivial > Attachments: HADOOP-14300, HADOOP-14300.001.patch > > > According to the Graphite documentation, the data sent must be in the > following format: > But some tag values that are flattened in the metricpath string sometimes > include whitespaces and violate the carbon/graphite format. > For example, the dirpath enumeration from the datanode : > FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current, > /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609 > Currently, none of the aggregators available in the Graphite ecosystem can > handle these malformed metrics. These whitespaces should be replaced by > underscores. > I will be happy to attach a small patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14307) GraphiteSink can specify which tags to export into the metrics prefix
Damien Claveau created HADOOP-14307: --- Summary: GraphiteSink can specify which tags to export into the metrics prefix Key: HADOOP-14307 URL: https://issues.apache.org/jira/browse/HADOOP-14307 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 2.7.3, 2.7.2, 2.7.1, 2.7.0 Reporter: Damien Claveau Assignee: Damien Claveau Priority: Minor Fix For: 2.7.3, 2.7.2, 2.7.1, 2.7.0 This Jira is a feature proposal to add the ability in GraphiteSink to specify which Tag (Name/Value) pairs are to be flattened in the metric prefix. The motivation for this is that currently, all the tags are included in the prefix like this : graphite_prefix.Context=$context.Process=$process.Tag1=$value1.Tag2=$value2..Tag9=$value9.metric This requires a bunch of rewriting rules (and complex regexp) in the metric aggregation servers (carbon-relay, carbon-aggregator, ...) . The feature would be exactly the same as the solution implemented in the GangliaSink : https://issues.apache.org/jira/browse/HADOOP-7507 See also the commit : https://github.com/apache/hadoop/commit/2ca9c8d926a8eeb871b2868e6eb4dfb97d7dc63d -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[ https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968778#comment-15968778 ] Damien Claveau commented on HADOOP-14300: - Hi Wei-Chiu, thank you for the precommit check. I don't think the failed junit tests on hadoop.security.TestKDiag can be related to the patch. > GraphiteSink reports metrics containing undesirable whitespaces > --- > > Key: HADOOP-14300 > URL: https://issues.apache.org/jira/browse/HADOOP-14300 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3 >Reporter: Damien Claveau >Assignee: Damien Claveau >Priority: Trivial > Attachments: HADOOP-14300, HADOOP-14300.001.patch > > > According to the Graphite documentation, the data sent must be in the > following format: > But some tag values that are flattened in the metricpath string sometimes > include whitespaces and violate the carbon/graphite format. > For example, the dirpath enumeration from the datanode : > FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current, > /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609 > Currently, none of the aggregators available in the Graphite ecosystem can > handle these malformed metrics. These whitespaces should be replaced by > underscores. > I will be happy to attach a small patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14308) GraphiteSink gives up reconnecting to the graphite server
Damien Claveau created HADOOP-14308: --- Summary: GraphiteSink gives up reconnecting to the graphite server Key: HADOOP-14308 URL: https://issues.apache.org/jira/browse/HADOOP-14308 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.3, 2.7.2, 2.7.1, 2.7.0 Reporter: Damien Claveau Assignee: Damien Claveau Priority: Minor Since https://issues.apache.org/jira/browse/HADOOP-11400 the GraphiteSink now handles the reconnection, but it gives up silently after 5 failed attempts. As discussed here https://issues.apache.org/jira/browse/HADOOP-11400?focusedCommentId=14977496&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14977496, the sink should better throw an exception then rely on the MetricsSinkAdapter logic to retry with the configured retryDelay and retryBackoff parameters. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org