[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688036#comment-16688036
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r233837326
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+import org.influxdb.dto.BatchPoints;
+
+import javax.annotation.Nullable;
+
+import java.time.Instant;
+import java.util.ConcurrentModificationException;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.DB;
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.HOST;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.PASSWORD;
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.PORT;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.USERNAME;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.getInteger;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.getString;
+
+/**
+ * {@link MetricReporter} that exports {@link Metric Metrics} via InfluxDB.
+ */
+@Experimental
+public class InfluxdbReporter extends AbstractReporter
implements Scheduled {
+
+ private String database;
+ private InfluxDB influxDB;
+
+ public InfluxdbReporter() {
+ super(new MeasurementInfoProvider());
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ String host = getString(config, HOST);
+ int port = getInteger(config, PORT);
+ if (host == null || host.isEmpty() || port < 1) {
Review comment:
We can update other reporters in a follow-up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688042#comment-16688042
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r233838430
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
+ when(counter.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, counter),
+ "count=42");
+ }
+
+ @Test
+ public void testMapHistogram() {
+ HistogramStatistics statistics =
mock(HistogramStatistics.class);
Review comment:
> Only methods required for a test are mocked.
So if I change the reporter implementation to access any other method it
will now suddenly fail with an NPE since the method was never implemented.
Which actually never happens at runtime.
Test should be independent of the implementation and continue to work if
things change; otherwise they provide little value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688041#comment-16688041
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r233837808
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MeasurementInfoProviderTest.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerJobMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskMetricGroup;
+import org.apache.flink.util.AbstractID;
+
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ExecutionException;
+
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyChar;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MeasurementInfoProvider}.
+ */
+public class MeasurementInfoProviderTest {
+ private static final Random RANDOM = new Random();
+
+ private final MeasurementInfoProvider provider = new
MeasurementInfoProvider();
+
+ @Test
+ public void testGetMetricInfo() {
Review comment:
There should be no difference.
If you really want to test how it performs at runtime then please actually
start a cluster and optionally run a job; right now you're duplicating what
exists at runtime which may change in the future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686267#comment-16686267
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r233375997
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+import org.influxdb.dto.BatchPoints;
+
+import javax.annotation.Nullable;
+
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.ConcurrentModificationException;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+/**
+ * {@link MetricReporter} that exports {@link Metric Metrics} via InfluxDB.
+ */
+@Experimental
+public class InfluxdbReporter extends AbstractReporter
implements Scheduled {
+
+ private static final String ARG_PROTOCOL = "protocol";
+ private static final List SUPPORTED_PROTOCOLS =
Arrays.asList("http", "https");
+ private static final String ARG_HOST = "host";
Review comment:
dang, I was worried that might happen. Let me think about if for a bit...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686231#comment-16686231
]
ASF GitHub Bot commented on FLINK-7155:
---
1u0 commented on a change in pull request #6976: [FLINK-7155][metrics] Add new
metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r233364405
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+import org.influxdb.dto.BatchPoints;
+
+import javax.annotation.Nullable;
+
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.ConcurrentModificationException;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+/**
+ * {@link MetricReporter} that exports {@link Metric Metrics} via InfluxDB.
+ */
+@Experimental
+public class InfluxdbReporter extends AbstractReporter
implements Scheduled {
+
+ private static final String ARG_PROTOCOL = "protocol";
+ private static final List SUPPORTED_PROTOCOLS =
Arrays.asList("http", "https");
+ private static final String ARG_HOST = "host";
Review comment:
There is a test failure in
`ConfigOptionsDocsCompletenessITCase.testFullReferenceCompleteness`:
```
Documentation contains distinct descriptions for host in
prometheus_push_gateway_reporter_configuration.html and
influxdb_reporter_configuration.html.
```
Sounds like `flink-docs` now support only unique config keys.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684952#comment-16684952
]
ASF GitHub Bot commented on FLINK-7155:
---
1u0 commented on a change in pull request #6976: [FLINK-7155][metrics] Add new
metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232961776
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+import org.influxdb.dto.BatchPoints;
+
+import javax.annotation.Nullable;
+
+import java.time.Instant;
+import java.util.ConcurrentModificationException;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.DB;
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.HOST;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.PASSWORD;
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.PORT;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.USERNAME;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.getInteger;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.getString;
+
+/**
+ * {@link MetricReporter} that exports {@link Metric Metrics} via InfluxDB.
+ */
+@Experimental
+public class InfluxdbReporter extends AbstractReporter
implements Scheduled {
+
+ private String database;
+ private InfluxDB influxDB;
+
+ public InfluxdbReporter() {
+ super(new MeasurementInfoProvider());
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ String host = getString(config, HOST);
+ int port = getInteger(config, PORT);
+ if (host == null || host.isEmpty() || port < 1) {
Review comment:
Other reporters are not checking port's upper bound. To be more consistent,
should it be addressed in a separate PR for all reporters?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684949#comment-16684949
]
ASF GitHub Bot commented on FLINK-7155:
---
1u0 commented on a change in pull request #6976: [FLINK-7155][metrics] Add new
metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232960478
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
+ when(counter.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, counter),
+ "count=42");
+ }
+
+ @Test
+ public void testMapHistogram() {
+ HistogramStatistics statistics =
mock(HistogramStatistics.class);
Review comment:
`Mockito` is already used in many places of Flink codebase.
Only methods required for a test are mocked. With inheritance, all abstract
methods need to be implemented (even if they are not related to a test), which
is more verbose in Java and makes some noise.
> Actually implementing the interface is more maintainable as methods added
to the interface have to be taken care of in every test, instead of flying
under the radar breaking somehow, sometime, somewhere down the line.
It's not obvious for me, how using an interface in a test is more
maintainable.
Mock also "implements" the interface. If the interface changes, the mock may
also need to be updated.
If something changes, but the test passes and allows to sneak a bug. Then
it's question of test improvement, rather if some details use mocks or real,
imo.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels:
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681556#comment-16681556
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232280371
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MetricMapper.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+
+import java.time.Instant;
+import java.util.concurrent.TimeUnit;
+
+class MetricMapper {
+
+ static Point map(MeasurementInfo info, Instant timestamp, Gauge
gauge) {
+ Point.Builder builder = builder(info, timestamp);
+ Object value = gauge.getValue();
+ if (value instanceof Number) {
+ builder.addField("value", (Number) value);
+ } else {
+ builder.addField("value", String.valueOf(value));
+ }
+ return builder.build();
+ }
+
+ static Point map(MeasurementInfo info, Instant timestamp, Counter
counter) {
+ return builder(info, timestamp)
+ .addField("count", counter.getCount())
+ .build();
+ }
+
+ static Point map(MeasurementInfo info, Instant timestamp, Histogram
histogram) {
+ HistogramStatistics statistics = histogram.getStatistics();
+ return builder(info, timestamp)
+ .addField("run-count", histogram.getCount())
Review comment:
unless this naming scheme is endorsed/encouraged by influx-db themselves I'd
like to make Flink reporters more consistent so it's easier for users to
migrate from one reporter to another.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681550#comment-16681550
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232279530
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
+ when(counter.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, counter),
+ "count=42");
+ }
+
+ @Test
+ public void testMapHistogram() {
+ HistogramStatistics statistics =
mock(HistogramStatistics.class);
Review comment:
there's no reason to rely on another library when you can just implement the
interface. you mock every method anyway.
Actually implementing the interface is more maintainable as methods added to
the interface have to be taken care of in every test, instead of flying under
the radar breaking somehow, sometime, somewhere down the line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681483#comment-16681483
]
ASF GitHub Bot commented on FLINK-7155:
---
1u0 commented on a change in pull request #6976: [FLINK-7155][metrics] Add new
metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232263625
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MetricMapper.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+
+import java.time.Instant;
+import java.util.concurrent.TimeUnit;
+
+class MetricMapper {
+
+ static Point map(MeasurementInfo info, Instant timestamp, Gauge
gauge) {
+ Point.Builder builder = builder(info, timestamp);
+ Object value = gauge.getValue();
+ if (value instanceof Number) {
+ builder.addField("value", (Number) value);
+ } else {
+ builder.addField("value", String.valueOf(value));
+ }
+ return builder.build();
+ }
+
+ static Point map(MeasurementInfo info, Instant timestamp, Counter
counter) {
+ return builder(info, timestamp)
+ .addField("count", counter.getCount())
+ .build();
+ }
+
+ static Point map(MeasurementInfo info, Instant timestamp, Histogram
histogram) {
+ HistogramStatistics statistics = histogram.getStatistics();
+ return builder(info, timestamp)
+ .addField("run-count", histogram.getCount())
Review comment:
Personally I'm fine with this, but this would brake compatibility with other
[reporter](https://github.com/davidB/metrics-influxdb/blob/master/src/main/java/metrics_influxdb/measurements/MeasurementReporter.java#L129-L139)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681441#comment-16681441
]
ASF GitHub Bot commented on FLINK-7155:
---
1u0 commented on a change in pull request #6976: [FLINK-7155][metrics] Add new
metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232256095
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MeasurementInfoProviderTest.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerJobMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskMetricGroup;
+import org.apache.flink.util.AbstractID;
+
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ExecutionException;
+
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyChar;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MeasurementInfoProvider}.
+ */
+public class MeasurementInfoProviderTest {
+ private static final Random RANDOM = new Random();
+
+ private final MeasurementInfoProvider provider = new
MeasurementInfoProvider();
+
+ @Test
+ public void testGetMetricInfo() {
+ // MetricRegistry, required as the first argument in metric
groups.
+ MetricRegistryImpl metricRegistry = new MetricRegistryImpl(
+ MetricRegistryConfiguration.fromConfiguration(new
Configuration()));
+
+ // Create an example, nested metric group: taskmanager -> job
-> task
+ // Variables: ,
+ String hostname = "loc<>al\"::host\".:";
+ String taskManagerId = "tas:kMana::ger";
+ TaskManagerMetricGroup tmMetricGroup = new
TaskManagerMetricGroup(
+ metricRegistry, hostname, taskManagerId);
+
+ // Variables: ,
+ JobID jobID = new JobID();
+ String jobName = "testJ\"ob:-!ax..?";
+ TaskManagerJobMetricGroup tmJobMetricGroup = new
TaskManagerJobMetricGroup(
+ metricRegistry, tmMetricGroup, jobID, jobName);
+
+ // Variables: , , ,
,
+ JobVertexID taskId = new JobVertexID();
+ AbstractID taskAttemptID = new AbstractID();
+ String taskName = "test\"Ta\"..sk";
+ int subtaskIndex = RANDOM.nextInt();
+ int taskAttemptNum = RANDOM.nextInt();
+ TaskMetricGroup taskMetricGroup = new TaskMetricGroup(
+ metricRegistry, tmJobMetricGroup, taskId,
taskAttemptID, taskName, subtaskIndex, taskAttemptNum);
+
+ String metricName = "testCounter";
+ MetricGroup metricGroup = new FrontMetricGroup<>(0,
taskMetricGroup);
+
+ MeasurementInfo info = provider.getMetricInfo(metricName,
metricGroup);
+ assertNotNull(info);
+ assertEquals(
+ String.join("" +
MeasurementInfoProvider.SCOPE_SEPARATOR, "taskmanager", "job", "task",
metricName),
+ info.getName());
+ assertThat(info.getTags(), hasEntry("host", hostname));
+ assertThat(info.getTags(),
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681439#comment-16681439
]
ASF GitHub Bot commented on FLINK-7155:
---
1u0 commented on a change in pull request #6976: [FLINK-7155][metrics] Add new
metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232255912
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MeasurementInfoProviderTest.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerJobMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskMetricGroup;
+import org.apache.flink.util.AbstractID;
+
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ExecutionException;
+
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyChar;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MeasurementInfoProvider}.
+ */
+public class MeasurementInfoProviderTest {
+ private static final Random RANDOM = new Random();
+
+ private final MeasurementInfoProvider provider = new
MeasurementInfoProvider();
+
+ @Test
+ public void testGetMetricInfo() {
Review comment:
It can be removed, but it's more realistic case that happens in runtime. The
following test is narrowed down and too simplified test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681433#comment-16681433
]
ASF GitHub Bot commented on FLINK-7155:
---
1u0 commented on a change in pull request #6976: [FLINK-7155][metrics] Add new
metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232254767
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
+ when(counter.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, counter),
+ "count=42");
+ }
+
+ @Test
+ public void testMapHistogram() {
+ HistogramStatistics statistics =
mock(HistogramStatistics.class);
Review comment:
Are there any reasons to not use mocks in this test class?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681330#comment-16681330
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232227097
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
+ when(counter.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, counter),
+ "count=42");
+ }
+
+ @Test
+ public void testMapHistogram() {
+ HistogramStatistics statistics =
mock(HistogramStatistics.class);
+ when(statistics.getMax()).thenReturn(-5L);
+ when(statistics.getMin()).thenReturn(50L);
+ when(statistics.getMean()).thenReturn(1.2);
+ when(statistics.getStdDev()).thenReturn(0.7);
+ when(statistics.getQuantile(.5)).thenReturn(1.0);
+ when(statistics.getQuantile(.75)).thenReturn(2.0);
+ when(statistics.getQuantile(.95)).thenReturn(3.0);
+ when(statistics.getQuantile(.98)).thenReturn(4.0);
+ when(statistics.getQuantile(.99)).thenReturn(5.0);
+ when(statistics.getQuantile(.999)).thenReturn(6.0);
+
+ Histogram histogram = mock(Histogram.class);
+ when(histogram.getStatistics()).thenReturn(statistics);
+ when(histogram.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, histogram),
+ "50-percentile=1.0",
+ "75-percentile=2.0",
+ "95-percentile=3.0",
+ "98-percentile=4.0",
+ "99-percentile=5.0",
+ "999-percentile=6.0",
+ "count=0",
+ "max=-5",
+ "mean=1.2",
+ "min=50",
+ "run-count=42",
+ "std-dev=0.7");
+ }
+
+ @Test
+ public void testMapMeter() {
+ Meter meter = mock(Meter.class);
Review comment:
replace mock wit actual implementation
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681339#comment-16681339
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232227048
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
+ when(counter.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, counter),
+ "count=42");
+ }
+
+ @Test
+ public void testMapHistogram() {
+ HistogramStatistics statistics =
mock(HistogramStatistics.class);
+ when(statistics.getMax()).thenReturn(-5L);
+ when(statistics.getMin()).thenReturn(50L);
+ when(statistics.getMean()).thenReturn(1.2);
+ when(statistics.getStdDev()).thenReturn(0.7);
+ when(statistics.getQuantile(.5)).thenReturn(1.0);
+ when(statistics.getQuantile(.75)).thenReturn(2.0);
+ when(statistics.getQuantile(.95)).thenReturn(3.0);
+ when(statistics.getQuantile(.98)).thenReturn(4.0);
+ when(statistics.getQuantile(.99)).thenReturn(5.0);
+ when(statistics.getQuantile(.999)).thenReturn(6.0);
+
+ Histogram histogram = mock(Histogram.class);
Review comment:
replace mock wit actual implementation
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
>
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681328#comment-16681328
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232227784
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/InfluxdbReporterTest.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.SimpleCounter;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+
+import com.github.tomakehurst.wiremock.common.ConsoleNotifier;
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.containing;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.verify;
+import static
com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Integration test for {@link InfluxdbReporter}.
+ */
+public class InfluxdbReporterTest {
+ private static final String TEST_INFLUXDB_DB = "test-42";
+ private static final String METRIC_HOSTNAME = "task-mgr-1";
+ private static final String METRIC_TM_ID = "tm-id-123";
+
+ @Rule
+ public WireMockRule wireMockRule = new
WireMockRule(wireMockConfig().dynamicPort().notifier(new
ConsoleNotifier(false)));
Review comment:
can be final
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681329#comment-16681329
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232226619
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MeasurementInfoProviderTest.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerJobMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskMetricGroup;
+import org.apache.flink.util.AbstractID;
+
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ExecutionException;
+
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyChar;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MeasurementInfoProvider}.
+ */
+public class MeasurementInfoProviderTest {
+ private static final Random RANDOM = new Random();
+
+ private final MeasurementInfoProvider provider = new
MeasurementInfoProvider();
+
+ @Test
+ public void testGetMetricInfo() {
+ // MetricRegistry, required as the first argument in metric
groups.
+ MetricRegistryImpl metricRegistry = new MetricRegistryImpl(
+ MetricRegistryConfiguration.fromConfiguration(new
Configuration()));
+
+ // Create an example, nested metric group: taskmanager -> job
-> task
+ // Variables: ,
+ String hostname = "loc<>al\"::host\".:";
+ String taskManagerId = "tas:kMana::ger";
+ TaskManagerMetricGroup tmMetricGroup = new
TaskManagerMetricGroup(
+ metricRegistry, hostname, taskManagerId);
+
+ // Variables: ,
+ JobID jobID = new JobID();
+ String jobName = "testJ\"ob:-!ax..?";
+ TaskManagerJobMetricGroup tmJobMetricGroup = new
TaskManagerJobMetricGroup(
+ metricRegistry, tmMetricGroup, jobID, jobName);
+
+ // Variables: , , ,
,
+ JobVertexID taskId = new JobVertexID();
+ AbstractID taskAttemptID = new AbstractID();
+ String taskName = "test\"Ta\"..sk";
+ int subtaskIndex = RANDOM.nextInt();
+ int taskAttemptNum = RANDOM.nextInt();
+ TaskMetricGroup taskMetricGroup = new TaskMetricGroup(
+ metricRegistry, tmJobMetricGroup, taskId,
taskAttemptID, taskName, subtaskIndex, taskAttemptNum);
+
+ String metricName = "testCounter";
+ MetricGroup metricGroup = new FrontMetricGroup<>(0,
taskMetricGroup);
+
+ MeasurementInfo info = provider.getMetricInfo(metricName,
metricGroup);
+ assertNotNull(info);
+ assertEquals(
+ String.join("" +
MeasurementInfoProvider.SCOPE_SEPARATOR, "taskmanager", "job", "task",
metricName),
+ info.getName());
+ assertThat(info.getTags(), hasEntry("host", hostname));
+ assertThat(info.getTags(),
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681336#comment-16681336
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232228781
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/InfluxdbReporterTest.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.SimpleCounter;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+
+import com.github.tomakehurst.wiremock.common.ConsoleNotifier;
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.containing;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.verify;
+import static
com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Integration test for {@link InfluxdbReporter}.
+ */
+public class InfluxdbReporterTest {
+ private static final String TEST_INFLUXDB_DB = "test-42";
+ private static final String METRIC_HOSTNAME = "task-mgr-1";
+ private static final String METRIC_TM_ID = "tm-id-123";
+
+ @Rule
+ public WireMockRule wireMockRule = new
WireMockRule(wireMockConfig().dynamicPort().notifier(new
ConsoleNotifier(false)));
+
+ @Test
+ public void test() throws Exception {
+ String configPrefix = ConfigConstants.METRICS_REPORTER_PREFIX +
"test.";
+ Configuration configuration = new Configuration();
+ configuration.setString(
+ configPrefix +
ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX,
+ InfluxdbReporter.class.getTypeName());
+ configuration.setString(configPrefix + "host", "localhost");
+ configuration.setString(configPrefix + "port",
String.valueOf(wireMockRule.port()));
+ configuration.setString(configPrefix + "db", TEST_INFLUXDB_DB);
+
+ MetricRegistryImpl metricRegistry = new
MetricRegistryImpl(MetricRegistryConfiguration.fromConfiguration(configuration));
+ MetricReporter reporter = metricRegistry.getReporters().get(0);
+ assertTrue(reporter instanceof InfluxdbReporter);
+ InfluxdbReporter influxdbReporter = (InfluxdbReporter) reporter;
+
+ String metricName = "TestCounter";
+ Counter counter = registerTestMetric(metricName,
metricRegistry);
+ MeasurementInfo measurementInfo =
influxdbReporter.counters.get(counter);
+ assertNotNull("test metric must be registered in the reporter",
measurementInfo);
+ String fullMetricName = "taskmanager_" + metricName;
+ assertEquals(fullMetricName,
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681337#comment-16681337
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232226981
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
+ when(counter.getCount()).thenReturn(42L);
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, counter),
+ "count=42");
+ }
+
+ @Test
+ public void testMapHistogram() {
+ HistogramStatistics statistics =
mock(HistogramStatistics.class);
Review comment:
replace mock with actual implementation
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681338#comment-16681338
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232228289
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/InfluxdbReporterTest.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.SimpleCounter;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+
+import com.github.tomakehurst.wiremock.common.ConsoleNotifier;
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.containing;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.verify;
+import static
com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Integration test for {@link InfluxdbReporter}.
+ */
+public class InfluxdbReporterTest {
+ private static final String TEST_INFLUXDB_DB = "test-42";
+ private static final String METRIC_HOSTNAME = "task-mgr-1";
+ private static final String METRIC_TM_ID = "tm-id-123";
+
+ @Rule
+ public WireMockRule wireMockRule = new
WireMockRule(wireMockConfig().dynamicPort().notifier(new
ConsoleNotifier(false)));
+
+ @Test
+ public void test() throws Exception {
+ String configPrefix = ConfigConstants.METRICS_REPORTER_PREFIX +
"test.";
+ Configuration configuration = new Configuration();
+ configuration.setString(
+ configPrefix +
ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX,
+ InfluxdbReporter.class.getTypeName());
+ configuration.setString(configPrefix + "host", "localhost");
+ configuration.setString(configPrefix + "port",
String.valueOf(wireMockRule.port()));
+ configuration.setString(configPrefix + "db", TEST_INFLUXDB_DB);
+
+ MetricRegistryImpl metricRegistry = new
MetricRegistryImpl(MetricRegistryConfiguration.fromConfiguration(configuration));
+ MetricReporter reporter = metricRegistry.getReporters().get(0);
+ assertTrue(reporter instanceof InfluxdbReporter);
+ InfluxdbReporter influxdbReporter = (InfluxdbReporter) reporter;
+
+ String metricName = "TestCounter";
+ Counter counter = registerTestMetric(metricName,
metricRegistry);
+ MeasurementInfo measurementInfo =
influxdbReporter.counters.get(counter);
+ assertNotNull("test metric must be registered in the reporter",
measurementInfo);
+ String fullMetricName = "taskmanager_" + metricName;
+ assertEquals(fullMetricName,
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681333#comment-16681333
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232225906
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MetricMapper.java
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+
+import java.time.Instant;
+import java.util.concurrent.TimeUnit;
+
+class MetricMapper {
+
+ static Point map(MeasurementInfo info, Instant timestamp, Gauge
gauge) {
+ Point.Builder builder = builder(info, timestamp);
+ Object value = gauge.getValue();
+ if (value instanceof Number) {
+ builder.addField("value", (Number) value);
+ } else {
+ builder.addField("value", String.valueOf(value));
+ }
+ return builder.build();
+ }
+
+ static Point map(MeasurementInfo info, Instant timestamp, Counter
counter) {
+ return builder(info, timestamp)
+ .addField("count", counter.getCount())
+ .build();
+ }
+
+ static Point map(MeasurementInfo info, Instant timestamp, Histogram
histogram) {
+ HistogramStatistics statistics = histogram.getStatistics();
+ return builder(info, timestamp)
+ .addField("run-count", histogram.getCount())
Review comment:
let's keep the names in sync with the StatsD/Slf4j reporter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681332#comment-16681332
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232228849
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/InfluxdbReporterTest.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.SimpleCounter;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+
+import com.github.tomakehurst.wiremock.common.ConsoleNotifier;
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.containing;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.verify;
+import static
com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Integration test for {@link InfluxdbReporter}.
+ */
+public class InfluxdbReporterTest {
+ private static final String TEST_INFLUXDB_DB = "test-42";
+ private static final String METRIC_HOSTNAME = "task-mgr-1";
+ private static final String METRIC_TM_ID = "tm-id-123";
+
+ @Rule
+ public WireMockRule wireMockRule = new
WireMockRule(wireMockConfig().dynamicPort().notifier(new
ConsoleNotifier(false)));
+
+ @Test
+ public void test() throws Exception {
+ String configPrefix = ConfigConstants.METRICS_REPORTER_PREFIX +
"test.";
+ Configuration configuration = new Configuration();
+ configuration.setString(
+ configPrefix +
ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX,
+ InfluxdbReporter.class.getTypeName());
+ configuration.setString(configPrefix + "host", "localhost");
+ configuration.setString(configPrefix + "port",
String.valueOf(wireMockRule.port()));
+ configuration.setString(configPrefix + "db", TEST_INFLUXDB_DB);
+
+ MetricRegistryImpl metricRegistry = new
MetricRegistryImpl(MetricRegistryConfiguration.fromConfiguration(configuration));
+ MetricReporter reporter = metricRegistry.getReporters().get(0);
+ assertTrue(reporter instanceof InfluxdbReporter);
+ InfluxdbReporter influxdbReporter = (InfluxdbReporter) reporter;
+
+ String metricName = "TestCounter";
+ Counter counter = registerTestMetric(metricName,
metricRegistry);
+ MeasurementInfo measurementInfo =
influxdbReporter.counters.get(counter);
+ assertNotNull("test metric must be registered in the reporter",
measurementInfo);
+ String fullMetricName = "taskmanager_" + metricName;
+ assertEquals(fullMetricName,
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681334#comment-16681334
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232228632
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/InfluxdbReporterTest.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.SimpleCounter;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+
+import com.github.tomakehurst.wiremock.common.ConsoleNotifier;
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.containing;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.verify;
+import static
com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Integration test for {@link InfluxdbReporter}.
+ */
+public class InfluxdbReporterTest {
+ private static final String TEST_INFLUXDB_DB = "test-42";
+ private static final String METRIC_HOSTNAME = "task-mgr-1";
+ private static final String METRIC_TM_ID = "tm-id-123";
+
+ @Rule
+ public WireMockRule wireMockRule = new
WireMockRule(wireMockConfig().dynamicPort().notifier(new
ConsoleNotifier(false)));
+
+ @Test
+ public void test() throws Exception {
+ String configPrefix = ConfigConstants.METRICS_REPORTER_PREFIX +
"test.";
+ Configuration configuration = new Configuration();
+ configuration.setString(
+ configPrefix +
ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX,
+ InfluxdbReporter.class.getTypeName());
+ configuration.setString(configPrefix + "host", "localhost");
+ configuration.setString(configPrefix + "port",
String.valueOf(wireMockRule.port()));
+ configuration.setString(configPrefix + "db", TEST_INFLUXDB_DB);
+
+ MetricRegistryImpl metricRegistry = new
MetricRegistryImpl(MetricRegistryConfiguration.fromConfiguration(configuration));
+ MetricReporter reporter = metricRegistry.getReporters().get(0);
+ assertTrue(reporter instanceof InfluxdbReporter);
+ InfluxdbReporter influxdbReporter = (InfluxdbReporter) reporter;
+
+ String metricName = "TestCounter";
+ Counter counter = registerTestMetric(metricName,
metricRegistry);
+ MeasurementInfo measurementInfo =
influxdbReporter.counters.get(counter);
+ assertNotNull("test metric must be registered in the reporter",
measurementInfo);
+ String fullMetricName = "taskmanager_" + metricName;
+ assertEquals(fullMetricName,
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681335#comment-16681335
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232227842
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/InfluxdbReporterTest.java
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.SimpleCounter;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+
+import com.github.tomakehurst.wiremock.common.ConsoleNotifier;
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.containing;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.postRequestedFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo;
+import static com.github.tomakehurst.wiremock.client.WireMock.verify;
+import static
com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Integration test for {@link InfluxdbReporter}.
+ */
+public class InfluxdbReporterTest {
Review comment:
should extend `TestLogger` (also applies to other tests)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681327#comment-16681327
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232224560
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+import org.influxdb.dto.BatchPoints;
+
+import javax.annotation.Nullable;
+
+import java.time.Instant;
+import java.util.ConcurrentModificationException;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.DB;
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.HOST;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.PASSWORD;
+import static org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.PORT;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.USERNAME;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.getInteger;
+import static
org.apache.flink.metrics.influxdb.InfluxdbReporterOptions.getString;
+
+/**
+ * {@link MetricReporter} that exports {@link Metric Metrics} via InfluxDB.
+ */
+@Experimental
+public class InfluxdbReporter extends AbstractReporter
implements Scheduled {
+
+ private String database;
+ private InfluxDB influxDB;
+
+ public InfluxdbReporter() {
+ super(new MeasurementInfoProvider());
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ String host = getString(config, HOST);
+ int port = getInteger(config, PORT);
+ if (host == null || host.isEmpty() || port < 1) {
Review comment:
also check upper bound for port
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681331#comment-16681331
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232226938
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MetricMapperTest.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+
+import org.influxdb.dto.Point;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MetricMapper} checking that metrics are converted to
InfluxDB client objects as expected.
+ */
+public class MetricMapperTest {
+
+ private final String name = "a-metric-name";
+ private final MeasurementInfo info = getMeasurementInfo(name);
+ private final Instant timestamp = Instant.now();
+
+ @Test
+ public void testMapGauge() {
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42),
+ "value=42");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
null),
+ "value=null");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
"hello"),
+ "value=hello");
+
+ verifyPoint(
+ MetricMapper.map(info, timestamp, (Gauge) () ->
42L),
+ "value=42");
+ }
+
+ @Test
+ public void testMapCounter() {
+ Counter counter = mock(Counter.class);
Review comment:
Use a `SimpleCounter` and set value manually to 42.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681326#comment-16681326
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r232226784
##
File path:
flink-metrics/flink-metrics-influxdb/src/test/java/org/apache/flink/metrics/influxdb/MeasurementInfoProviderTest.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.MetricRegistryImpl;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerJobMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskMetricGroup;
+import org.apache.flink.util.AbstractID;
+
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ExecutionException;
+
+import static org.hamcrest.Matchers.hasEntry;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyChar;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for {@link MeasurementInfoProvider}.
+ */
+public class MeasurementInfoProviderTest {
+ private static final Random RANDOM = new Random();
+
+ private final MeasurementInfoProvider provider = new
MeasurementInfoProvider();
+
+ @Test
+ public void testGetMetricInfo() {
Review comment:
this test seems unnecessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678206#comment-16678206 ] ASF GitHub Bot commented on FLINK-7155: --- rmetzger commented on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-436618075 Okay, then let's not create the db automatically by the reporter. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677965#comment-16677965 ] ASF GitHub Bot commented on FLINK-7155: --- zentol commented on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-436573129 > What do you think about making db configuration parameter as required? Makes sense to me. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677966#comment-16677966 ] ASF GitHub Bot commented on FLINK-7155: --- zentol edited a comment on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-436573129 > What do you think about making db configuration parameter as required? Makes sense to me. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677962#comment-16677962
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r231446336
##
File path: docs/monitoring/metrics.md
##
@@ -672,6 +672,30 @@ metrics.reporter.grph.protocol: TCP
{% endhighlight %}
+### InfluxDB (org.apache.flink.metrics.influxdb.InfluxdbReporter)
+
+In order to use this reporter you must copy
`/opt/flink-metrics-influxdb-{{site.version}}.jar` into the `/lib` folder
+of your Flink distribution.
+
+Parameters:
+
+{% include generated/influxdb_reporter_configuration.html %}
Review comment:
you also have to commit the generated file ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677947#comment-16677947 ] ASF GitHub Bot commented on FLINK-7155: --- 1u0 commented on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-436570526 @zentol, an exception would be thrown on `report()`. What do you think about making `db` configuration parameter as required? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677945#comment-16677945 ] ASF GitHub Bot commented on FLINK-7155: --- zentol commented on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-436569484 I agree that the reporter should not create a database, at the very least not in the first version. If the db doesn't exist, does the reporter fail in `open()` or `report()`? If it is the latter then this would be acceptable behavior. (since it will simply be retried later on). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677932#comment-16677932 ] ASF GitHub Bot commented on FLINK-7155: --- 1u0 commented on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-436566905 @rmetzger > Will the influxdb client just throw an exception if the db doesn't exist? Yes, if the database doesn't exists, the InfluxDB service would return an error response. This would result in an exception during reporting. > Does it make sense to consider automatically creating the db if it doesn't exist? Imo, better **not** to do it in the reporter. This is mainly due to how InfluxDB operates. Metrics in InfluxDB are stored under `.`. If the reporter creates just ``, then the InfluxDB (by default) would generate a default `` with infinite duration. This may be not desired for production use. Technically, the report can create the database. As follow up, it can also be extended to create the retention policy, but this would require much more parameters (name, duration, replication, shard duration). At the end this results in configuration bloat in the reporter and possible wrong configuration in production use. **Note:** Maybe, to better reflect that the InfluxDB database should be created separately, make the `db` configuration parameter as required (without default). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676493#comment-16676493 ] ASF GitHub Bot commented on FLINK-7155: --- rmetzger commented on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-436199529 Thank you for the pull request. One thing I'd like to discuss is the behavior regarding automatic database creation. From the code, it seems that there is no special handling for the case when the database the user configured doesn't exist. Will the influxdb client just throw an exception if the db doesn't exist? Does it make sense to consider automatically creating the db if it doesn't exist? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670204#comment-16670204
]
ASF GitHub Bot commented on FLINK-7155:
---
patricklucas closed pull request #4299: [FLINK-7155] [metrics] Add Influxdb
reporter
URL: https://github.com/apache/flink/pull/4299
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-metrics/flink-metrics-influxdb/pom.xml
b/flink-metrics/flink-metrics-influxdb/pom.xml
new file mode 100644
index 000..57ac07ccc57
--- /dev/null
+++ b/flink-metrics/flink-metrics-influxdb/pom.xml
@@ -0,0 +1,91 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.4-SNAPSHOT
+ ..
+
+
+ flink-metrics-influxdb
+ flink-metrics-influxdb
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-dropwizard
+ ${project.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-core
+ ${metrics.version}
+
+
+
+ com.github.davidb
+ metrics-influxdb
+ 0.9.3
+
+
+
+
+
+
+ maven-assembly-plugin
+ 2.4
+
+
+
jar-with-dependencies
+
+
+
+
+ make-assembly
+ package
+
+ single
+
+
+
+
+
+
+
diff --git
a/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
b/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
new file mode 100644
index 000..19575539175
--- /dev/null
+++
b/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.dropwizard.ScheduledDropwizardReporter;
+import org.apache.flink.metrics.MetricConfig;
+
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.ScheduledReporter;
+import metrics_influxdb.HttpInfluxdbProtocol;
+import
metrics_influxdb.api.measurements.CategoriesMetricMeasurementTransformer;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class acts as a factory for the {@link
metrics_influxdb.InfluxdbReporter} and allows using it as a Flink
+ * reporter.
+ */
+@PublicEvolving
+public class InfluxdbReporter extends ScheduledDropwizardReporter {
+
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670186#comment-16670186
]
ASF GitHub Bot commented on FLINK-7155:
---
zentol commented on a change in pull request #6976: [FLINK-7155][metrics] Add
new metrics reporter to InfluxDB
URL: https://github.com/apache/flink/pull/6976#discussion_r229722569
##
File path:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+
+import org.influxdb.InfluxDB;
+import org.influxdb.InfluxDBFactory;
+import org.influxdb.dto.BatchPoints;
+
+import javax.annotation.Nullable;
+
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.ConcurrentModificationException;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+/**
+ * {@link MetricReporter} that exports {@link Metric Metrics} via InfluxDB.
+ */
+@Experimental
+public class InfluxdbReporter extends AbstractReporter
implements Scheduled {
+
+ private static final String ARG_PROTOCOL = "protocol";
+ private static final List SUPPORTED_PROTOCOLS =
Arrays.asList("http", "https");
+ private static final String ARG_HOST = "host";
Review comment:
as a quick note, if you rewrite the options similar to one of the
[prometheus
reporters](https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporterOptions.java)
then documenting them becomes trivial as you can just plug this class into the
[configuration docs
generator](https://github.com/apache/flink/blob/master/flink-docs/src/main/java/org/apache/flink/docs/configuration/ConfigOptionsDocGenerator.java#L58)
and include the generated table into the docs with `{% include
generated/_configuration.html %}`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>Priority: Major
> Labels: pull-request-available
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670081#comment-16670081 ] ASF GitHub Bot commented on FLINK-7155: --- 1u0 commented on issue #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976#issuecomment-434679481 Notes: * this PR is an alternative of #4299, but uses different InfluxDB client library. Also, it's (maybe) not directly compatible with other open source solutions. * please pay attention if the configuration options makes sense for the community. I can continue with this PR and update the documentation in `docs/monitoring/metrics.md`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670064#comment-16670064 ] ASF GitHub Bot commented on FLINK-7155: --- 1u0 opened a new pull request #6976: [FLINK-7155][metrics] Add new metrics reporter to InfluxDB URL: https://github.com/apache/flink/pull/6976 ## What is the purpose of the change This PR introduces new, push-style, Flink metric reporter to [InfluxDB](https://docs.influxdata.com/influxdb/v1.6/about_the_project/) service using official [influxdb-java](https://github.com/influxdata/influxdb-java) client. ## Brief change log - Add InfluxDB metrics reporter ## Verifying this change This change added tests and can be verified as follows: - Added unit tests for functional parts of the reporter - Added integration test for the reporter (similar to existing ones, like `StatsDReporterTest`) ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): don't know - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: don't know - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? not documented (yet) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas >Priority: Major > Labels: pull-request-available > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307152#comment-16307152
]
ASF GitHub Bot commented on FLINK-7155:
---
Github user gobozov commented on a diff in the pull request:
https://github.com/apache/flink/pull/4299#discussion_r159136038
--- Diff:
flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/InfluxdbReporter.java
---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.influxdb;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.dropwizard.ScheduledDropwizardReporter;
+import org.apache.flink.metrics.MetricConfig;
+
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.ScheduledReporter;
+import metrics_influxdb.HttpInfluxdbProtocol;
+import
metrics_influxdb.api.measurements.CategoriesMetricMeasurementTransformer;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class acts as a factory for the {@link
metrics_influxdb.InfluxdbReporter} and allows using it as a Flink
+ * reporter.
+ */
+@PublicEvolving
+public class InfluxdbReporter extends ScheduledDropwizardReporter {
+
+ public static final String ARG_USER = "user";
+ public static final String ARG_PASSWORD = "password";
+ public static final String ARG_DB = "db";
+
+ @Override
+ public ScheduledReporter getReporter(final MetricConfig config) {
+ final String host = config.getString(ARG_HOST, "localhost");
+ final Integer port = config.getInteger(ARG_PORT,
HttpInfluxdbProtocol.DEFAULT_PORT);
+ final String user = config.getString(ARG_USER, null);
+ final String password = config.getString(ARG_PASSWORD, null);
+ final String db = config.getString(ARG_DB, "flink");
+ final String conversionRate =
config.getString(ARG_CONVERSION_RATE, null);
+ final String conversionDuration =
config.getString(ARG_CONVERSION_DURATION, null);
+
+ final metrics_influxdb.InfluxdbReporter.Builder builder =
+ metrics_influxdb.InfluxdbReporter.forRegistry(registry);
+
+ builder.protocol(new HttpInfluxdbProtocol(host, port, user,
password, db));
--- End diff --
We built same thing for our needs using same InfluxDb reporter. The
disadvantage is that it does not support https, we modified it to support
sslContext. I also don't see way to provide tags to reporter builder.
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162576#comment-16162576 ] ASF GitHub Bot commented on FLINK-7155: --- Github user yew1eb commented on the issue: https://github.com/apache/flink/pull/4299 Hi @patricklucas , Coding guidelines “No WIP pull requests.” Maybe you should see: http://flink.apache.org/contribute-code.html BTW, If you do not have free time to finish it, I am willing to take over. :smile: Best, Hai Zhou > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Patrick Lucas >Assignee: Patrick Lucas > Fix For: 1.4.0 > > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082514#comment-16082514
]
ASF GitHub Bot commented on FLINK-7155:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4299#discussion_r126748789
--- Diff: flink-metrics/flink-metrics-influxdb/pom.xml ---
@@ -0,0 +1,97 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.4-SNAPSHOT
+ ..
+
+
+ flink-metrics-influxdb
+ flink-metrics-influxdb
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-dropwizard
+ ${project.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-core
+ ${metrics.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-graphite
+ ${metrics.version}
+
+
+
+ com.github.davidb
+ metrics-influxdb
+ 0.9.3
+
+
+
+
+
+
+ maven-assembly-plugin
+ 2.4
+
+
+
jar-with-dependencies
--- End diff --
We intend to use the shade plugin for the graphite reporter as well, see
https://issues.apache.org/jira/browse/FLINK-4173
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
> Fix For: 1.4.0
>
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082477#comment-16082477
]
ASF GitHub Bot commented on FLINK-7155:
---
Github user patricklucas commented on a diff in the pull request:
https://github.com/apache/flink/pull/4299#discussion_r126742051
--- Diff: flink-metrics/flink-metrics-influxdb/pom.xml ---
@@ -0,0 +1,97 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.4-SNAPSHOT
+ ..
+
+
+ flink-metrics-influxdb
+ flink-metrics-influxdb
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-dropwizard
+ ${project.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-core
+ ${metrics.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-graphite
--- End diff --
This was errant; I'll remove it.
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
> Fix For: 1.4.0
>
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082480#comment-16082480
]
ASF GitHub Bot commented on FLINK-7155:
---
Github user patricklucas commented on a diff in the pull request:
https://github.com/apache/flink/pull/4299#discussion_r126742152
--- Diff: flink-metrics/flink-metrics-influxdb/pom.xml ---
@@ -0,0 +1,97 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.4-SNAPSHOT
+ ..
+
+
+ flink-metrics-influxdb
+ flink-metrics-influxdb
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-dropwizard
+ ${project.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-core
+ ${metrics.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-graphite
+ ${metrics.version}
+
+
+
+ com.github.davidb
+ metrics-influxdb
+ 0.9.3
+
+
+
+
+
+
+ maven-assembly-plugin
+ 2.4
+
+
+
jar-with-dependencies
--- End diff --
This is exactly how the graphite reporter does it, which is extremely
similar to this one. I'm not sure exactly what you want me to change here.
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
> Fix For: 1.4.0
>
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082416#comment-16082416
]
ASF GitHub Bot commented on FLINK-7155:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4299#discussion_r126723260
--- Diff: flink-metrics/flink-metrics-influxdb/pom.xml ---
@@ -0,0 +1,97 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.4-SNAPSHOT
+ ..
+
+
+ flink-metrics-influxdb
+ flink-metrics-influxdb
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-dropwizard
+ ${project.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-core
+ ${metrics.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-graphite
--- End diff --
why is this dependency needed?
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
> Fix For: 1.4.0
>
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082417#comment-16082417
]
ASF GitHub Bot commented on FLINK-7155:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4299#discussion_r126723609
--- Diff: flink-metrics/flink-metrics-influxdb/pom.xml ---
@@ -0,0 +1,97 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.4-SNAPSHOT
+ ..
+
+
+ flink-metrics-influxdb
+ flink-metrics-influxdb
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-dropwizard
+ ${project.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-core
+ ${metrics.version}
+
+
+
+ io.dropwizard.metrics
+ metrics-graphite
+ ${metrics.version}
+
+
+
+ com.github.davidb
+ metrics-influxdb
+ 0.9.3
+
+
+
+
+
+
+ maven-assembly-plugin
+ 2.4
+
+
+
jar-with-dependencies
--- End diff --
please use the maven-shade plugin instead.
> Add Influxdb metrics reporter
> -
>
> Key: FLINK-7155
> URL: https://issues.apache.org/jira/browse/FLINK-7155
> Project: Flink
> Issue Type: Improvement
>Reporter: Patrick Lucas
>Assignee: Patrick Lucas
> Fix For: 1.4.0
>
>
> [~jgrier] has a [simple Influxdb metrics reporter for
> Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter]
> that is a thing wrapper around [a lightweight, public-domain Influxdb
> reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics.
> We can implement this very easily in Java in the same as as
> flink-metrics-graphite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7155) Add Influxdb metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082340#comment-16082340 ] ASF GitHub Bot commented on FLINK-7155: --- GitHub user patricklucas opened a pull request: https://github.com/apache/flink/pull/4299 [FLINK-7155] [metrics] Add Influxdb reporter You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricklucas/flink FLINK-7155_add_influxdb_metrics_reporter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4299.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4299 > Add Influxdb metrics reporter > - > > Key: FLINK-7155 > URL: https://issues.apache.org/jira/browse/FLINK-7155 > Project: Flink > Issue Type: Improvement >Reporter: Patrick Lucas >Assignee: Patrick Lucas > Fix For: 1.4.0 > > > [~jgrier] has a [simple Influxdb metrics reporter for > Flink|https://github.com/jgrier/flink-stuff/tree/master/flink-influx-reporter] > that is a thing wrapper around [a lightweight, public-domain Influxdb > reporter|https://github.com/davidB/metrics-influxdb] for Codahale metrics. > We can implement this very easily in Java in the same as as > flink-metrics-graphite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
