[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036500#comment-17036500 ] ASF subversion and git services commented on AIRFLOW-2906: -- Commit ed2f3dc4ca28609bcead681f95b6e26e13d64c28 in airflow's branch refs/heads/master from chandu kavar [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ed2f3dc ] [AIRFLOW-2906] Add support for DataDog's dogstatsd when emitting metrics (#7376) > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036499#comment-17036499 ] ASF GitHub Bot commented on AIRFLOW-2906: - ashb commented on pull request #7376: [AIRFLOW-2906] Add datadog(dogstatsd) support to send airflow metrics URL: https://github.com/apache/airflow/pull/7376 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025294#comment-17025294 ] Andrew Tilley commented on AIRFLOW-2906: I'm confused about the existence of this issue; isn't Datadog integration already available in [https://github.com/apache/airflow/commits/master/airflow/contrib/hooks/datadog_hook.py]? > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025395#comment-17025395 ] Kamil Bregula commented on AIRFLOW-2906: Hook allows you to send additional metrics if someone uses it in your operator. This PR talks about sending StatsD metrics. [https://airflow.readthedocs.io/en/latest/metrics.html] Airflow does not yet have direct integration with Datadog, but this can be achieved by DogStatsD - [https://docs.datadoghq.com/developers/dogstatsd/?tab=python]. > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029616#comment-17029616 ] Chandu Kavar commented on AIRFLOW-2906: --- Yes, I have made some changes to my local and stuck in unit tests for that. > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031634#comment-17031634 ] ASF GitHub Bot commented on AIRFLOW-2906: - chandulal commented on pull request #7376: [AIRFLOW-2906] Add datadog support to send airflow metrics URL: https://github.com/apache/airflow/pull/7376 Currently, Airflow sends metric to Datadog using statsd and you won't be able to associated tags to metrics send by airflow. This PR has dogstatsd (Datadog) support. After this PR, you can associate constant tags to all metrics as well as tags specific to single metric as well. --- Issue link: [AIRFLOW-2906](https://issues.apache.org/jira/browse/AIRFLOW-2906) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881743#comment-16881743 ] Chandu Kavar commented on AIRFLOW-2906: --- We also need this functionlities to collect metrics from multiple airflow instances and aggregate by tags. Currently, we can't send the tags using statsd. Is there any progress on this? > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Austin Hsu >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886779#comment-16886779 ] Chandu Kavar commented on AIRFLOW-2906: --- I started working on to use Datadog Python library in case anyone wants to submit tags to Datadog. > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067113#comment-17067113 ] ASF subversion and git services commented on AIRFLOW-2906: -- Commit df13f8b45428e26661d5c5feb836eff3a511ae79 in airflow's branch refs/heads/v1-10-test from chandu kavar [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=df13f8b ] [AIRFLOW-2906] Add support for DataDog's dogstatsd when emitting metrics (#7376) (cherry picked from commit ed2f3dc4ca28609bcead681f95b6e26e13d64c28) > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > Fix For: 1.10.10 > > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067197#comment-17067197 ] ASF subversion and git services commented on AIRFLOW-2906: -- Commit fd21efcde2f42e98ea719bb55476107793a8e321 in airflow's branch refs/heads/v1-10-test from chandu kavar [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fd21efc ] [AIRFLOW-2906] Add support for DataDog's dogstatsd when emitting metrics (#7376) (cherry picked from commit ed2f3dc4ca28609bcead681f95b6e26e13d64c28) > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > Fix For: 1.10.10 > > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2906) DataDog Integration for Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985498#comment-16985498 ] jack commented on AIRFLOW-2906: --- [~cckavar] do you have something ready? > DataDog Integration for Airflow > --- > > Key: AIRFLOW-2906 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2906 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.8.0 >Reporter: Austin Hsu >Assignee: Chandu Kavar >Priority: Minor > Labels: metrics > > Add functionality to Airflow to enable sending of metrics to DataDog. > DataDog provides support for tags which allows us to aggregate data more > easily and visualize it. We can utilize the [Datadog python > library|https://github.com/DataDog/datadogpy] python library and the [Datadog > ThreadStats > module|https://datadogpy.readthedocs.io/en/latest/#datadog-threadstats-module] > to send metrics directly to DataDog without needing to spin up an agent to > forward the metrics. The current implementation in 1.8 uses the statsd > library to send the metrics which provides us with much less control to > filter our data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
