AutomationDev85 opened a new pull request, #47773: URL: https://github.com/apache/airflow/pull/47773
Hi Airflow community, with this PR we want to add a new metric named ti.running with the queue label. Let me explain why I decided to introduce this new metric: In our Airflow deployment we are using different queues to shift different workflow to different worker types (EdgeWorker CeleryWorker). We want to have a dashboard which shows number of running tasks for the different queues. 1) The issue by using the ti.start and ti.finish counters are that they are running out of sync if the statsd resets counters so only added the queue field into the metrics and make a query which sub ti.finish from ti.start does not work. 2) Then we have the executor.running_tasks metric. The issue with this metric is that the self.running in the executor contains only the TaskInstanceKey values which do not contain the queue. We are open for new ideas to export this metric in a different way! But for the moment we ended in this solution. We think also other people are interested in this metric by filter on the queue label in der future when using the Celery worker workload and Edge worker workload in the same Airflow instance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
