potiuk commented on issue #12771:
URL: https://github.com/apache/airflow/issues/12771#issuecomment-1383856897

   > Hi folks! Just curious: is there a way we could auto instrument Airflow 
through the use of cluster policies? For example, what if we created a task 
policy that a wraps the callback and execute methods of a task? The wrapper 
could perform actions similar to the 
[Celery](https://github.com/uber-common/opentracing-python-instrumentation/blob/master/opentracing_instrumentation/client_hooks/celery.py)
 example above.
   
   You could use cluster policies, yes. But also you have (in Airlfow 2.5.0) 
listeners that you could use for that purpose. 
https://airflow.apache.org/docs/apache-airflow/stable/listeners.html - they 
were designed for a different purpose, but I believe you can go a long way by 
utilising these two mechanisms if you think the "standard" approach of Airlfow 
will not be good enough for you. Airlow in it's "2" line (now at 2.5) is more 
and more "platform" that you can extend in various ways and we are approaching 
it in the way that you can write your own extensions by plugging in your code 
with well-defined APIs in the way that will allow you to customize it heavily, 
while relying on the fact that you will be able to upgrade Airflow without 
losing your customizations. We have a few documentation updates coming in 2.5.1 
specifically targetting to make it clearer what you CAN and what you CAN't 
modify in this way. Both cluster policies and listeners are "public interface" 
of Airlow and you 
 can utilize them for that purpose.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to