Hi Airflow community members, Apache Airflow has been moving steadily from conventional statsd based telemetry into OpenTelemetry (opentelemetry.io), which has gained popularity and support for the last few years. There has been AIP proposals which had a 'phased' approach to incorporate
1. Metrics 2. Traces 3. Logs Based on the current availability and maturity of Otel's support. Since the metrics emission based on statsd were the fastest low-hanging fruit that we could implement OTEL metrics support relatively easy, which we did as our first try. Second was the Traces. OTEL support on traces for Python was very mature and well established. Even though Airflow did not have any prior implementation on emitting Traces, we were able to instrument it and produce traces on both Airflow specific processes (jobs) (e.g. schedulers, triggers, executors), as well as task runs. However, the area for Logs were not done yet, due at that time a relatively early and immature state of logs support in Otel. Now, it seems like Logging support on Python has become stable and reliable that we can use it to emit log messages in Otel format. So, the timing seems to be quite perfect for us to start up a discussion on it. Logging support in Airflow is basically provided via 'provider' model where Airflow itself does not have a specific implementation of how logging is done, but requests contributors to provide provider that can plug-in their logging support. Would we want to move to that direction, of having 'opentelemetry' provider ? Or do we want to have logging support as more of a native support way? The reason why I would like to start this discussion thread was to gain opinions from the community on whether Airflow has the interest to support OTEL logging first, and then any opinions on specifics of what would be the best way to support OTEL logging. Please feel free to reply and add any comments to the thread to voice your opinions. If there isn't any substantial interest in OTEL logging, well, we can live without it.
