Very much agree with Ash here. On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <a...@apache.org> wrote:
> Hi Howard, > > Your timing is…. Interesting given we are focusing on getting 3.0 shipped > and are in the final throws of that. > > Let's revisit this in May or June once the dust has settled from 3.0, any > immediate bug fixes we might need, and give people a little bit of time to > unwind after the crunch. > > -ash > > > On 14 Apr 2025, at 16:22, Howard Yoo <howard...@gmail.com> wrote: > > > > Hi Airflow community members, > > > > Apache Airflow has been moving steadily from conventional statsd based > > telemetry into OpenTelemetry (opentelemetry.io), which has gained > > popularity and support for the last few years. There has been AIP > proposals > > which had a 'phased' approach to incorporate > > > > 1. Metrics > > 2. Traces > > 3. Logs > > > > Based on the current availability and maturity of Otel's support. Since > the > > metrics emission based on statsd were the fastest low-hanging fruit that > we > > could implement OTEL metrics support relatively easy, which we did as our > > first try. > > > > Second was the Traces. OTEL support on traces for Python was very mature > > and well established. Even though Airflow did not have any prior > > implementation on emitting Traces, we were able to instrument it and > > produce traces on both Airflow specific processes (jobs) (e.g. > schedulers, > > triggers, executors), as well as task runs. > > > > However, the area for Logs were not done yet, due at that time a > relatively > > early and immature state of logs support in Otel. > > > > Now, it seems like Logging support on Python has become stable and > reliable > > that we can use it to emit log messages in Otel format. So, the timing > > seems to be quite perfect for us to start up a discussion on it. > > > > Logging support in Airflow is basically provided via 'provider' model > where > > Airflow itself does not have a specific implementation of how logging is > > done, but requests contributors to provide provider that can plug-in > their > > logging support. > > > > Would we want to move to that direction, of having 'opentelemetry' > provider > > ? > > Or do we want to have logging support as more of a native support way? > > > > The reason why I would like to start this discussion thread was to gain > > opinions from the community on whether Airflow has the interest to > support > > OTEL logging first, and then any opinions on specifics of what would be > the > > best way to support OTEL logging. > > > > Please feel free to reply and add any comments to the thread to voice > your > > opinions. If there isn't any substantial interest in OTEL logging, well, > we > > can live without it. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
