+1 to what Ash, Jarek, and Kaxil said above. Let's chat in late May or June on this topic.
On Tue, Apr 15, 2025 at 7:05 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > Yeah, no strength or bandwidth right now for a discussion. And would very > much prefer we focus on getting 3.0.0 out and any bug fixes after that > > On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote: > > > Very much agree with Ash here. > > > > On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > Hi Howard, > > > > > > Your timing is…. Interesting given we are focusing on getting 3.0 > shipped > > > and are in the final throws of that. > > > > > > Let's revisit this in May or June once the dust has settled from 3.0, > any > > > immediate bug fixes we might need, and give people a little bit of time > > to > > > unwind after the crunch. > > > > > > -ash > > > > > > > On 14 Apr 2025, at 16:22, Howard Yoo <howard...@gmail.com> wrote: > > > > > > > > Hi Airflow community members, > > > > > > > > Apache Airflow has been moving steadily from conventional statsd > based > > > > telemetry into OpenTelemetry (opentelemetry.io), which has gained > > > > popularity and support for the last few years. There has been AIP > > > proposals > > > > which had a 'phased' approach to incorporate > > > > > > > > 1. Metrics > > > > 2. Traces > > > > 3. Logs > > > > > > > > Based on the current availability and maturity of Otel's support. > Since > > > the > > > > metrics emission based on statsd were the fastest low-hanging fruit > > that > > > we > > > > could implement OTEL metrics support relatively easy, which we did as > > our > > > > first try. > > > > > > > > Second was the Traces. OTEL support on traces for Python was very > > mature > > > > and well established. Even though Airflow did not have any prior > > > > implementation on emitting Traces, we were able to instrument it and > > > > produce traces on both Airflow specific processes (jobs) (e.g. > > > schedulers, > > > > triggers, executors), as well as task runs. > > > > > > > > However, the area for Logs were not done yet, due at that time a > > > relatively > > > > early and immature state of logs support in Otel. > > > > > > > > Now, it seems like Logging support on Python has become stable and > > > reliable > > > > that we can use it to emit log messages in Otel format. So, the > timing > > > > seems to be quite perfect for us to start up a discussion on it. > > > > > > > > Logging support in Airflow is basically provided via 'provider' model > > > where > > > > Airflow itself does not have a specific implementation of how logging > > is > > > > done, but requests contributors to provide provider that can plug-in > > > their > > > > logging support. > > > > > > > > Would we want to move to that direction, of having 'opentelemetry' > > > provider > > > > ? > > > > Or do we want to have logging support as more of a native support > way? > > > > > > > > The reason why I would like to start this discussion thread was to > gain > > > > opinions from the community on whether Airflow has the interest to > > > support > > > > OTEL logging first, and then any opinions on specifics of what would > be > > > the > > > > best way to support OTEL logging. > > > > > > > > Please feel free to reply and add any comments to the thread to voice > > > your > > > > opinions. If there isn't any substantial interest in OTEL logging, > > well, > > > we > > > > can live without it. > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
