Thanks for sparking the discussion anyways, Howard! Even though we do not have bandwidth right now I am glad you started this discussion.
We will revisit this once we are in a better shape with AF3 (may or june :)) Thanks & Regards, Amogh Desai On Wed, Apr 16, 2025 at 3:32 AM Howard Yoo <howard...@gmail.com> wrote: > Hey, absolutely! > Given the needed focus and attention on Airflow 3.0, I am totally okay with > revisiting this topic in May or June! I was just wondering if this was > something that we want to eventually do, when the Airflow enters into the > world of 3.0. > > Regards, > Howard > > On Tue, Apr 15, 2025 at 12:14 PM Ferruzzi, Dennis > <ferru...@amazon.com.invalid> wrote: > > > +1 to what Ash, Jarek, Kaxil, and Vikram said above. > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Vikram Koka <vik...@astronomer.io.INVALID> > > Sent: Tuesday, April 15, 2025 6:57 AM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] [DISCUSSION] Opentelemetry support for LOGS in Apache > > Airflow > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > +1 to what Ash, Jarek, and Kaxil said above. > > > > Let's chat in late May or June on this topic. > > > > On Tue, Apr 15, 2025 at 7:05 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > Yeah, no strength or bandwidth right now for a discussion. And would > very > > > much prefer we focus on getting 3.0.0 out and any bug fixes after that > > > > > > On Tue, 15 Apr 2025 at 14:08, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > Very much agree with Ash here. > > > > > > > > On Tue, Apr 15, 2025 at 10:30 AM Ash Berlin-Taylor <a...@apache.org> > > > wrote: > > > > > > > > > Hi Howard, > > > > > > > > > > Your timing is…. Interesting given we are focusing on getting 3.0 > > > shipped > > > > > and are in the final throws of that. > > > > > > > > > > Let's revisit this in May or June once the dust has settled from > 3.0, > > > any > > > > > immediate bug fixes we might need, and give people a little bit of > > time > > > > to > > > > > unwind after the crunch. > > > > > > > > > > -ash > > > > > > > > > > > On 14 Apr 2025, at 16:22, Howard Yoo <howard...@gmail.com> > wrote: > > > > > > > > > > > > Hi Airflow community members, > > > > > > > > > > > > Apache Airflow has been moving steadily from conventional statsd > > > based > > > > > > telemetry into OpenTelemetry (opentelemetry.io), which has > gained > > > > > > popularity and support for the last few years. There has been AIP > > > > > proposals > > > > > > which had a 'phased' approach to incorporate > > > > > > > > > > > > 1. Metrics > > > > > > 2. Traces > > > > > > 3. Logs > > > > > > > > > > > > Based on the current availability and maturity of Otel's support. > > > Since > > > > > the > > > > > > metrics emission based on statsd were the fastest low-hanging > fruit > > > > that > > > > > we > > > > > > could implement OTEL metrics support relatively easy, which we > did > > as > > > > our > > > > > > first try. > > > > > > > > > > > > Second was the Traces. OTEL support on traces for Python was very > > > > mature > > > > > > and well established. Even though Airflow did not have any prior > > > > > > implementation on emitting Traces, we were able to instrument it > > and > > > > > > produce traces on both Airflow specific processes (jobs) (e.g. > > > > > schedulers, > > > > > > triggers, executors), as well as task runs. > > > > > > > > > > > > However, the area for Logs were not done yet, due at that time a > > > > > relatively > > > > > > early and immature state of logs support in Otel. > > > > > > > > > > > > Now, it seems like Logging support on Python has become stable > and > > > > > reliable > > > > > > that we can use it to emit log messages in Otel format. So, the > > > timing > > > > > > seems to be quite perfect for us to start up a discussion on it. > > > > > > > > > > > > Logging support in Airflow is basically provided via 'provider' > > model > > > > > where > > > > > > Airflow itself does not have a specific implementation of how > > logging > > > > is > > > > > > done, but requests contributors to provide provider that can > > plug-in > > > > > their > > > > > > logging support. > > > > > > > > > > > > Would we want to move to that direction, of having > 'opentelemetry' > > > > > provider > > > > > > ? > > > > > > Or do we want to have logging support as more of a native support > > > way? > > > > > > > > > > > > The reason why I would like to start this discussion thread was > to > > > gain > > > > > > opinions from the community on whether Airflow has the interest > to > > > > > support > > > > > > OTEL logging first, and then any opinions on specifics of what > > would > > > be > > > > > the > > > > > > best way to support OTEL logging. > > > > > > > > > > > > Please feel free to reply and add any comments to the thread to > > voice > > > > > your > > > > > > opinions. If there isn't any substantial interest in OTEL > logging, > > > > well, > > > > > we > > > > > > can live without it. > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > >
