Thanks Howard! Sorry for the delay, this was a long read. The PDF alone is 19 pages :) looks very good!
I have 6 questions/points to raise: 1. I'm not clear about what is to happen with StatsD . It states "Make OpenTelemetry and StatsD optional and interchangeable." But do we want to support both in the long run? It doesn't specify if we are deprecating statsD along with completion of this AIP. 2. regarding adding metrics. Do we intend to let users define their own KPIs/metrics to be measured or it will be a closed list set by Airflow? 3. The POC specifies it uses a feature of open-telemetry (add metrics) which was not yet released. Do we know the timeline for the feature to be released? Can we vote on a plan to use something which is not yet publicly available? 4. Are the Grafana dashboards / other dashboards to be part of the Airflow core code base? I wonder if this should be in a dedicated repo? 5. Could you please clarify in the AIP page what changes are required to the Airflow code base? Some of them appear in the PDF but I'm not sure if that is all of them? 6. The PDF has several open questions/ideas. I think it would be best to add to the AIP a scope paragraph listing what will be included in the first phase and what is left for other phases. On Fri, Apr 1, 2022 at 4:26 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Or maybe that people are so stunned by the beauty and usefulness of it > that they cannot even say a word :) > > On Thu, Mar 31, 2022 at 11:37 PM Howard Yoo <howard...@gmail.com> wrote: > > > > I think it may mean all is well, perhaps :-) > > Howard > > > > On Thu, Mar 31, 2022 at 9:29 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >> Hello everyone, > >> > >> Would be great to get some comments and reviews - especially from > >> those who are users and are doing monitoring. Howard made a lot of > >> effort to show examples of how OTEL might help in this. > >> > >> Otherwise, is the silence sign that all is good ? > >> > >> J. > >> > >> On Fri, Mar 25, 2022 at 9:53 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > > >> > And let me add to it - we laid some foundations for it with Melodie > >> > Ezeani - the Outreachy intern where we did some internal integration > >> > work that let us understand the challenges and state of > >> > open-telemetry. > >> > > >> > I am super excited about what open-telemetry can bring to Airflow - > >> > both short term (in metrics and instrumentation) and longer term - in > >> > logging when logging is mature enough. > >> > > >> > The open-telemetry mov is at the heart of the principles we have - > >> > making Airflow "modern" but at the same time delegating what's not > >> > "core" to those who can do it better. Open Telemetry is precisely > >> > about that, > >> > > >> > Howard particularly brought a great experience from using Open > >> > Telemetry before and making some good judgements and POC on the > >> > metrics produced by Airflow and how they can be useful from the > >> > "maintainer value" side. I looked at it from the technical integration > >> > POV - and Melodie helped to validate some of the assumptions and > >> > expose some of the technical challenges. > >> > The composite result is good, but we are looking with Howard on some > >> > insightful comments and critique - especially from the users of > >> > Airflow! > >> > > >> > I look forward to more cool stuff on Airflow! > >> > > >> > J. > >> > > >> > On Fri, Mar 25, 2022 at 9:39 PM Howard Yoo <howard...@gmail.com> > wrote: > >> > > > >> > > Hi all, > >> > > > >> > > I am pleased to announce the start of the discussion for the new > AIP draft that was recently been published: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow > >> > > > >> > > Jarek Potuik and I have been discussing about this proposal since > early this year. During that time, we worked together on drafting this > proposal, as well as doing another round of mini-POC to refresh and test on > feasibility of OpenTelemetry on Apache Airflow. > >> > > > >> > > As the POC was successful in terms of testing the OpenTelemetry on > Airflow, we would like to expand the discussion to a wider user community > here this mailing list to gather more consensus, comments, and feedbacks > regarding this AIP. > >> > > > >> > > Sincerely, > >> > > Howard and Jarek >
