Yes, I would agree. As for the deferred tasks, I believe the Otel should
create 'spans' when the task instances are actually 'running' during their
deferred tasks were running - so even though the task id would be the same,
we would have multiple 'spans' on the course of its lifetime.
On Fri, May
On Sat, May 21, 2022 at 1:22 AM Malthe wrote:
> But isn't the span uniquely identified by the task instance and attempt
> number?
>
> I would think that you've already got all the information without
> persisting any additional data.
>
This doesn't seem quite right for deferred tasks, does it? I
That's a cool idea Leah.
Also John - maybe there is a way we can reach out to Amazon
Open-Telemetry team? I know Amazon is very, very active in this. This
might be a great idea to reach for industry feedback.
I actually also will reach out to my friends in SumoLogic who also are
very actively wor
I agree! :-) looking forward to the Airflow Summit next week!
Howard
On Sun, May 22, 2022 at 10:29 AM Jarek Potiuk wrote:
> I think once we get some of the people who commented (Elad/Malthe)
> confirm that their comments were addressed and maybe some other voices
> of support, it could be ready
I think once we get some of the people who commented (Elad/Malthe)
confirm that their comments were addressed and maybe some other voices
of support, it could be ready for a Voting attempt actually :). I'd
wait however with it till after the Summit (like with few other
discussions we are having now
But isn't the span uniquely identified by the task instance and attempt
number?
--> True, but the thing about OpenTelemetry is that span ID and trace ID
are in UUID format (and has to be), so unless we devise a way to uniquely
create the task instance and attempt number into UUID format (I guess we
On Wed, 18 May 2022 at 16:44, Howard Yoo wrote:
> 2. So, the reason why I ended up implementing span_json was that between the
> scheduler who submits the tasks to be processed, and the worker that needs to
> pick them up from the queue (which is implemented in meta database of
> airflow) - nee
Hi Malthe,
1. You are right - those were merely added to those 'counter' metrics to
see if populating helpful attributes alongside of them would be feasible
and also helpful or not - and does not represent the final form of the
implementation. I would hope the actual implementation of this proposa
On Fri, 25 Mar 2022 at 20:40, Howard Yoo wrote:
> I am pleased to announce the start of the discussion for the new AIP draft
> that was recently been published:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow
Some comments:
1. The optional
Finally had some time to go over the changes and wrote summary of it in AIP
for better understanding, and updated the proposal doc:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow#AIP49OpenTelemetrySupportforApacheAirflow-CodeChangesSummary
Hope
Got it. Will describe the changes that might be subjected to be done -
however, please note that the changes were the things that I did on my own
during the POC, so the final changes may obviously be different.
On Fri, Apr 29, 2022 at 3:57 AM Elad Kalif wrote:
> I'd prefer that the Code Changes
I'd prefer that the Code Changes section would be more detailed. This is a
very important part.
I'm not sure referring to a private repo is the way to do it but even if
you should at least list the components which are subject to changes.
You don't need to specify every change you are going to do -
Hi Elad, I have updated the AIP-49 with the appropriate changes to contain
what you requested in your previous comments. I think the AIP is fairly
comprehensive and ready to be voted, unless there is any objections.
Sincerely,
Howard
On Mon, Apr 25, 2022 at 11:03 PM Howard Yoo wrote:
> Hi Elad,
Hi Elad,
Will take a look at it and let you know!
Thanks,
Howard
On Mon, Apr 25, 2022 at 6:35 AM Elad Kalif wrote:
> Hi Howard,
> Have you made progress with addressing the comments?
> I think once points are addressed we can maybe start a vote?
>
> On Tue, Apr 12, 2022 at 12:39 PM Jarek Potiuk
Hi Howard,
Have you made progress with addressing the comments?
I think once points are addressed we can maybe start a vote?
On Tue, Apr 12, 2022 at 12:39 PM Jarek Potiuk wrote:
> > 1. I would assume the path for StatsD (dogstatsd) would be for
> deprecation - we will perhaps comment or mark it
> 1. I would assume the path for StatsD (dogstatsd) would be for deprecation -
> we will perhaps comment or mark it as deprecation - and should follow the
> established (or usual?) process of feature deprecation of Airflow, once the
> opentelemetry is in place.
Yep. Deprecation should be. And m
Thank you, Elad!
Yeah, it might have been TL;DR; moment on the PDF, but the description of
the work required such amount, unfortunately (thanks for bearing with it!).
I may not have all the answers, but do have some opinions on your 6
questions/points, so if I may add a little of them:
1. I would
Thanks Howard!
Sorry for the delay, this was a long read. The PDF alone is 19 pages :)
looks very good!
I have 6 questions/points to raise:
1. I'm not clear about what is to happen with StatsD .
It states "Make OpenTelemetry and StatsD optional and interchangeable."
But do we want to support bo
Or maybe that people are so stunned by the beauty and usefulness of it
that they cannot even say a word :)
On Thu, Mar 31, 2022 at 11:37 PM Howard Yoo wrote:
>
> I think it may mean all is well, perhaps :-)
> Howard
>
> On Thu, Mar 31, 2022 at 9:29 AM Jarek Potiuk wrote:
>>
>> Hello everyone,
>>
I think it may mean all is well, perhaps :-)
Howard
On Thu, Mar 31, 2022 at 9:29 AM Jarek Potiuk wrote:
> Hello everyone,
>
> Would be great to get some comments and reviews - especially from
> those who are users and are doing monitoring. Howard made a lot of
> effort to show examples of how O
Hello everyone,
Would be great to get some comments and reviews - especially from
those who are users and are doing monitoring. Howard made a lot of
effort to show examples of how OTEL might help in this.
Otherwise, is the silence sign that all is good ?
J.
On Fri, Mar 25, 2022 at 9:53 PM Jare
And let me add to it - we laid some foundations for it with Melodie
Ezeani - the Outreachy intern where we did some internal integration
work that let us understand the challenges and state of
open-telemetry.
I am super excited about what open-telemetry can bring to Airflow -
both short term (in m
Hi all,
I am pleased to announce the start of the discussion for the new AIP draft
that was recently been published:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow
Jarek Potuik and I have been discussing about this proposal since early
this ye
23 matches
Mail list logo
