Re: Request for feedback on proposal for new OpenLineage provider in Airflow

And here is the recording: https://youtu.be/fAqvoMzz7Tk On Fri, Mar 31, 2023 at 1:51 PM Julien Le Dem wrote: > Thank you all who attended. > Here are the slides we presented today: > > https://docs.google.com/presentation/d/1o8VnXHXME_Vf-eQpQ5qvC7fb951CjBHmrsCsLW1Z1_8/edit?usp=sharing > I'll als

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Thank you all who attended. Here are the slides we presented today: https://docs.google.com/presentation/d/1o8VnXHXME_Vf-eQpQ5qvC7fb951CjBHmrsCsLW1Z1_8/edit?usp=sharing I'll also post the recording once available. Julien On Fri, Mar 24, 2023 at 2:40 AM Jarek Potiuk wrote: > Added :) > > On Fri,

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Added :) On Fri, Mar 24, 2023 at 4:30 AM Bowrna Prabhakaran wrote: > > Can I get added to the invitation as well? (mailbow...@gmail.com) > Thanks > > On Fri, Mar 24, 2023 at 2:37 AM Jarek Potiuk wrote: > > > did > > > > On Thu, Mar 23, 2023 at 9:22 PM c c wrote: > > > > > > Can I be added to th

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Can I get added to the invitation as well? (mailbow...@gmail.com) Thanks On Fri, Mar 24, 2023 at 2:37 AM Jarek Potiuk wrote: > did > > On Thu, Mar 23, 2023 at 9:22 PM c c wrote: > > > > Can I be added to the invitation as well(changcheng12...@gmail.com)? > > thanks! > > > > On Thu, Mar 23, 2023

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

did On Thu, Mar 23, 2023 at 9:22 PM c c wrote: > > Can I be added to the invitation as well(changcheng12...@gmail.com)? > thanks! > > On Thu, Mar 23, 2023 at 12:59 PM Jarek Potiuk wrote: > > > I added all those who asked. It's really cool we have so much interest :). > > > > Julien, Maciej: NO P

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Can I be added to the invitation as well(changcheng12...@gmail.com)? thanks! On Thu, Mar 23, 2023 at 12:59 PM Jarek Potiuk wrote: > I added all those who asked. It's really cool we have so much interest :). > > Julien, Maciej: NO PRESSURE > > > ---

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

I added all those who asked. It's really cool we have so much interest :). Julien, Maciej: NO PRESSURE - To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

airflow.apache.org > Subject: RE: [EXTERNAL]Request for feedback on proposal for new > OpenLineage provider in Airflow > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

I'd like to join as well! (oliveira...@gmail.com) From: Igor Kholopov Sent: Wednesday, March 22, 2023 4:01:40 PM To: dev@airflow.apache.org Subject: RE: [EXTERNAL]Request for feedback on proposal for new OpenLineage provider in Airflow CAUTION: This

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

+1, would be happy to join the session! (Please add either ikholo...@google.com or kholopo...@gmail.com). Best, Igor On Wed, Mar 22, 2023 at 11:27 PM Pierre Jeambrun wrote: > Same here if you can add me please. > > Looking forward to this session. > > Le mer. 22 mars 2023 à 23:07, Mehta, Shubha

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Same here if you can add me please. Looking forward to this session. Le mer. 22 mars 2023 à 23:07, Mehta, Shubham a écrit : > Please include me, I will try my best to join (shubhammehta...@gmail.com) > > Best, > Shubham > > On 2023-03-22, 2:24 PM, "Jarek Potiuk" ja...@potiuk.com>> wrote: > >

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Please include me, I will try my best to join (shubhammehta...@gmail.com) Best, Shubham On 2023-03-22, 2:24 PM, "Jarek Potiuk" mailto:ja...@potiuk.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the se

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

There are some strange behaviours in the calendar entry - I think you cannot add yourself, only guests can add others :) I've added you Eugen, maybe if someone wants to be also added - please post here with your gmail/calendar addresses. J. On Wed, Mar 22, 2023 at 9:56 PM Eugen Kosteev wrote: >

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Hi Julien. Can you, please, include me there as well: eu...@kosteev.com or kost...@google.com. Looking forward to see presentation. - Eugene On Wed, Mar 22, 2023 at 8:36 PM Julien Le Dem wrote: > Hello all, > I have to move the OpenLineage presentation to next week. > Sorry for the change. > I

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Hello all, I have to move the OpenLineage presentation to next week. Sorry for the change. It will be Friday next week March 31st at 5pm CET 9am PT. https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MTF1bHRrdTdrM29vMGZyamdzc2JuZWFkMHEganVsaWVuQGFzdHJvbm9tZXIuaW8&tmsrc=julien%40astron

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

We are planning to do this session next Thursday at 5pm CET 9am PT. I will send a zoom link in advance. Julien On Sat, Feb 25, 2023 at 05:59 Jarek Potiuk wrote: > Cool. I am looking forward to it :). It would be great to get some > insight from those who attempted to get the lineage working in s

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Cool. I am looking forward to it :). It would be great to get some insight from those who attempted to get the lineage working in several versions of Open Lineage and finally arrived at the current specs/integration. On Wed, Feb 22, 2023 at 7:02 PM Julien Le Dem wrote: > > Thank you Jarek, > I am

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Thank you Jarek, I am happy to organize a zoom presentation about OpenLineage and answer any question. It is indeed a spec decoupling the data transformation layer from the Metadata store people are using. Just like OpenTelemetry is for service metrics/traces. Best, Julien

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

And to add a little "parallel" - I think Open Lineage integration replacing our "generic lineage" is very similar step to the new "Multi-tenant"-ready authentication interface we are discussing in https://lists.apache.org/thread/cc9dj680nwz494k8n51w6qqohzm4wgck Yes - we have a generic authenticati

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Hey Rafał (Eugene, Michal - and others who are looking), I think I know where your/Eugen/Michał concerns are coming from. And I think it would be great if we can talk it over a bit. I believe this is - in parts - quite a misunderstanding of what Open Lineage really is, how much of an integration

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Hi, I second/echo the input provided by Eugene and Michal. In general, Airflow should provide generic interfaces to lineage backends so it's easy to configure the one preferred by the user. Whether it's Open Lineage, proprietary solution, Dataplex Lineage, etc. it should be the user's choice. We

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Dear Airflow community, I have transferred the content of the working google doc I shared a few weeks ago to the Airflow confluence: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-53+OpenLineage+in+Airflow All comments have been answered, I added clarifications to the doc accordingly and I

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Thank you for the email Jarek, and Eugene for your suggestions, I do agree with Jarek's assessment. I don't have very much to add to his argument, it is very thoughtful! OpenLineage was started to avoid the cartesian complexity that Eugene mentions. There's actually that specific illustration in th

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Just a quick personal view on it, Eugene (I bet Julian's answer will be more thoughtful). I think you are right to the "agnostic" part. But I have one question - what are we considering "agnostic"? There is no "widespread" standard for lineage (yet). Open Lineage with its donation to Linux Found

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Hi Julien. I reviewed the design doc. The general idea looks good to me, but I have some concerns that I would like to share. If I understand correctly the proposed design is to fill in "operators" with self-methods to extract lineage metadata from it, and I agree with the motivation. If those ar

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Hello Michał, Thank you for your input. I would clarify that OpenLineage doesn't make any assumption about the backend being used to store lineage and is an adapter-like layer. OpenLineage exists as the spec specifically for that purpose of avoiding the problem of every lineage consumer having to u

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Hi everyone, As Airflow already supports lineage functionality through pluggable lineage backends, I think OpenLineage and other lineage systems integration should follow this path. I think more 'native' integration with OpenLineage (or any other lineage system) in Airflow while maintaining the ge

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Thank you Eugen, This sounds very aligned with the goals of OpenLineage and I think this would work well. Here are the sections in the doc that I think address your points: *- generalize lineage metadata extraction as self-method in each operator, using generic lineage entities* See: OpenLineage su

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

++ Michal Modras On Tue, Jan 31, 2023 at 3:49 PM Eugen Kosteev wrote: > Cloud Composer recently launched "Data lineage with Dataplex" feature > which effectively means to generate lineage out of DAG/task executions and > export it to Data Lineage (Data Catalog service) for further analysis. > ht

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Cloud Composer recently launched "Data lineage with Dataplex" feature which effectively means to generate lineage out of DAG/task executions and export it to Data Lineage (Data Catalog service) for further analysis. https://cloud.google.com/composer/docs/composer-2/lineage-integration This feature

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

Thank you very much for your input Jarek. I am responding in the comments and adding to the doc accordingly. I would also love to hear from more stakeholders. Thanks to all who provided feedback so far. Julien On Fri, Jan 27, 2023 at 12:57 AM Jarek Potiuk wrote: > General comment from my side: I

Re: Request for feedback on proposal for new OpenLineage provider in Airflow

General comment from my side: I think Open Lineage is (and should be even more) a feature of Airflow that expands Airflow's capabilities greatly and opens up the direction we've been all working on - Airflow as a Platform. I think closely integrating it with Open-Lineage goes the same direction (a

Request for feedback on proposal for new OpenLineage provider in Airflow

Dear Airflow Community, I have been working on a proposal to bring an OpenLineage provider to Airflow . I am looking for feedback with the goal to post an official AIP. Please feel free to comment in the doc abo