Hey all, Apologies for the delay!
I have updated our meeting notes document to summarize the discussion from our 20th June dev call for Airflow 3.0. Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-20June2024 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree with anything in the Summary, please voice your opinion. I have triaged all the AIPs based on our discussion and updated the AIP template <https://cwiki.apache.org/confluence/pages/templates2/viewpagetemplate.action?entityId=90210323&key=AIRFLOW> to include migration efforts. I will send a separate email for the agenda for the next meeting on 27th June. Regards, Kaxil ------ Including the Summary here too (might break formatting): The team reviewed the existing status of AIPs and marked them for AF 3.0, 3.1+ and AF 2 as needed. Here is the summary: AIP Decision AIP-1 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89066609> Superseded by AIP-44 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API> & AIP-67 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>. Close it AIP-5 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher?src=contextnavpagetreemode> Superseded by AIP-66 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode> AIP-14 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-14+Create+composable+operators+using+common+interfaces?src=contextnavpagetreemode> Mark as *completed* by AIP-58 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565&src=contextnavpagetreemode> AIP-20 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest?src=contextnavpagetreemode> Superseded by AIP-66 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode> AIP-22 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-22%3A+Group+ORM+models+by+their+logical+usage+instead+of+type?src=contextnavpagetreemode> Superseded by AIP-72 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode> due to Task Interface. If someone has time, it can be PRs, doesn't need AIP AIP-30 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-30%3A+State+persistence?src=contextnavpagetreemode> Airflow >= 3.1 AIP-35 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow?src=contextnavpagetreemode> *Abandoned* AIP-38 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application?src=contextnavpagetreemode> Keep it for 3.0. Brent will update this. There was an Ask about expanding the Plugin migration strategy. AIP-44 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API?src=contextnavpagetreemode> Replaced by AIP-72 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode> for AF 3, and we can ship what we have in main for 2.10 as "*experimental*" and stated to change for AF 3 AIP-46 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-46+Runtime+isolation+for+airflow+tasks+and+dag+parsing?src=contextnavpagetreemode> Replaced by AIP-72 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode> and some parts of AIP-67 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode> AIP-49 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow?src=contextnavpagetreemode> Phase 1 is in 2.10 and Phase 2 will be in 3.1 and above unless Howard Yoo <https://cwiki.apache.org/confluence/display/~howardyoo> wants to lead and implement it for AF 3.0 AIP-51 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow?src=contextnavpagetreemode> Marking it for Airflow 2.10. Niko Oliveira <https://cwiki.apache.org/confluence/display/~onikolas> finishing up the last issue. AIP-55 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-55+Rule-based+timetable+with+logical+composition?src=contextnavpagetreemode> *Abandoned* AIP-57 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature?src=contextnavpagetreemode> Airflow 3.0, Shubham owns it. There is a PR <https://github.com/apache/airflow/pull/36639> already with this comment <https://github.com/apache/airflow/pull/36639#issuecomment-2021080626> contesting the proposed solution. We will review this again in few weeks AIP-59 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework?src=contextnavpagetreemode> Bartosz is working on it. Is OOO right now but will resume once back. Will be used in AF 2 & AF 3 to compare performance. Will target it for AF 2 AIP-61 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution?src=contextnavpagetreemode> Marking it for Airflow 2.10. Niko Oliveira <https://cwiki.apache.org/confluence/display/~onikolas> to get it done in the next few weeks AIP-62 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-62+Getting+Lineage+from+Hook+Instrumentation?src=contextnavpagetreemode> Marking it for Airflow 3.0 for now. But we will review it again in a couple of weeks; Maciej Obuchowski <https://cwiki.apache.org/confluence/display/~mobuchowski> owns it. AIP-64 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-64%3A+Keep+TaskInstance+try+history?src=contextnavpagetreemode> Airflow 2.10 AIP-65 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-65%3A+Improve+DAG+history+in+UI?src=contextnavpagetreemode> Airflow 3.0 AIP-66 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode> Airflow 3.0 AIP-67 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode> Airflow 3.0 AIP-68 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels?src=contextnavpagetreemode> Marking it for Airflow 3.0 for now, but it will be re-evaluated when Jens Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl> creates a VOTE thread after discussing with Brent. Concern for keeping this for AF 2.10 was around adding a React Plugin interface that will need changing creating a migration pain. AIP-69 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor?src=contextnavpagetreemode> *Tentatively* marking it for 3.0. Jens Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl> has a PR <https://github.com/apache/airflow/pull/40224> that needs review. The concern for keeping this for AF 2.10 even if most changes are in providers is that it builds on top of AIP-44 which will be experimental and replaced in AF 3. Will be reviewed in a few weeks AIP-70 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-70+Migrating+to+asynchronous+programming?src=contextnavpagetreemode> Airflow 3.1+ AIP-71 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-71+Generalizing+DAG+Loader+and+Processor+for+Ephemeral+Storage?src=contextnavpagetreemode> The general consensus was against this AIP and favouring the DAG bundles concept that is part of AIP-66 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode> . AIP-72 <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode> Airflow 3.0 The team also decided to review the Airflow 3 Workstreams <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams> doc asynchronously. If the reviewers are interested in leading any workstream without an owner, they should add their names in the Owner column. Other Action items: - Review SLA PR <https://github.com/apache/airflow/pull/36639> - Jens Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl> to talk to Brent for AIP-68 - Review PR <https://github.com/apache/airflow/pull/40224> from Jens for AIP-69 (Remote Executor) - Complete AIP-51 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow?src=contextnavpagetreemode> & AIP-61 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution?src=contextnavpagetreemode> for Airflow 2.10 - Workstream owner should Start AIP or draft AIPs for items in *Other Candidates* items (under AF 3.0 section) in Airflow 3 Workstreams <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams>
