Thanks Kaxil, I really appreciate the diligent follow up here. Both the preparation and follow through is excellent!
Vikram On Mon, Jun 24, 2024 at 5:09 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > Hey all, > > Apologies for the delay! > > I have updated our meeting notes document to summarize the discussion > from our 20th > June dev call for Airflow 3.0. > > Link: > > https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-20June2024 > > To all those who attended, can you please double-check and add if > I have missed anything? > > To all those who didn't join, if you disagree with anything in the Summary, > please voice your opinion. > > I have triaged all the AIPs based on our discussion and updated the AIP > template > < > https://cwiki.apache.org/confluence/pages/templates2/viewpagetemplate.action?entityId=90210323&key=AIRFLOW > > > to > include migration efforts. > > I will send a separate email for the agenda for the next meeting on > 27th June. > > Regards, > Kaxil > > ------ > > Including the Summary here too (might break formatting): > > The team reviewed the existing status of AIPs and marked them for AF 3.0, > 3.1+ and AF 2 as needed. Here is the summary: > > AIP > Decision > > > AIP-1 > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89066609 > > > Superseded > by AIP-44 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API > > > & AIP-67 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components > >. > Close it > AIP-5 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher?src=contextnavpagetreemode > > > Superseded > by AIP-66 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > AIP-14 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-14+Create+composable+operators+using+common+interfaces?src=contextnavpagetreemode > > > Mark > as *completed* by AIP-58 > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565&src=contextnavpagetreemode > > > AIP-20 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest?src=contextnavpagetreemode > > > Superseded > by AIP-66 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > AIP-22 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-22%3A+Group+ORM+models+by+their+logical+usage+instead+of+type?src=contextnavpagetreemode > > > Superseded > by AIP-72 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > due > to Task Interface. If someone has time, it can be PRs, doesn't need AIP > AIP-30 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-30%3A+State+persistence?src=contextnavpagetreemode > > > Airflow > >= 3.1 > AIP-35 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow?src=contextnavpagetreemode > > > *Abandoned* > AIP-38 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application?src=contextnavpagetreemode > > > Keep > it for 3.0. Brent will update this. There was an Ask about expanding the > Plugin migration strategy. > AIP-44 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API?src=contextnavpagetreemode > > > Replaced > by AIP-72 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > for > AF 3, and we can ship what we have in main for 2.10 as "*experimental*" and > stated to change for AF 3 > AIP-46 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-46+Runtime+isolation+for+airflow+tasks+and+dag+parsing?src=contextnavpagetreemode > > > Replaced > by AIP-72 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > and > some parts of AIP-67 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode > > > AIP-49 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow?src=contextnavpagetreemode > > > > Phase 1 is in 2.10 and Phase 2 will be in 3.1 and above unless Howard Yoo > <https://cwiki.apache.org/confluence/display/~howardyoo> wants to lead > and > implement it for AF 3.0 > AIP-51 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow?src=contextnavpagetreemode > > > > Marking it for Airflow 2.10. Niko Oliveira > <https://cwiki.apache.org/confluence/display/~onikolas> finishing up the > last issue. > AIP-55 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-55+Rule-based+timetable+with+logical+composition?src=contextnavpagetreemode > > > *Abandoned* > AIP-57 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature?src=contextnavpagetreemode > > > > Airflow 3.0, Shubham owns it. There is a PR > <https://github.com/apache/airflow/pull/36639> already with this comment > <https://github.com/apache/airflow/pull/36639#issuecomment-2021080626> > contesting > the proposed solution. > > We will review this again in few weeks > AIP-59 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework?src=contextnavpagetreemode > > > Bartosz > is working on it. Is OOO right now but will resume once back. Will be used > in AF 2 & AF 3 to compare performance. Will target it for AF 2 > AIP-61 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution?src=contextnavpagetreemode > > > > Marking it for Airflow 2.10. Niko Oliveira > <https://cwiki.apache.org/confluence/display/~onikolas> to get it done in > the next few weeks > AIP-62 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-62+Getting+Lineage+from+Hook+Instrumentation?src=contextnavpagetreemode > > > > Marking it for Airflow 3.0 for now. But we will review it again in a couple > of weeks; Maciej Obuchowski > <https://cwiki.apache.org/confluence/display/~mobuchowski> owns it. > AIP-64 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-64%3A+Keep+TaskInstance+try+history?src=contextnavpagetreemode > > > Airflow > 2.10 > AIP-65 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-65%3A+Improve+DAG+history+in+UI?src=contextnavpagetreemode > > > Airflow > 3.0 > AIP-66 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > Airflow > 3.0 > AIP-67 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode > > > Airflow > 3.0 > AIP-68 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels?src=contextnavpagetreemode > > > > Marking it for Airflow 3.0 for now, but it will be re-evaluated when Jens > Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl> creates > a > VOTE thread after discussing with Brent. > > Concern for keeping this for AF 2.10 was around adding a React Plugin > interface that will need changing creating a migration pain. > AIP-69 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor?src=contextnavpagetreemode > > > > *Tentatively* marking it for 3.0. Jens Scheffler > <https://cwiki.apache.org/confluence/display/~jscheffl> has a PR > <https://github.com/apache/airflow/pull/40224> that needs review. > The concern for keeping this for AF 2.10 even if most changes are in > providers is that it builds on top of AIP-44 which will be experimental and > replaced in AF 3. > > Will be reviewed in a few weeks > AIP-70 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-70+Migrating+to+asynchronous+programming?src=contextnavpagetreemode > > > Airflow > 3.1+ > AIP-71 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-71+Generalizing+DAG+Loader+and+Processor+for+Ephemeral+Storage?src=contextnavpagetreemode > > > The > general consensus was against this AIP and favouring the DAG bundles > concept that is part of AIP-66 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > . > AIP-72 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > Airflow > 3.0 > The team also decided to review the Airflow 3 Workstreams > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams> > doc > asynchronously. If the reviewers are interested in leading any workstream > without an owner, they should add their names in the Owner column. > > Other Action items: > > - Review SLA PR <https://github.com/apache/airflow/pull/36639> > - Jens Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl> > to > talk to Brent for AIP-68 > - Review PR <https://github.com/apache/airflow/pull/40224> from Jens > for > AIP-69 (Remote Executor) > - Complete AIP-51 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow?src=contextnavpagetreemode > > > & AIP-61 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution?src=contextnavpagetreemode > > > for > Airflow 2.10 > - Workstream owner should Start AIP or draft AIPs for items in *Other > Candidates* items (under AF 3.0 section) in Airflow 3 Workstreams > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams> >
