Thanks Vikram, appreciate it. On Tue, 25 Jun 2024 at 21:16, Vikram Koka <[email protected]> wrote:
> Thanks Kaxil, I really appreciate the diligent follow up here. > Both the preparation and follow through is excellent! > > Vikram > > > On Mon, Jun 24, 2024 at 5:09 PM Kaxil Naik <[email protected]> wrote: > > > Hey all, > > > > Apologies for the delay! > > > > I have updated our meeting notes document to summarize the discussion > > from our 20th > > June dev call for Airflow 3.0. > > > > Link: > > > > > https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-20June2024 > > > > To all those who attended, can you please double-check and add if > > I have missed anything? > > > > To all those who didn't join, if you disagree with anything in the > Summary, > > please voice your opinion. > > > > I have triaged all the AIPs based on our discussion and updated the AIP > > template > > < > > > https://cwiki.apache.org/confluence/pages/templates2/viewpagetemplate.action?entityId=90210323&key=AIRFLOW > > > > > to > > include migration efforts. > > > > I will send a separate email for the agenda for the next meeting on > > 27th June. > > > > Regards, > > Kaxil > > > > ------ > > > > Including the Summary here too (might break formatting): > > > > The team reviewed the existing status of AIPs and marked them for AF 3.0, > > 3.1+ and AF 2 as needed. Here is the summary: > > > > AIP > > Decision > > > > > > AIP-1 > > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89066609 > > > > > Superseded > > by AIP-44 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API > > > > > & AIP-67 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components > > >. > > Close it > > AIP-5 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher?src=contextnavpagetreemode > > > > > Superseded > > by AIP-66 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > > > AIP-14 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-14+Create+composable+operators+using+common+interfaces?src=contextnavpagetreemode > > > > > Mark > > as *completed* by AIP-58 > > < > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565&src=contextnavpagetreemode > > > > > AIP-20 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest?src=contextnavpagetreemode > > > > > Superseded > > by AIP-66 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > > > AIP-22 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-22%3A+Group+ORM+models+by+their+logical+usage+instead+of+type?src=contextnavpagetreemode > > > > > Superseded > > by AIP-72 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > > > due > > to Task Interface. If someone has time, it can be PRs, doesn't need AIP > > AIP-30 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-30%3A+State+persistence?src=contextnavpagetreemode > > > > > Airflow > > >= 3.1 > > AIP-35 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow?src=contextnavpagetreemode > > > > > *Abandoned* > > AIP-38 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application?src=contextnavpagetreemode > > > > > Keep > > it for 3.0. Brent will update this. There was an Ask about expanding the > > Plugin migration strategy. > > AIP-44 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API?src=contextnavpagetreemode > > > > > Replaced > > by AIP-72 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > > > for > > AF 3, and we can ship what we have in main for 2.10 as "*experimental*" > and > > stated to change for AF 3 > > AIP-46 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-46+Runtime+isolation+for+airflow+tasks+and+dag+parsing?src=contextnavpagetreemode > > > > > Replaced > > by AIP-72 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > > > and > > some parts of AIP-67 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode > > > > > AIP-49 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow?src=contextnavpagetreemode > > > > > > > Phase 1 is in 2.10 and Phase 2 will be in 3.1 and above unless Howard Yoo > > <https://cwiki.apache.org/confluence/display/~howardyoo> wants to lead > > and > > implement it for AF 3.0 > > AIP-51 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow?src=contextnavpagetreemode > > > > > > > Marking it for Airflow 2.10. Niko Oliveira > > <https://cwiki.apache.org/confluence/display/~onikolas> finishing up the > > last issue. > > AIP-55 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-55+Rule-based+timetable+with+logical+composition?src=contextnavpagetreemode > > > > > *Abandoned* > > AIP-57 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature?src=contextnavpagetreemode > > > > > > > Airflow 3.0, Shubham owns it. There is a PR > > <https://github.com/apache/airflow/pull/36639> already with this comment > > <https://github.com/apache/airflow/pull/36639#issuecomment-2021080626> > > contesting > > the proposed solution. > > > > We will review this again in few weeks > > AIP-59 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework?src=contextnavpagetreemode > > > > > Bartosz > > is working on it. Is OOO right now but will resume once back. Will be > used > > in AF 2 & AF 3 to compare performance. Will target it for AF 2 > > AIP-61 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution?src=contextnavpagetreemode > > > > > > > Marking it for Airflow 2.10. Niko Oliveira > > <https://cwiki.apache.org/confluence/display/~onikolas> to get it done > in > > the next few weeks > > AIP-62 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-62+Getting+Lineage+from+Hook+Instrumentation?src=contextnavpagetreemode > > > > > > > Marking it for Airflow 3.0 for now. But we will review it again in a > couple > > of weeks; Maciej Obuchowski > > <https://cwiki.apache.org/confluence/display/~mobuchowski> owns it. > > AIP-64 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-64%3A+Keep+TaskInstance+try+history?src=contextnavpagetreemode > > > > > Airflow > > 2.10 > > AIP-65 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-65%3A+Improve+DAG+history+in+UI?src=contextnavpagetreemode > > > > > Airflow > > 3.0 > > AIP-66 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > > > Airflow > > 3.0 > > AIP-67 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components?src=contextnavpagetreemode > > > > > Airflow > > 3.0 > > AIP-68 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-68+Extended+Plugin+Interface+for+Custom+Grid+View+Panels?src=contextnavpagetreemode > > > > > > > Marking it for Airflow 3.0 for now, but it will be re-evaluated when Jens > > Scheffler <https://cwiki.apache.org/confluence/display/~jscheffl> > creates > > a > > VOTE thread after discussing with Brent. > > > > Concern for keeping this for AF 2.10 was around adding a React Plugin > > interface that will need changing creating a migration pain. > > AIP-69 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor?src=contextnavpagetreemode > > > > > > > *Tentatively* marking it for 3.0. Jens Scheffler > > <https://cwiki.apache.org/confluence/display/~jscheffl> has a PR > > <https://github.com/apache/airflow/pull/40224> that needs review. > > The concern for keeping this for AF 2.10 even if most changes are in > > providers is that it builds on top of AIP-44 which will be experimental > and > > replaced in AF 3. > > > > Will be reviewed in a few weeks > > AIP-70 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-70+Migrating+to+asynchronous+programming?src=contextnavpagetreemode > > > > > Airflow > > 3.1+ > > AIP-71 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-71+Generalizing+DAG+Loader+and+Processor+for+Ephemeral+Storage?src=contextnavpagetreemode > > > > > The > > general consensus was against this AIP and favouring the DAG bundles > > concept that is part of AIP-66 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-66%3A+Execution+of+specific+DAG+versions?src=contextnavpagetreemode > > > > > . > > AIP-72 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-72+Task+Execution+Interface+aka+Task+SDK?src=contextnavpagetreemode > > > > > Airflow > > 3.0 > > The team also decided to review the Airflow 3 Workstreams > > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams> > > doc > > asynchronously. If the reviewers are interested in leading any workstream > > without an owner, they should add their names in the Owner column. > > > > Other Action items: > > > > - Review SLA PR <https://github.com/apache/airflow/pull/36639> > > - Jens Scheffler < > https://cwiki.apache.org/confluence/display/~jscheffl> > > to > > talk to Brent for AIP-68 > > - Review PR <https://github.com/apache/airflow/pull/40224> from Jens > > for > > AIP-69 (Remote Executor) > > - Complete AIP-51 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow?src=contextnavpagetreemode > > > > > & AIP-61 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-61+Hybrid+Execution?src=contextnavpagetreemode > > > > > for > > Airflow 2.10 > > - Workstream owner should Start AIP or draft AIPs for items in *Other > > Candidates* items (under AF 3.0 section) in Airflow 3 Workstreams > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams> > > >
