Hey all, I have updated our meeting notes document to summarize the discussion from our last dev call.
Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-19September2024 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree with anything in the Summary, please voice your opinion. I will send a separate email with the agenda for the next meeting but if someone has an item for the agenda, please reach out to me or feel free to add it to the doc <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda> . Regards, Kaxil ------ Including the Summary here too (might break formatting): *Catch-up on action items from last call* - Kaxil Naik showed the GitHub project boards <https://github.com/apache/airflow/projects?query=is%3Aopen> for some of the existing AIPs. Kaxil will follow up with all other AIP-leads to have the project board created. - Kaxil Naik discussed a proposal for how the intermediate milestones would be created to track progress. Kaxil will follow up with a draft with Vikram & Rajesh and then follow up with all the AIP leads to have the milestones defined. The general agreement was to have 3 intermediate milestones: Week of 24 Oct, 18 Nov and 16 Dec - Vikram Koka confirmed that Astronomer did not end up using Casbin itself but just the interfaces. Shubham Mehta mentioned Niko Oliveira is going to explore Casbin and Keycloak and start implementation in about a week. The Dex project <https://dexidp.io/> was suggested as yet another option by Mathew Wicks at the Airflow Summit. This was used by them in the Kubeflow project. His suggestion was to utilize it for Authentication only and continue doing Authorization on the Airflow side. Shubham will reach out to him. *Discussions* - Split into Operators / Executors / Plugins (Jens) - The goal of the discussion was to split the providers into Executors, plugins, and what's needed for DAG Authors (i.e User Code) like Operators and hooks. This way, we can limit the Python dependencies needed for Webserver, Scheduler, Worker, etc. An example is an AWS provider that has an Executor as well as an Operator. If we split them, then Scheduler can use the Provider that has an Executor, and Worker (including Triggerer & DAG Parser) can use the AWS provider that has an Operator but not an Executor. Airflow 3 is the right time to split this out if we decide to go that way. - re External Operator Links: Some ideas were discussed, like the parser storing the links in the DB, which is then retrieved by the Webserver via an API, so the Webserver does not need to install a provider just for Operator Links. - Jens Scheffler will redraw the above diagram to separate out the Scheduler and the DAG Parser as the Scheduler does not run User Code, but the Parser does. As a follow-up to it, Jens will have a proposal around what the separation of providers would like to discuss on the next call and/or the dev mailing list. Vikram Koka and Kaxil Naik have volunteered to work with Jens on it offline before the next call. - Airflow Standard Provider: Bash <https://github.com/apache/airflow/pull/42252> & Python <https://github.com/apache/airflow/pull/42081> Operator (Pavan Kumar) - Pavan brought up the concern that System tests will break if we change the example DAGs for Bash & Python from core to using Standard provider until it is released. This was because some of providers like Google use the Bash or Python Operators. - Ideally, it should work with Airflow 3 since the provider will be pre-installed (via required_deps) with Airflow core/SDK. - The team decided to take it offline. Reviews would be appreciated. - Potentially moving dev calls on Oct 3 to Oct 9 or Oct 10. And one on Oct 31 [*Diwali*] to Nov 1 (Kaxil Naik ) - Decided to move 03 Oct 2024 call to10 Oct 2024 and move the 01 Nov 2024 to 07 Nov 2024
