Hi all, As discussed in the previous email thread, the first dev call has been pushed to next Tuesday (4th June 2024).
If you would like to participate in the development of Airflow 3, please join the dev calls starting next week. The calls will be open to anyone in the community. *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 PM EST | 9 AM PST) *One-time registration Link*: https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG The meeting notes from the call will also be posted on the dev mailing list and Confluence for archival purposes at https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes. *Proposed Agenda*: 1) Agreeing on the Principles to drive Airflow 3 development 2) Agreeing on the Guidelines that help decide if a feature should be in Airflow 3 or not For (1), I propose the following principles: - Considering Airflow 3.0 for early adopters and breaking (and removing) things. Things can be re-added as needed in upcoming minor releases. - Optimize to get foundational pieces in and not "let perfect be the enemy of good" - Smoother migration path between AF 2 & 3 especially for DAG Authors with the existing official Airflow providers. - Working on features that solidify Airflow as the modern Orchestrator that also has state-of-the-art support for Data, AI & ML workloads. - This includes improving scalability & performance of all the Airflow components. - Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability - Set up the codebase for the next 3-5 years. - Reducing matrix of supported combinations for reducing complexity in testing & development. E.g Remove MySQL support to reduce the test matrix - Simplifying codebase & standardize architecture (e.g consolidating serialization methods) - Remove deprecations - Simplify the Learning Curve for new Airflow users - Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10. This should continue for a longer period of time after AF 3 release - Target a shorter cycle to release Airflow 3 - so that Airflow 2 branches for features don't diverge - have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations For (2), I propose the following guidelines: - Alignment with Core Principles - Community Demand and Feedback - Impact on Scalability and Performance - Implementation Complexity and Maintenance - Backward Compatibility and Migration Effort - Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found - For big epics, AIPs & a successful vote on the dev mailing list Please reply if anyone has anything to add to the agenda or comment on anything if you disagree. Looking forward to the call. Regards, Kaxil
