Kaxil - thank you for creating the wiki, setting up call invites, and starting this thread. The first draft of the principles looks great. I have commented on the wiki with some feedback and personal thoughts. I'm not adding the comments on this thread to keep the wiki as the single place for discussions and feedback.
Thanks Shubham On 2024-05-28, 12:25 PM, "Kaxil Naik" <kaxiln...@apache.org <mailto:kaxiln...@apache.org>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. If the formatting of bullets was lost, check below or at https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024> Proposed Agenda: 1) Agreeing on the Principles to drive Airflow 3 development 2) Agreeing on the Guidelines that help decide if a feature should be in Airflow 3 or not For (1), I propose the following principles: - Considering Airflow 3.0 for early adopters and breaking (and removing) things. Things can be re-added as needed in upcoming minor releases. - Optimize to get foundational pieces in and not "let perfect be the enemy of good" - Smoother migration path between AF 2 & 3 especially for DAG Authors with the existing official Airflow providers. - Working on features that solidify Airflow as the modern Orchestrator that also has state-of-the-art support for Data, AI & ML workloads. - This includes improving scalability & performance of all the Airflow components. - Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability - Set up the codebase for the next 3-5 years. - Reducing matrix of supported combinations for reducing complexity in testing & development. E.g Remove MySQL support to reduce the test matrix - Simplifying codebase & standardize architecture (e.g consolidating serialization methods) - Remove deprecations - Simplify the Learning Curve for new Airflow users - Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10. This should continue for a longer period of time after AF 3 release - Target a shorter cycle to release Airflow 3 - so that Airflow 2 branches for features don't diverge - have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations For (2), I propose the following guidelines: - Alignment with Core Principles - Community Demand and Feedback - Impact on Scalability and Performance - Implementation Complexity and Maintenance - Backward Compatibility and Migration Effort - Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found - For big epics, AIPs & a successful vote on the dev mailing list Please reply if anyone has anything to add to the agenda or comment on anything if you disagree. Looking forward to the call. Regards, Kaxil On 2024/05/28 19:11:14 Kaxil Naik wrote: > Hi all, > > As discussed in the previous email thread, the first dev call has been > pushed to next Tuesday (4th June 2024). > > If you would like to participate in the development of Airflow 3, please > join the dev calls starting next week. The calls will be open to anyone in > the community. > > *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 PM > EST | 9 AM PST) > *One-time registration Link*: > https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG > > <https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG> > > The meeting notes from the call will also be posted on the dev mailing list > and Confluence for archival purposes > at > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes>. > > > > *Proposed Agenda*: > 1) Agreeing on the Principles to drive Airflow 3 development > 2) Agreeing on the Guidelines that help decide if a feature should be in > Airflow 3 or not > > > For (1), I propose the following principles: > > - Considering Airflow 3.0 for early adopters and breaking (and > removing) things. Things can be re-added as needed in upcoming minor > releases. > - Optimize to get foundational pieces in and not "let perfect be > the enemy of good" > - Smoother migration path between AF 2 & 3 especially for DAG Authors > with the existing official Airflow providers. > - Working on features that solidify Airflow as the modern Orchestrator > that also has state-of-the-art support for Data, AI & ML workloads. > - This includes improving scalability & performance of all the > Airflow components. > - Making Airflow aware of what's happening in the task to provide > better auditability, lineage & observability > - Set up the codebase for the next 3-5 years. > - Reducing matrix of supported combinations for reducing complexity > in testing & development. E.g Remove MySQL support to reduce the test matrix > - Simplifying codebase & standardize architecture (e.g > consolidating serialization methods) > - Remove deprecations > - Simplify the Learning Curve for new Airflow users > - Shift focus on Airflow 2 to stability: bug fixes + security fixes > after AF 2.10. This should continue for a longer period of time after AF 3 > release > - Target a shorter cycle to release Airflow 3 > - so that Airflow 2 branches for features don't diverge > - have enough time between Airflow 3 release and Airflow Summit > 2025, so we can have talks about Successful migrations > > For (2), I propose the following guidelines: > > - Alignment with Core Principles > - Community Demand and Feedback > - Impact on Scalability and Performance > - Implementation Complexity and Maintenance > - Backward Compatibility and Migration Effort > - Workstream Ownership (can be more than one). If no one is available > to lead the workstream, the feature will be parked until a dedicated owner > is found > - For big epics, AIPs & a successful vote on the dev mailing list > > Please reply if anyone has anything to add to the agenda or comment on > anything if you disagree. > > Looking forward to the call. > > Regards, > Kaxil > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto:dev-unsubscr...@airflow.apache.org> For additional commands, e-mail: dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org>
