Thanks Shubham and Jens, I will take a look later today. On Wed, 29 May 2024 at 03:07, Mehta, Shubham <[email protected]> wrote:
> Kaxil - thank you for creating the wiki, setting up call invites, and > starting this thread. The first draft of the principles looks great. I have > commented on the wiki with some feedback and personal thoughts. I'm not > adding the comments on this thread to keep the wiki as the single place for > discussions and feedback. > > Thanks > Shubham > > On 2024-05-28, 12:25 PM, "Kaxil Naik" <[email protected] <mailto: > [email protected]>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > If the formatting of bullets was lost, check below or at > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > Proposed Agenda: > 1) Agreeing on the Principles to drive Airflow 3 development > 2) Agreeing on the Guidelines that help decide if a feature should be in > Airflow 3 or not > > > > > For (1), I propose the following principles: > > > - Considering Airflow 3.0 for early adopters and breaking (and removing) > things. Things can be re-added as needed in upcoming minor releases. > - Optimize to get foundational pieces in and not "let perfect be the enemy > of good" > - Smoother migration path between AF 2 & 3 especially for DAG Authors with > the existing official Airflow providers. > - Working on features that solidify Airflow as the modern Orchestrator > that also has state-of-the-art support for Data, AI & ML workloads. > - This includes improving scalability & performance of all the Airflow > components. > - Making Airflow aware of what's happening in the task to provide better > auditability, lineage & observability > - Set up the codebase for the next 3-5 years. > - Reducing matrix of supported combinations for reducing complexity in > testing & development. E.g Remove MySQL support to reduce the test matrix > - Simplifying codebase & standardize architecture (e.g consolidating > serialization methods) > - Remove deprecations > - Simplify the Learning Curve for new Airflow users > - Shift focus on Airflow 2 to stability: bug fixes + security fixes after > AF 2.10. This should continue for a longer period of time after AF 3 release > - Target a shorter cycle to release Airflow 3 > - so that Airflow 2 branches for features don't diverge > - have enough time between Airflow 3 release and Airflow Summit 2025, so > we can have talks about Successful migrations > > > For (2), I propose the following guidelines: > > > - Alignment with Core Principles > - Community Demand and Feedback > - Impact on Scalability and Performance > - Implementation Complexity and Maintenance > - Backward Compatibility and Migration Effort > - Workstream Ownership (can be more than one). If no one is available to > lead the workstream, the feature will be parked until a dedicated owner is > found > - For big epics, AIPs & a successful vote on the dev mailing list > > > Please reply if anyone has anything to add to the agenda or comment on > anything if you disagree. > > > Looking forward to the call. > > > Regards, > Kaxil > > > On 2024/05/28 19:11:14 Kaxil Naik wrote: > > Hi all, > > > > As discussed in the previous email thread, the first dev call has been > > pushed to next Tuesday (4th June 2024). > > > > If you would like to participate in the development of Airflow 3, please > > join the dev calls starting next week. The calls will be open to anyone > in > > the community. > > > > *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 PM > > EST | 9 AM PST) > > *One-time registration Link*: > > > https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG > < > https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG > > > > > > The meeting notes from the call will also be posted on the dev mailing > list > > and Confluence for archival purposes > > at > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes > >. > > > > > > > > *Proposed Agenda*: > > 1) Agreeing on the Principles to drive Airflow 3 development > > 2) Agreeing on the Guidelines that help decide if a feature should be in > > Airflow 3 or not > > > > > > For (1), I propose the following principles: > > > > - Considering Airflow 3.0 for early adopters and breaking (and > > removing) things. Things can be re-added as needed in upcoming minor > > releases. > > - Optimize to get foundational pieces in and not "let perfect be > > the enemy of good" > > - Smoother migration path between AF 2 & 3 especially for DAG Authors > > with the existing official Airflow providers. > > - Working on features that solidify Airflow as the modern Orchestrator > > that also has state-of-the-art support for Data, AI & ML workloads. > > - This includes improving scalability & performance of all the > > Airflow components. > > - Making Airflow aware of what's happening in the task to provide > > better auditability, lineage & observability > > - Set up the codebase for the next 3-5 years. > > - Reducing matrix of supported combinations for reducing complexity > > in testing & development. E.g Remove MySQL support to reduce the test > matrix > > - Simplifying codebase & standardize architecture (e.g > > consolidating serialization methods) > > - Remove deprecations > > - Simplify the Learning Curve for new Airflow users > > - Shift focus on Airflow 2 to stability: bug fixes + security fixes > > after AF 2.10. This should continue for a longer period of time after AF > 3 > > release > > - Target a shorter cycle to release Airflow 3 > > - so that Airflow 2 branches for features don't diverge > > - have enough time between Airflow 3 release and Airflow Summit > > 2025, so we can have talks about Successful migrations > > > > For (2), I propose the following guidelines: > > > > - Alignment with Core Principles > > - Community Demand and Feedback > > - Impact on Scalability and Performance > > - Implementation Complexity and Maintenance > > - Backward Compatibility and Migration Effort > > - Workstream Ownership (can be more than one). If no one is available > > to lead the workstream, the feature will be parked until a dedicated > owner > > is found > > - For big epics, AIPs & a successful vote on the dev mailing list > > > > Please reply if anyone has anything to add to the agenda or comment on > > anything if you disagree. > > > > Looking forward to the call. > > > > Regards, > > Kaxil > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] <mailto: > [email protected]> > For additional commands, e-mail: [email protected] <mailto: > [email protected]> > > > > > >
