Added my comments too :). Looks great On Wed, May 29, 2024 at 12:14 PM Kaxil Naik <kaxiln...@gmail.com> wrote:
> Thanks Shubham and Jens, I will take a look later today. > > On Wed, 29 May 2024 at 03:07, Mehta, Shubham <shu...@amazon.com.invalid> > wrote: > > > Kaxil - thank you for creating the wiki, setting up call invites, and > > starting this thread. The first draft of the principles looks great. I > have > > commented on the wiki with some feedback and personal thoughts. I'm not > > adding the comments on this thread to keep the wiki as the single place > for > > discussions and feedback. > > > > Thanks > > Shubham > > > > On 2024-05-28, 12:25 PM, "Kaxil Naik" <kaxiln...@apache.org <mailto: > > kaxiln...@apache.org>> wrote: > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > If the formatting of bullets was lost, check below or at > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > > > > > Proposed Agenda: > > 1) Agreeing on the Principles to drive Airflow 3 development > > 2) Agreeing on the Guidelines that help decide if a feature should be in > > Airflow 3 or not > > > > > > > > > > For (1), I propose the following principles: > > > > > > - Considering Airflow 3.0 for early adopters and breaking (and removing) > > things. Things can be re-added as needed in upcoming minor releases. > > - Optimize to get foundational pieces in and not "let perfect be the > enemy > > of good" > > - Smoother migration path between AF 2 & 3 especially for DAG Authors > with > > the existing official Airflow providers. > > - Working on features that solidify Airflow as the modern Orchestrator > > that also has state-of-the-art support for Data, AI & ML workloads. > > - This includes improving scalability & performance of all the Airflow > > components. > > - Making Airflow aware of what's happening in the task to provide better > > auditability, lineage & observability > > - Set up the codebase for the next 3-5 years. > > - Reducing matrix of supported combinations for reducing complexity in > > testing & development. E.g Remove MySQL support to reduce the test matrix > > - Simplifying codebase & standardize architecture (e.g consolidating > > serialization methods) > > - Remove deprecations > > - Simplify the Learning Curve for new Airflow users > > - Shift focus on Airflow 2 to stability: bug fixes + security fixes after > > AF 2.10. This should continue for a longer period of time after AF 3 > release > > - Target a shorter cycle to release Airflow 3 > > - so that Airflow 2 branches for features don't diverge > > - have enough time between Airflow 3 release and Airflow Summit 2025, so > > we can have talks about Successful migrations > > > > > > For (2), I propose the following guidelines: > > > > > > - Alignment with Core Principles > > - Community Demand and Feedback > > - Impact on Scalability and Performance > > - Implementation Complexity and Maintenance > > - Backward Compatibility and Migration Effort > > - Workstream Ownership (can be more than one). If no one is available to > > lead the workstream, the feature will be parked until a dedicated owner > is > > found > > - For big epics, AIPs & a successful vote on the dev mailing list > > > > > > Please reply if anyone has anything to add to the agenda or comment on > > anything if you disagree. > > > > > > Looking forward to the call. > > > > > > Regards, > > Kaxil > > > > > > On 2024/05/28 19:11:14 Kaxil Naik wrote: > > > Hi all, > > > > > > As discussed in the previous email thread, the first dev call has been > > > pushed to next Tuesday (4th June 2024). > > > > > > If you would like to participate in the development of Airflow 3, > please > > > join the dev calls starting next week. The calls will be open to anyone > > in > > > the community. > > > > > > *Schedule*: June 4, 2024, Tuesday, at 05:00 PM BST (4 PM GMT/UTC | 12 > PM > > > EST | 9 AM PST) > > > *One-time registration Link*: > > > > > > https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG > > < > > > https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG > > > > > > > > > The meeting notes from the call will also be posted on the dev mailing > > list > > > and Confluence for archival purposes > > > at > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes > > >. > > > > > > > > > > > > *Proposed Agenda*: > > > 1) Agreeing on the Principles to drive Airflow 3 development > > > 2) Agreeing on the Guidelines that help decide if a feature should be > in > > > Airflow 3 or not > > > > > > > > > For (1), I propose the following principles: > > > > > > - Considering Airflow 3.0 for early adopters and breaking (and > > > removing) things. Things can be re-added as needed in upcoming minor > > > releases. > > > - Optimize to get foundational pieces in and not "let perfect be > > > the enemy of good" > > > - Smoother migration path between AF 2 & 3 especially for DAG Authors > > > with the existing official Airflow providers. > > > - Working on features that solidify Airflow as the modern Orchestrator > > > that also has state-of-the-art support for Data, AI & ML workloads. > > > - This includes improving scalability & performance of all the > > > Airflow components. > > > - Making Airflow aware of what's happening in the task to provide > > > better auditability, lineage & observability > > > - Set up the codebase for the next 3-5 years. > > > - Reducing matrix of supported combinations for reducing complexity > > > in testing & development. E.g Remove MySQL support to reduce the test > > matrix > > > - Simplifying codebase & standardize architecture (e.g > > > consolidating serialization methods) > > > - Remove deprecations > > > - Simplify the Learning Curve for new Airflow users > > > - Shift focus on Airflow 2 to stability: bug fixes + security fixes > > > after AF 2.10. This should continue for a longer period of time after > AF > > 3 > > > release > > > - Target a shorter cycle to release Airflow 3 > > > - so that Airflow 2 branches for features don't diverge > > > - have enough time between Airflow 3 release and Airflow Summit > > > 2025, so we can have talks about Successful migrations > > > > > > For (2), I propose the following guidelines: > > > > > > - Alignment with Core Principles > > > - Community Demand and Feedback > > > - Impact on Scalability and Performance > > > - Implementation Complexity and Maintenance > > > - Backward Compatibility and Migration Effort > > > - Workstream Ownership (can be more than one). If no one is available > > > to lead the workstream, the feature will be parked until a dedicated > > owner > > > is found > > > - For big epics, AIPs & a successful vote on the dev mailing list > > > > > > Please reply if anyone has anything to add to the agenda or comment on > > > anything if you disagree. > > > > > > Looking forward to the call. > > > > > > Regards, > > > Kaxil > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: > > dev-unsubscr...@airflow.apache.org> > > For additional commands, e-mail: dev-h...@airflow.apache.org <mailto: > > dev-h...@airflow.apache.org> > > > > > > > > > > > > >
