Thanks for initiating this discussion @Kaxil Naik <ka...@astronomer.io>!
Read through the document and the emails here and I like the direction in which we are proceeding. I also agree that removing code is almost as important as adding code. We need to have a clear checklist of what we want to remove and get a vote for the same from the community. I also see that we have multiple ways of doing a thing, which is not needed (serialising for example) and removing which can lead to simpler interfaces for users, and contributors too. I also find the *secret masker* overly complex personally, why can't it be simpler :) However, I have a small concern with the proportion of the users that would like to migrate to an early version of Airflow 3 if we do not support various deployment modes. If we dont get enough initial feedback, we do not know how the new major is doing, no? Thanks & Regards, Amogh Desai On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <a...@apache.org> wrote: > Strong agree - I think one of the things I regret most about Airflow 2 was > that we didn't remove enough. It probably eased the upgrade process for > users though. > > > but I was hoping one day when we remove providers > > from main development, I can do it personally and beat Ash to it. > > Grrr ;l > > The other thing I want to look at (but don't have concrete ideas of yet) > is some of the lesser used dag/task concurrency and dependency features - > if we can remove a few things and get a faster scheduler than it's worth > having a discussion over. > > On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com> wrote: > >I hope more voices will be coming soon as well :). > > > >Constance - few words of explanation, for my "remove" things. I do not say > >we "have to" remove those things or even that we "should". > > > >My main motivation for mentioning the list is that we often forget that > >when we have a new major version of software - removing stuff is as > >important as adding. So I think when it comes to final decisions about > >Airflow 3 we should be well aware about both: what we add and what we > >remove. So my main point is - "removal" of things should be as important a > >part in our future discussions as "adding", and I'd say with 10 years of > >Airflow history, it's more important to remove things than to add them. > > > >Side comment: Ash for one mentioned at multiple occasions how proud he is > >that his contributions to Airflow are net-negative - he removed more code > >than added. Unfortunately it's not shown any more here > >https://github.com/apache/airflow/graphs/contributors - because we have > >more than 10.000 commits, but I was hoping one day when we remove > providers > >from main development, I can do it personally and beat Ash to it. > > > >Just wanted to stress a few things on why I think "discussing potential > >removals" is important: > > > >* I think it is super important to deliver Airflow 3 really fast. It took > > > >1.5 year to release Airflow 2 and it was far too long and keeping Airflow > 1 > >and 2 in parallel was a huge pain and drain and it slowed us down > >enormously. We should absolutely limit the time when we actively develop > >both Airflow 2 features and Airflow 3 features. > >* I think most of the current deprecations of the 100+ we have does not > >slow us down AT ALL. Removal of those is mostly cosmetic change, a little > >bit clutter to remove but they have little-to-no impact on actual speed of > >development, it's just an old code that keeps on being around > >* on the other hand - some of the things I mentioned ARE slowing us down A > >LOT and will continue to do so until we remove them. For example, I > believe > >"postgresql + mysql + sqlite complete versioning solution for all the > >possible variants of storage" will be quite a bit more complex and testing > >will take far more time than if we drop mysql and choose a single storage > >approach for Airflow 3.0. I have no hard data to back it up, but my gut > >feeling is that it can take at least twice as long to get it out in the > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for > >all the options we have there. > >* the telemetry will be cool - but even if we add it for 2.10, the first > >time meaningful data will be available is maybe 2 years from now when we > do > >Airflow 4. For now we can completely forget about it because it's only > >going to show us some early adopters of Airflow 2.10. But we have surveys > + > >we can collect data from Astronomer, Google. Amazon for some usage and > >identify who will be early adopters of Airflow 3 and target them first. > >* the fact that we will drop something for Airflow 3.0, does not mean that > >we have to drop it "forever". We can safely assume that Airflow 3.0 will > be > >only used by early adopters, and many of our users will wait for 3.1, 3.2 > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those "slower > >moving users" from the 3.0. We can re-add things in 3.1 building on > >foundations of 3.0 after it is in the hands of those early adopters. I > >personally think we should prioritize speed of delivery of 3.0 over > >"complete support of every deployment option of Airflow 2" - to allow > >**some** of our early adopter users to migrate quickly, and add what's > >missing for those who would like to move later in later versions. > >* I think - this is the most important "product management" decision here > - > >based on the data we have available. Which users do we target for Airflow > >3.0, and what we can drop to deliver it faster (and possibly re-add more > >stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0 > >initiative and the most important decision from the Product management > side > >that will impact the timeline of Airflow 3.0. While a lot depends on how > >much time and effort individuals will spend on implementing things, the > >decision of what we drop will impact everyone's speed and overall delivery > >of Airflow 3.0. It's a decision we should not take lightly. > > > >So my main point is: let's add "what we remove" as a very important point > >in the product discussions we are going to have. > > > >J. > > > > > > > > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar > ><rbish...@amazon.com.invalid> wrote: > > > >> Like Constance and many others, there was time needed to process this > >> fantastical summary and approach to Airflow 3. __ > >> A lot of the points raised by Jarek make sense, ie being prescriptive on > >> what features we want to be there on Day 1 vs launched on 3.x - this > does > >> allow us to move quickly for our users. > >> In line with Constance's concern, we need to be mindful of what we > decide > >> to keep and what we are willing to cut - which could be a painstaking > >> process that could offset any gains of trying to be faster. I also > believe > >> that bringing all these new amazing features on Airflow 3 will peak the > >> interest of early adopters and eventually get others interested in > >> migration. However, I believe this migration will be a slow process and > >> will present a gap in certain functionalities that users may want before > >> entertaining any move to Airflow 3. There are still a lot of folks using > >> v1.10 today. There were several tactical initiatives in the past few > months > >> with intent on bringing new functionality, ie Multi-team, to Airflow 2.x > >> and I feel that while these efforts are not wasted, there should still > be > >> an option to continue improving Airflow 2 to avoid alienating our users > on > >> the basis of a future promise in Airflow 3, that may not be easy to > migrate > >> towards. > >> > >> -- Rajesh > >> > >> > >> > >> > >> > >> > >> > >> > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> < > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org>> > >> > >> > >> On 2024-05-07, 12:26 PM, "Constance Martineau" > >> <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> > >> <mailto:consta...@astronomer.io.inva <mailto: > consta...@astronomer.io.inva>>LID> > >> wrote: > >> > >> > >> > >> > >> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > >> le contenu ne présente aucun risque. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> Thank you Jarek for the detailed input. I've taken some time to digest > your > >> points before responding. > >> > >> > >> > >> > >> You've outlined a bold vision for Airflow 3, and I agree that being > >> decisive about the features and architectures will be the key to > success. > >> However, before we make final decisions on what features to cut or > retain, > >> it would be beneficial to have a more comprehensive understanding of how > >> the current features are utilized by the open-source community. > >> > >> > >> > >> > >> @Kaxil Naik <ka...@astronomer.io <mailto:ka...@astronomer.io> <mailto: > >> ka...@astronomer.io <mailto:ka...@astronomer.io>>> recently initiated a > >> discussion on > >> collecting telemetry from open-source deployments: > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m < > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>>. > >> This data > >> could be critical in ensuring our decisions are well-informed and > reflect > >> the real-world usage patterns of our users, not just those from managed > >> environments like Astro, MWAA or GCC. > >> > >> > >> > >> > >> It's essential that we challenge our assumptions and base our decisions > on > >> a holistic view of feature usage. Identifying potential cuts is a > critical > >> step, but let's ensure our strategy aligns with the needs and > preferences > >> of the broader Airflow community. > >> > >> > >> > >> > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <ja...@potiuk.com <mailto: > >> ja...@potiuk.com> <mailto:ja...@potiuk.com <mailto:ja...@potiuk.com>>> > >> wrote: > >> > >> > >> > >> > >> > I am currently on sick leave, and still recovering - hoping to be > able to > >> > travel next week to the US as planned, so I just wanted to break out > of > >> it > >> > to make one comment here. > >> > > >> > I got a clearer head now a bit with medications hopefully working. I > am > >> > still taking it that should help me to get over the current state, > and I > >> > wanted to take a look at this discussion unraveling first. Over last > week > >> > I disconnected from "day-to-day" Airflow and put some thoughts (as > much > >> as > >> > I could in my current state) on it. The whole subject of this thread > was > >> > started from that - how the current discussions on AIP-67 and others > >> change > >> > if we consider Airflow 3 is "starting". > >> > > >> > The price for back-compat is speed of development and quality. More > >> > combinations to test, more unexpected issues uncovered, necessity to > keep > >> > parallel paths (old/new) while adding new features. All what Constance > >> > wrote about and what Ash explained. We already started to trip over > our > >> own > >> > feet mutliple times in a few last releases. Have we tested all > >> combinations > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we already > see > >> > that in a number of "combos" of features things are not working in as > >> > stable a way as they did before. > >> > > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a > long > >> > time (or even move out) as they will not want to move to Airflow 3. A > lot > >> > of the work implemented in AIP-44 and design of AIP-67 was done around > >> > back-compatibility. but yes - > >> > it would have been way easier if designed anew without > back-compatibility > >> > in mind. And if we implement it and release it in Airflow 2 it will > make > >> > new Airflow feature development even harder. That's why I wanted to > treat > >> > it as "tactical" solution - hoping that in Airflow 3 we can make it > >> > "properly" - and that's why I started the discussion here when I > sensed > >> > that we are "close" to Airflow 3 discussion, because I wanted to see > what > >> > options we have there. This is why I have not yet concluded voting on > >> > AIP-67 waiting for the result of this discussion here. > >> > > >> > But if we are ready to go for Airflow.3 then I'd say there are two > >> > important things that should be part of the vision. > >> > > >> > 1) *We should be far more opinionated and have far fewer options of > >> > running things in Airflow 3*. Even an order of magnitude more > >> opinionated. > >> > Make choices, stick to it, perfect those opinionated choices to suit > >> 80/20 > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not > fitting > >> > the 20% that might choose to stay at Airflow 2. We can choose now > which > >> > ~20% of cases we do not want to handle deliberately. And we should be > >> very, > >> > very strict about it. Default should be "no choice". This will > radically > >> > simplify deployment and should make it easier to simplify Airflow > >> > development and DAG authoring experience because we will have less > cases > >> to > >> > support. Even if we plan to add more options in the future, the first > >> > version of Airflow 3 should support one deployment approach only. > This is > >> > the only way we can deliver it fast. And we should be very bold there. > >> > Choose one option and go for it in pretty much every place we have > >> choices > >> > now. We should Aim for Airflow 3.0 to support only a subset of current > >> > users - but those who are most likely to migrate first and those with > the > >> > biggest need for the new features. We can think 3.x to support more > >> cases, > >> > but 3.0 should be as opinionated as humanly possible. > >> > > >> > And this deployment option should be also something ALL our > stakeholders > >> > will feel OK with as a way forward in their offering. > >> > > >> > My candidates (and yes, some are bold): > >> > > >> > * *Drop MySQL*. If we have a single thing that makes us avoid our > schema > >> > and DB migration - this is the case. Let's choose Postgres 15+ and use > >> some > >> > of the great features there. This will also enable much faster async > SQL > >> > implementation and a number of other optimisations - not to mention > >> cutting > >> > every single change in development and testing time by literally half. > >> And > >> > we should not look back to adding MySQL. > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S only > (and > >> > AWS/Google others can continue developing theirs of course in parallel > >> and > >> > continue Hybrid executor work). Later - we figure out a better > solution > >> to > >> > support "small" tasks using some new K8S features and possibly non-k8s > >> > solutions (Ray-based?) > >> > * *Cut Connection and Variable Management from DB/UI*. Leave only > Secrets > >> > Management. Later when we have a 100% extensible React UI, we can add > a > >> > "local DB secrets manager" add-on > >> > * *Choose a single way for DAG storage that will support versioning > from > >> > day one*. Bear in mind we can add others later. Bolke's idea of using > >> > FSspec is an interesting one, we should see if it is feasible. > >> > * *Drop FAB completely (including custom plugins) and invest in > >> > implementing Auth Manager based on a dedicated, external solution* > >> > (KeyCloak > >> > that we've discussed before as a likely candidate) > >> > * *Leave Providers with Airflow 2 and add tests to make sure they are > >> > Airflow 3 future-compatible *- develop a way where we continue > >> development > >> > and contributions for Providers with Airflow 2 and add complete tests > to > >> > run them with Airflow 3. This way we can continue developing Provider > >> > features independently, and make them work for Airflow 2 (and continue > >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes), > while > >> > also gradually fix any Airflow3 incompatibilities and instead of > >> > "back-compatibility" tests make provider "forward-compatibility" > tests so > >> > that future Providers are tested and work on Airflow 3. Also it will > make > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested without > >> > investing in changing the current CI / test harness. > >> > * *Simplify Test Harness for Airflow 3 from the start *- without > >> providers > >> > and 790+ dependencies, we could vastly simplify Airflow3 testing > >> (basically > >> > make CI jobs from scratch) using mostly standard Python tooling > (while we > >> > can continue making use of the current test harness for Airflow 2 + > >> > Providers and extend it with Airflow 3 future-compatibility tests). > That > >> > means Breeze would be only staying in Airflow 2 + Providers repo as we > >> > should be able to achieve most of what we have there with local venv/ > >> > tooling (especially with uv as underlying tooling). > >> > > >> > 2) *I think we only add very few new "important" features. *Absolute > >> > minimum to make Airflow 3 appealing and add them only in Airflow 3: > >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it > makes > >> no > >> > sense to invest into Airflow 2 if we already know Airflow 3 is coming > - > >> > that generally triples effort needed to get them out. We should drop > new > >> > features development in Airflow 2. This will give users incentive to > move > >> > to 3 if the new features will be worth it. Even paying > >> > compatibility/migration price. > >> > > >> > Versionig, for example: I believe if we decide to go only with > Airflow 3 > >> > and cut some of the above (Postgres only, Single versioning DAG > storage) > >> we > >> > can make bolder decisions in versioning and support simpler models > from > >> the > >> > get go (and deliver it faster). And we should add only a few - but > >> > important - features that our users clearly asked for and focus on > >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or > >> 2.11). > >> > Similarly - multi-team can be simplified if we cut things from the > list > >> > above and have Task isolation as first-class citizens in Airflow (and > the > >> > only option). > >> > > >> > My candidates very much concur with the list shared by Kaxil in the > doc + > >> > I'd add multi-team (but simplified thanks to the cuts). But I also > here > >> > would mostly revert to Astronomer, Google. AWS team to define > >> collectively > >> > what is the absolute minimum set of features that would get the > "target" > >> > part of their customers happy. And ONLY do that. > >> > > >> > So in short - I think the big part of our discussion should be what we > >> are > >> > ready to drop when we start airflow 3 and be very bold. Once we know > we > >> > should figure out the absolute minimum of things that we can add that > >> will > >> > benefit a significant part of our users (and make use of increased > speed > >> > because we dropped things). > >> > > >> > J. > >> > > >> > > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau > >> > <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> > >> <mailto:consta...@astronomer.io.inva <mailto: > consta...@astronomer.io.inva>>lid> > >> wrote: > >> > > >> > > Hi Michal, > >> > > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate > your > >> > > concerns about the migration overhead for our users with a major new > >> > > version and see the appeal in your suggestion to integrate many of > the > >> > > proposed changes into Airflow 2 through separate AIPs. It’s a valid > >> point > >> > > and certainly aligns with the value of making incremental > improvements. > >> > > > >> > > However, after looking closely at the enhancements outlined for > Airflow > >> > 3, > >> > > I'm convinced they warrant a new major release. Here’s why: > >> > > > >> > > 1. *Core Architectural Changes:* We’re looking at foundational > changes > >> > > with Airflow 3—like redefining task priorities, separating task > >> > > definition > >> > > and task execution, and new AIPs like DAG versioning. remote > execution > >> > > and restricting database access from workers. These aren’t just > >> > > incremental > >> > > improvements but major shifts that will set the stage for the next > >> > > decade > >> > > of Airflow’s architecture. Grouping these changes into a major > release > >> > > will > >> > > help us make these transitions more cleanly and with fewer > constraints > >> > > from > >> > > past decisions. > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140 > >> > deprecated > >> > > issues, and this will only grow if we continue without a major > >> > cleanup. > >> > > This makes it increasingly difficult to implement new features > >> > > effectively > >> > > while maintaining backward compatibility. A major release allows us > to > >> > > address these issues head-on, reducing technical debt and paving the > >> > way > >> > > for a more robust platform. > >> > > 3. *Managing Breaking Changes:* Let’s take the example of > restricting > >> > > database access from workers. It’s a necessary move for better > >> > security > >> > > and > >> > > also potentially scalability reasons (reduces DB load). Many users > >> > have > >> > > workflows that interact with the DB, either by using raw sql or by > >> > > leveraging a session object. We could implement this feature in > >> > Airflow > >> > > 2 > >> > > and avoid breaking existing workflows by continuing to have the old > >> > > standard mode as default - much of the work is already done - but > that > >> > > would mean supporting both the new secure mode and the old standard > >> > mode > >> > > indefinitely and design new features with the assumption that most > >> > will > >> > > continue using the old standard mode. With Airflow 3, we can make > >> > secure > >> > > mode the default or even the only option, simplifying implementation > >> > and > >> > > future development. This is just one example where it is feasible to > >> > > implement in Airflow 2, but is better if we release it under the > >> > > context of > >> > > Airflow 3. > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up > >> > > possibilities for handling workflows beyond batch processing. > Features > >> > > like > >> > > real-time DAG execution through API and multi-language task support > >> > are > >> > > big > >> > > steps forward, significantly expanding Airflow’s utility. > >> > > > >> > > > >> > > While integrating these updates into Airflow 2 might look less > >> disruptive > >> > > initially, the scale and nature of the required changes really > support > >> a > >> > > move to Airflow 3. It’s not just about adding new features; it’s > about > >> > > setting up Airflow so that it continues to remain relevant for the > next > >> > ten > >> > > years. > >> > > > >> > > Constance > >> > > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <a...@apache.org > >> <mailto:a...@apache.org> <mailto:a...@apache.org <mailto:a...@apache.org > >>> > >> wrote: > >> > > > >> > > > There's a lot of technical debt hiding in Airflow, especially the > >> > > > scheduler that makes it harder and harder to efficiently add new > >> > > features. > >> > > > > >> > > > At some point, very soon, we are going to have to remove some very > >> > > > infrequently used back compat shims that negatively affect > >> performance. > >> > > > Without doing that the pace at which we can realistically add > some of > >> > the > >> > > > more exciting features tends towards zero. Developer speed of > >> > > contributors > >> > > > is a factor here too! > >> > > > > >> > > > So while we are still using SemVer, that necessitates v3. > >> > > > > >> > > > Ash > >> > > > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" < > michalmod...@google.com > >> <mailto:michalmod...@google.com> <mailto:michalmod...@google.com > <mailto: > >> michalmod...@google.com>> > >> > > .INVALID> > >> > > > wrote: > >> > > > >+1 to Jens's & Bolke's points here and in the doc > >> > > > > > >> > > > >I agree we should work on clarifying the directions we would like > >> > > Airflow > >> > > > >to go. Introducing a new major Airflow version is a massive > overhead > >> > for > >> > > > >users, who would need to plan for migrations, onboarding the new > >> > Airflow > >> > > > >(with a slightly different architecture), etc., and effectively > >> > Airflow > >> > > 2 > >> > > > >would live in parallel for a long time. > >> > > > > > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc > are > >> > > > valuable > >> > > > >projects of their own, and I could imagine all of them being > >> delivered > >> > > as > >> > > > >separate AIPs within Airflow 2 (surely new minor versions of > Airflow > >> > > 2). I > >> > > > >am not sure if the scope of changes and the goal we want to > achieve > >> is > >> > > a) > >> > > > >clear enough b) broad enough to call for a new major version. > >> > > > > > >> > > > >Best, > >> > > > >Michal > >> > > > > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T) > >> > > > ><jens.scheff...@de.bosch.com.inva <mailto: > >> jens.scheff...@de.bosch.com.inva> <mailto: > jens.scheff...@de.bosch.com.inva > >> <mailto:jens.scheff...@de.bosch.com.inva>>lid> wrote: > >> > > > > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is > mostly a > >> > > > vision > >> > > > >> statement. > >> > > > >> > >> > > > >> Looking forward for a larger addendum where we can collect > things > >> > that > >> > > > we > >> > > > >> all can vote and agree on as targets. > >> > > > >> > >> > > > >> As I started earlier with a confluence page and it seems this > is > >> not > >> > > > >> accessible to all, shall we convert this to a Google Doc for > >> better > >> > > > >> collaboration and item collection? > >> > > > >> > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> < > >> https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> < > >> https://aka.ms/o0ukef&gt;>> > >> > > > >> ________________________________ > >> > > > >> From: Vikram Koka <vik...@astronomer.io.inva <mailto: > >> vik...@astronomer.io.inva> <mailto:vik...@astronomer.io.inva <mailto: > >> vik...@astronomer.io.inva>>LID> > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM > >> > > > >> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> < > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow > 2) > >> vs > >> > > > >> strategic (Airflow 3) approach > >> > > > >> > >> > > > >> Thank you for your feedback, Bolke and Andrey! > >> > > > >> > >> > > > >> Bolke, > >> > > > >> I have replied to some of your comments in the doc. > >> > > > >> I will provide a detailed write up on the "Interactive DAG run" > >> (or > >> > > > >> synchronous DAG run) capability, which has generated some early > >> > > > questions. > >> > > > >> I had intended to get an AIP published for that as a follow-up, > >> but > >> > I > >> > > > >> believe that a simpler write up would be useful ahead of the > AIP. > >> > > > >> > >> > > > >> Andrey, > >> > > > >> You raise an interesting point. > >> > > > >> > >> > > > >> As part of the Airflow 2.0 release, we as a community had > decided > >> to > >> > > > >> strictly adhere to Semver as detailed in the document you > >> > referenced. > >> > > We > >> > > > >> also consciously split out the "Core Airflow" releases from the > >> > > > "Provider" > >> > > > >> releases at that time. We had a clear expectation then for the > >> > cadence > >> > > > of > >> > > > >> both minor and patch releases, which we have generally adhered > to > >> > > since > >> > > > >> then. > >> > > > >> > >> > > > >> Personally, I am more concerned about our Provider releases > right > >> > now, > >> > > > as > >> > > > >> compared to the cadence of our major releases. I believe that > one > >> of > >> > > the > >> > > > >> proposed changes in the Airflow 3 document i.e. the clear > >> separation > >> > > for > >> > > > >> Task Execution will help here, but more may be needed. > >> > > > >> > >> > > > >> Definitely interested in more feedback on this as well. > >> > > > >> > >> > > > >> Vikram > >> > > > >> > >> > > > >> > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin < > >> > > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> <mailto: > >> andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>> > >> > > > > > >> > > > >> wrote: > >> > > > >> > >> > > > >> > I would like to propose to change (at least discuss) release > >> > policy > >> > > > >> around > >> > > > >> > the Major version of Airflow. > >> > > > >> > > >> > > > >> > Right now it is described as "These releases do not happen > with > >> > any > >> > > > >> regular > >> > > > >> > interval or on any predictable schedule." : > >> > > > >> > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > >> < > >> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > > >> < > >> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > > >> < > >> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> > >> ;> > >> > > > >> < > >> > > > >> > >> > > > > >> > > > >> > > >> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > >> < > >> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > >> < > >> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > >> < > >> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > >> ;> > >> > > > >> > > >> > > > >> > > >> > > > >> > So maybe it is time to make it schedulable, e.g. one per two > >> years > >> > > or > >> > > > so. > >> > > > >> > This one could help us to avoid such a discussion in the > future, > >> > > like > >> > > > "We > >> > > > >> > don't know when Airflow 4 is coming.". At the moment when the > >> new > >> > > > major > >> > > > >> > version will be released new features wouldn't be added in > the > >> old > >> > > > major > >> > > > >> > version, however we would support bug / security for a while, > >> > e.g. 1 > >> > > > year > >> > > > >> > for bug fixes, 3 years for security fixes with a total 5 year > >> > > > lifecycle > >> > > > >> per > >> > > > >> > a major version. These just are approximate time periods for > a > >> > > > definition > >> > > > >> > of current period, bugfix period and security fix period. > >> > > > >> > > >> > > > >> > In contributors' perspective it helps with dropping the > >> deprecated > >> > > > stuff > >> > > > >> > which resolves some old problem: we have to support > everything > >> > > > including > >> > > > >> > deprecated stuff and without schedulable lifecycle for the > >> > > deprecated > >> > > > >> stuff > >> > > > >> > it could be showstopper for the new feature, because > sometimes > >> it > >> > > > hard to > >> > > > >> > support two different approaches for long period of time > with no > >> > > hope > >> > > > >> that > >> > > > >> > it will happen soon. For some fundamental stuff which do not > >> > > require a > >> > > > >> lot > >> > > > >> > things time to support we could postponed removal for next > after > >> > the > >> > > > next > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in > Airflow 5 > >> > > > >> > > >> > > > >> > In the user perspective, they have at least bug fix support > for > >> a > >> > > > while, > >> > > > >> if > >> > > > >> > someone want to use legacy version it their choice, however > no > >> new > >> > > > >> > features, no new version of providers (after one year) > >> > > > >> > > >> > > > >> > > >> > > > >> > ---- > >> > > > >> > Best Wishes > >> > > > >> > *Andrey Anshin* > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin < > bdbr...@gmail.com > >> <mailto:bdbr...@gmail.com> <mailto:bdbr...@gmail.com <mailto: > >> bdbr...@gmail.com>>> > >> > > > wrote: > >> > > > >> > > >> > > > >> > > I have left several comments :-). And on interactive dag > runs > >> > even > >> > > > >> after > >> > > > >> > > the explanation of Vikram I still don't have a clue what we > >> want > >> > > to > >> > > > >> > > accomplish there :-P. > >> > > > >> > > > >> > > > >> > > I would like to see a mantra or team for Airflow 3. That > helps > >> > > > nudging > >> > > > >> > > people in the same direction. Suggestions in the comments. > >> > > > >> > > > >> > > > >> > > Bolke > >> > > > >> > > Sent from my iPhone > >> > > > >> > > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka > >> > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > <mailto: > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>>lid > >> > > > > > >> > > > >> > > wrote: > >> > > > >> > > > > >> > > > >> > > > Good point Jed. > >> > > > >> > > > I responded back to your comment in the doc as well and > very > >> > > open > >> > > > to > >> > > > >> > > > changing the term in the doc. > >> > > > >> > > > > >> > > > >> > > > Used the term "interactive DAG run" as the ability to > invoke > >> > or > >> > > > >> > trigger a > >> > > > >> > > > DAG run through the API, with the expectation of getting > >> back > >> > a > >> > > > >> result > >> > > > >> > > > immediately. An alternate term could be a "synchronous > DAG > >> > run". > >> > > > >> > > > > >> > > > >> > > > Regardless, this is a significant change so a good term > to > >> > > > indicate > >> > > > >> the > >> > > > >> > > > expansion from "batch runs only" is warranted. Very open > to > >> > > > different > >> > > > >> > > terms > >> > > > >> > > > here. > >> > > > >> > > > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham < > >> > > > >> > jedcunning...@apache.org <mailto:jedcunning...@apache.org> > >> <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org>> > >> > > > >> > > > > >> > > > >> > > >> wrote: > >> > > > >> > > >> > >> > > > >> > > >> Very exciting! Looks like we will have a busy period of > >> time > >> > > > ahead > >> > > > >> of > >> > > > >> > > us. > >> > > > >> > > >> Overall I like the plan so far, especially using this > >> year's > >> > > > Airflow > >> > > > >> > > Summit > >> > > > >> > > >> as an opportunity to announce and gather feedback, and > the > >> > 2025 > >> > > > >> > version > >> > > > >> > > to > >> > > > >> > > >> pitch upgrading. > >> > > > >> > > >> > >> > > > >> > > >> I left a comment in the doc, but we might want to > iterate > >> on > >> > > the > >> > > > >> > > >> terminology we use for high priority or "synchronous" > DAG > >> > runs > >> > > to > >> > > > >> > serve > >> > > > >> > > LLM > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit > confusing. > >> > > > >> > > >> > >> > > > >> > > > >> > > > >> > > > >> > > > > --------------------------------------------------------------------- > >> > > > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> <mailto:dev-unsubscr...@airflow.apache.org> <mailto: > >> dev-unsubscr...@airflow.apache.org <mailto: > >> dev-unsubscr...@airflow.apache.org>> > >> > > > >> > > For additional commands, e-mail: > dev-h...@airflow.apache.org > >> <mailto:dev-h...@airflow.apache.org> <mailto: > dev-h...@airflow.apache.org > >> <mailto:dev-h...@airflow.apache.org>> > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > >> > >> > >> > >> > >> > >> > >> >
