Strong agree - I think one of the things I regret most about Airflow 2 was that we didn't remove enough. It probably eased the upgrade process for users though.
> but I was hoping one day when we remove providers > from main development, I can do it personally and beat Ash to it. Grrr ;l The other thing I want to look at (but don't have concrete ideas of yet) is some of the lesser used dag/task concurrency and dependency features - if we can remove a few things and get a faster scheduler than it's worth having a discussion over. On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com> wrote: >I hope more voices will be coming soon as well :). > >Constance - few words of explanation, for my "remove" things. I do not say >we "have to" remove those things or even that we "should". > >My main motivation for mentioning the list is that we often forget that >when we have a new major version of software - removing stuff is as >important as adding. So I think when it comes to final decisions about >Airflow 3 we should be well aware about both: what we add and what we >remove. So my main point is - "removal" of things should be as important a >part in our future discussions as "adding", and I'd say with 10 years of >Airflow history, it's more important to remove things than to add them. > >Side comment: Ash for one mentioned at multiple occasions how proud he is >that his contributions to Airflow are net-negative - he removed more code >than added. Unfortunately it's not shown any more here >https://github.com/apache/airflow/graphs/contributors - because we have >more than 10.000 commits, but I was hoping one day when we remove providers >from main development, I can do it personally and beat Ash to it. > >Just wanted to stress a few things on why I think "discussing potential >removals" is important: > >* I think it is super important to deliver Airflow 3 really fast. It took > >1.5 year to release Airflow 2 and it was far too long and keeping Airflow 1 >and 2 in parallel was a huge pain and drain and it slowed us down >enormously. We should absolutely limit the time when we actively develop >both Airflow 2 features and Airflow 3 features. >* I think most of the current deprecations of the 100+ we have does not >slow us down AT ALL. Removal of those is mostly cosmetic change, a little >bit clutter to remove but they have little-to-no impact on actual speed of >development, it's just an old code that keeps on being around >* on the other hand - some of the things I mentioned ARE slowing us down A >LOT and will continue to do so until we remove them. For example, I believe >"postgresql + mysql + sqlite complete versioning solution for all the >possible variants of storage" will be quite a bit more complex and testing >will take far more time than if we drop mysql and choose a single storage >approach for Airflow 3.0. I have no hard data to back it up, but my gut >feeling is that it can take at least twice as long to get it out in the >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for >all the options we have there. >* the telemetry will be cool - but even if we add it for 2.10, the first >time meaningful data will be available is maybe 2 years from now when we do >Airflow 4. For now we can completely forget about it because it's only >going to show us some early adopters of Airflow 2.10. But we have surveys + >we can collect data from Astronomer, Google. Amazon for some usage and >identify who will be early adopters of Airflow 3 and target them first. >* the fact that we will drop something for Airflow 3.0, does not mean that >we have to drop it "forever". We can safely assume that Airflow 3.0 will be >only used by early adopters, and many of our users will wait for 3.1, 3.2 >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those "slower >moving users" from the 3.0. We can re-add things in 3.1 building on >foundations of 3.0 after it is in the hands of those early adopters. I >personally think we should prioritize speed of delivery of 3.0 over >"complete support of every deployment option of Airflow 2" - to allow >**some** of our early adopter users to migrate quickly, and add what's >missing for those who would like to move later in later versions. >* I think - this is the most important "product management" decision here - >based on the data we have available. Which users do we target for Airflow >3.0, and what we can drop to deliver it faster (and possibly re-add more >stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0 >initiative and the most important decision from the Product management side >that will impact the timeline of Airflow 3.0. While a lot depends on how >much time and effort individuals will spend on implementing things, the >decision of what we drop will impact everyone's speed and overall delivery >of Airflow 3.0. It's a decision we should not take lightly. > >So my main point is: let's add "what we remove" as a very important point >in the product discussions we are going to have. > >J. > > > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar ><rbish...@amazon.com.invalid> wrote: > >> Like Constance and many others, there was time needed to process this >> fantastical summary and approach to Airflow 3. __ >> A lot of the points raised by Jarek make sense, ie being prescriptive on >> what features we want to be there on Day 1 vs launched on 3.x - this does >> allow us to move quickly for our users. >> In line with Constance's concern, we need to be mindful of what we decide >> to keep and what we are willing to cut - which could be a painstaking >> process that could offset any gains of trying to be faster. I also believe >> that bringing all these new amazing features on Airflow 3 will peak the >> interest of early adopters and eventually get others interested in >> migration. However, I believe this migration will be a slow process and >> will present a gap in certain functionalities that users may want before >> entertaining any move to Airflow 3. There are still a lot of folks using >> v1.10 today. There were several tactical initiatives in the past few months >> with intent on bringing new functionality, ie Multi-team, to Airflow 2.x >> and I feel that while these efforts are not wasted, there should still be >> an option to continue improving Airflow 2 to avoid alienating our users on >> the basis of a future promise in Airflow 3, that may not be easy to migrate >> towards. >> >> -- Rajesh >> >> >> >> >> >> >> >> >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> < >> dev@airflow.apache.org <mailto:dev@airflow.apache.org>> >> >> >> On 2024-05-07, 12:26 PM, "Constance Martineau" >> <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> >> <mailto:consta...@astronomer.io.inva >> <mailto:consta...@astronomer.io.inva>>LID> >> wrote: >> >> >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> >> >> >> >> >> >> >> >> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que >> le contenu ne présente aucun risque. >> >> >> >> >> >> >> >> >> >> >> >> >> Thank you Jarek for the detailed input. I've taken some time to digest your >> points before responding. >> >> >> >> >> You've outlined a bold vision for Airflow 3, and I agree that being >> decisive about the features and architectures will be the key to success. >> However, before we make final decisions on what features to cut or retain, >> it would be beneficial to have a more comprehensive understanding of how >> the current features are utilized by the open-source community. >> >> >> >> >> @Kaxil Naik <ka...@astronomer.io <mailto:ka...@astronomer.io> <mailto: >> ka...@astronomer.io <mailto:ka...@astronomer.io>>> recently initiated a >> discussion on >> collecting telemetry from open-source deployments: >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m < >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>>. >> This data >> could be critical in ensuring our decisions are well-informed and reflect >> the real-world usage patterns of our users, not just those from managed >> environments like Astro, MWAA or GCC. >> >> >> >> >> It's essential that we challenge our assumptions and base our decisions on >> a holistic view of feature usage. Identifying potential cuts is a critical >> step, but let's ensure our strategy aligns with the needs and preferences >> of the broader Airflow community. >> >> >> >> >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <ja...@potiuk.com <mailto: >> ja...@potiuk.com> <mailto:ja...@potiuk.com <mailto:ja...@potiuk.com>>> >> wrote: >> >> >> >> >> > I am currently on sick leave, and still recovering - hoping to be able to >> > travel next week to the US as planned, so I just wanted to break out of >> it >> > to make one comment here. >> > >> > I got a clearer head now a bit with medications hopefully working. I am >> > still taking it that should help me to get over the current state, and I >> > wanted to take a look at this discussion unraveling first. Over last week >> > I disconnected from "day-to-day" Airflow and put some thoughts (as much >> as >> > I could in my current state) on it. The whole subject of this thread was >> > started from that - how the current discussions on AIP-67 and others >> change >> > if we consider Airflow 3 is "starting". >> > >> > The price for back-compat is speed of development and quality. More >> > combinations to test, more unexpected issues uncovered, necessity to keep >> > parallel paths (old/new) while adding new features. All what Constance >> > wrote about and what Ash explained. We already started to trip over our >> own >> > feet mutliple times in a few last releases. Have we tested all >> combinations >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we already see >> > that in a number of "combos" of features things are not working in as >> > stable a way as they did before. >> > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a long >> > time (or even move out) as they will not want to move to Airflow 3. A lot >> > of the work implemented in AIP-44 and design of AIP-67 was done around >> > back-compatibility. but yes - >> > it would have been way easier if designed anew without back-compatibility >> > in mind. And if we implement it and release it in Airflow 2 it will make >> > new Airflow feature development even harder. That's why I wanted to treat >> > it as "tactical" solution - hoping that in Airflow 3 we can make it >> > "properly" - and that's why I started the discussion here when I sensed >> > that we are "close" to Airflow 3 discussion, because I wanted to see what >> > options we have there. This is why I have not yet concluded voting on >> > AIP-67 waiting for the result of this discussion here. >> > >> > But if we are ready to go for Airflow.3 then I'd say there are two >> > important things that should be part of the vision. >> > >> > 1) *We should be far more opinionated and have far fewer options of >> > running things in Airflow 3*. Even an order of magnitude more >> opinionated. >> > Make choices, stick to it, perfect those opinionated choices to suit >> 80/20 >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not fitting >> > the 20% that might choose to stay at Airflow 2. We can choose now which >> > ~20% of cases we do not want to handle deliberately. And we should be >> very, >> > very strict about it. Default should be "no choice". This will radically >> > simplify deployment and should make it easier to simplify Airflow >> > development and DAG authoring experience because we will have less cases >> to >> > support. Even if we plan to add more options in the future, the first >> > version of Airflow 3 should support one deployment approach only. This is >> > the only way we can deliver it fast. And we should be very bold there. >> > Choose one option and go for it in pretty much every place we have >> choices >> > now. We should Aim for Airflow 3.0 to support only a subset of current >> > users - but those who are most likely to migrate first and those with the >> > biggest need for the new features. We can think 3.x to support more >> cases, >> > but 3.0 should be as opinionated as humanly possible. >> > >> > And this deployment option should be also something ALL our stakeholders >> > will feel OK with as a way forward in their offering. >> > >> > My candidates (and yes, some are bold): >> > >> > * *Drop MySQL*. If we have a single thing that makes us avoid our schema >> > and DB migration - this is the case. Let's choose Postgres 15+ and use >> some >> > of the great features there. This will also enable much faster async SQL >> > implementation and a number of other optimisations - not to mention >> cutting >> > every single change in development and testing time by literally half. >> And >> > we should not look back to adding MySQL. >> > * *Drop Celery/Sequential Executor* and start with Local + K8S only (and >> > AWS/Google others can continue developing theirs of course in parallel >> and >> > continue Hybrid executor work). Later - we figure out a better solution >> to >> > support "small" tasks using some new K8S features and possibly non-k8s >> > solutions (Ray-based?) >> > * *Cut Connection and Variable Management from DB/UI*. Leave only Secrets >> > Management. Later when we have a 100% extensible React UI, we can add a >> > "local DB secrets manager" add-on >> > * *Choose a single way for DAG storage that will support versioning from >> > day one*. Bear in mind we can add others later. Bolke's idea of using >> > FSspec is an interesting one, we should see if it is feasible. >> > * *Drop FAB completely (including custom plugins) and invest in >> > implementing Auth Manager based on a dedicated, external solution* >> > (KeyCloak >> > that we've discussed before as a likely candidate) >> > * *Leave Providers with Airflow 2 and add tests to make sure they are >> > Airflow 3 future-compatible *- develop a way where we continue >> development >> > and contributions for Providers with Airflow 2 and add complete tests to >> > run them with Airflow 3. This way we can continue developing Provider >> > features independently, and make them work for Airflow 2 (and continue >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes), while >> > also gradually fix any Airflow3 incompatibilities and instead of >> > "back-compatibility" tests make provider "forward-compatibility" tests so >> > that future Providers are tested and work on Airflow 3. Also it will make >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested without >> > investing in changing the current CI / test harness. >> > * *Simplify Test Harness for Airflow 3 from the start *- without >> providers >> > and 790+ dependencies, we could vastly simplify Airflow3 testing >> (basically >> > make CI jobs from scratch) using mostly standard Python tooling (while we >> > can continue making use of the current test harness for Airflow 2 + >> > Providers and extend it with Airflow 3 future-compatibility tests). That >> > means Breeze would be only staying in Airflow 2 + Providers repo as we >> > should be able to achieve most of what we have there with local venv/ >> > tooling (especially with uv as underlying tooling). >> > >> > 2) *I think we only add very few new "important" features. *Absolute >> > minimum to make Airflow 3 appealing and add them only in Airflow 3: >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it makes >> no >> > sense to invest into Airflow 2 if we already know Airflow 3 is coming - >> > that generally triples effort needed to get them out. We should drop new >> > features development in Airflow 2. This will give users incentive to move >> > to 3 if the new features will be worth it. Even paying >> > compatibility/migration price. >> > >> > Versionig, for example: I believe if we decide to go only with Airflow 3 >> > and cut some of the above (Postgres only, Single versioning DAG storage) >> we >> > can make bolder decisions in versioning and support simpler models from >> the >> > get go (and deliver it faster). And we should add only a few - but >> > important - features that our users clearly asked for and focus on >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or >> 2.11). >> > Similarly - multi-team can be simplified if we cut things from the list >> > above and have Task isolation as first-class citizens in Airflow (and the >> > only option). >> > >> > My candidates very much concur with the list shared by Kaxil in the doc + >> > I'd add multi-team (but simplified thanks to the cuts). But I also here >> > would mostly revert to Astronomer, Google. AWS team to define >> collectively >> > what is the absolute minimum set of features that would get the "target" >> > part of their customers happy. And ONLY do that. >> > >> > So in short - I think the big part of our discussion should be what we >> are >> > ready to drop when we start airflow 3 and be very bold. Once we know we >> > should figure out the absolute minimum of things that we can add that >> will >> > benefit a significant part of our users (and make use of increased speed >> > because we dropped things). >> > >> > J. >> > >> > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau >> > <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> >> <mailto:consta...@astronomer.io.inva >> <mailto:consta...@astronomer.io.inva>>lid> >> wrote: >> > >> > > Hi Michal, >> > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate your >> > > concerns about the migration overhead for our users with a major new >> > > version and see the appeal in your suggestion to integrate many of the >> > > proposed changes into Airflow 2 through separate AIPs. It’s a valid >> point >> > > and certainly aligns with the value of making incremental improvements. >> > > >> > > However, after looking closely at the enhancements outlined for Airflow >> > 3, >> > > I'm convinced they warrant a new major release. Here’s why: >> > > >> > > 1. *Core Architectural Changes:* We’re looking at foundational changes >> > > with Airflow 3—like redefining task priorities, separating task >> > > definition >> > > and task execution, and new AIPs like DAG versioning. remote execution >> > > and restricting database access from workers. These aren’t just >> > > incremental >> > > improvements but major shifts that will set the stage for the next >> > > decade >> > > of Airflow’s architecture. Grouping these changes into a major release >> > > will >> > > help us make these transitions more cleanly and with fewer constraints >> > > from >> > > past decisions. >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140 >> > deprecated >> > > issues, and this will only grow if we continue without a major >> > cleanup. >> > > This makes it increasingly difficult to implement new features >> > > effectively >> > > while maintaining backward compatibility. A major release allows us to >> > > address these issues head-on, reducing technical debt and paving the >> > way >> > > for a more robust platform. >> > > 3. *Managing Breaking Changes:* Let’s take the example of restricting >> > > database access from workers. It’s a necessary move for better >> > security >> > > and >> > > also potentially scalability reasons (reduces DB load). Many users >> > have >> > > workflows that interact with the DB, either by using raw sql or by >> > > leveraging a session object. We could implement this feature in >> > Airflow >> > > 2 >> > > and avoid breaking existing workflows by continuing to have the old >> > > standard mode as default - much of the work is already done - but that >> > > would mean supporting both the new secure mode and the old standard >> > mode >> > > indefinitely and design new features with the assumption that most >> > will >> > > continue using the old standard mode. With Airflow 3, we can make >> > secure >> > > mode the default or even the only option, simplifying implementation >> > and >> > > future development. This is just one example where it is feasible to >> > > implement in Airflow 2, but is better if we release it under the >> > > context of >> > > Airflow 3. >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up >> > > possibilities for handling workflows beyond batch processing. Features >> > > like >> > > real-time DAG execution through API and multi-language task support >> > are >> > > big >> > > steps forward, significantly expanding Airflow’s utility. >> > > >> > > >> > > While integrating these updates into Airflow 2 might look less >> disruptive >> > > initially, the scale and nature of the required changes really support >> a >> > > move to Airflow 3. It’s not just about adding new features; it’s about >> > > setting up Airflow so that it continues to remain relevant for the next >> > ten >> > > years. >> > > >> > > Constance >> > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <a...@apache.org >> <mailto:a...@apache.org> <mailto:a...@apache.org <mailto:a...@apache.org>>> >> wrote: >> > > >> > > > There's a lot of technical debt hiding in Airflow, especially the >> > > > scheduler that makes it harder and harder to efficiently add new >> > > features. >> > > > >> > > > At some point, very soon, we are going to have to remove some very >> > > > infrequently used back compat shims that negatively affect >> performance. >> > > > Without doing that the pace at which we can realistically add some of >> > the >> > > > more exciting features tends towards zero. Developer speed of >> > > contributors >> > > > is a factor here too! >> > > > >> > > > So while we are still using SemVer, that necessitates v3. >> > > > >> > > > Ash >> > > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <michalmod...@google.com >> <mailto:michalmod...@google.com> <mailto:michalmod...@google.com <mailto: >> michalmod...@google.com>> >> > > .INVALID> >> > > > wrote: >> > > > >+1 to Jens's & Bolke's points here and in the doc >> > > > > >> > > > >I agree we should work on clarifying the directions we would like >> > > Airflow >> > > > >to go. Introducing a new major Airflow version is a massive overhead >> > for >> > > > >users, who would need to plan for migrations, onboarding the new >> > Airflow >> > > > >(with a slightly different architecture), etc., and effectively >> > Airflow >> > > 2 >> > > > >would live in parallel for a long time. >> > > > > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc are >> > > > valuable >> > > > >projects of their own, and I could imagine all of them being >> delivered >> > > as >> > > > >separate AIPs within Airflow 2 (surely new minor versions of Airflow >> > > 2). I >> > > > >am not sure if the scope of changes and the goal we want to achieve >> is >> > > a) >> > > > >clear enough b) broad enough to call for a new major version. >> > > > > >> > > > >Best, >> > > > >Michal >> > > > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T) >> > > > ><jens.scheff...@de.bosch.com.inva <mailto: >> jens.scheff...@de.bosch.com.inva> <mailto:jens.scheff...@de.bosch.com.inva >> <mailto:jens.scheff...@de.bosch.com.inva>>lid> wrote: >> > > > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is mostly a >> > > > vision >> > > > >> statement. >> > > > >> >> > > > >> Looking forward for a larger addendum where we can collect things >> > that >> > > > we >> > > > >> all can vote and agree on as targets. >> > > > >> >> > > > >> As I started earlier with a confluence page and it seems this is >> not >> > > > >> accessible to all, shall we convert this to a Google Doc for >> better >> > > > >> collaboration and item collection? >> > > > >> >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> < >> https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> < >> https://aka.ms/o0ukef&gt;>> >> > > > >> ________________________________ >> > > > >> From: Vikram Koka <vik...@astronomer.io.inva <mailto: >> vik...@astronomer.io.inva> <mailto:vik...@astronomer.io.inva <mailto: >> vik...@astronomer.io.inva>>LID> >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM >> > > > >> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> < >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: >> dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) >> vs >> > > > >> strategic (Airflow 3) approach >> > > > >> >> > > > >> Thank you for your feedback, Bolke and Andrey! >> > > > >> >> > > > >> Bolke, >> > > > >> I have replied to some of your comments in the doc. >> > > > >> I will provide a detailed write up on the "Interactive DAG run" >> (or >> > > > >> synchronous DAG run) capability, which has generated some early >> > > > questions. >> > > > >> I had intended to get an AIP published for that as a follow-up, >> but >> > I >> > > > >> believe that a simpler write up would be useful ahead of the AIP. >> > > > >> >> > > > >> Andrey, >> > > > >> You raise an interesting point. >> > > > >> >> > > > >> As part of the Airflow 2.0 release, we as a community had decided >> to >> > > > >> strictly adhere to Semver as detailed in the document you >> > referenced. >> > > We >> > > > >> also consciously split out the "Core Airflow" releases from the >> > > > "Provider" >> > > > >> releases at that time. We had a clear expectation then for the >> > cadence >> > > > of >> > > > >> both minor and patch releases, which we have generally adhered to >> > > since >> > > > >> then. >> > > > >> >> > > > >> Personally, I am more concerned about our Provider releases right >> > now, >> > > > as >> > > > >> compared to the cadence of our major releases. I believe that one >> of >> > > the >> > > > >> proposed changes in the Airflow 3 document i.e. the clear >> separation >> > > for >> > > > >> Task Execution will help here, but more may be needed. >> > > > >> >> > > > >> Definitely interested in more feedback on this as well. >> > > > >> >> > > > >> Vikram >> > > > >> >> > > > >> >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin < >> > > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> <mailto: >> andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>> >> > > > > >> > > > >> wrote: >> > > > >> >> > > > >> > I would like to propose to change (at least discuss) release >> > policy >> > > > >> around >> > > > >> > the Major version of Airflow. >> > > > >> > >> > > > >> > Right now it is described as "These releases do not happen with >> > any >> > > > >> regular >> > > > >> > interval or on any predictable schedule." : >> > > > >> > >> > > > >> > >> > > > >> >> > > > >> > > >> > >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 >> < >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0> >> < >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0> >> < >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> >> ;> >> > > > >> < >> > > > >> >> > > > >> > > >> > >> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release >> < >> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> >> < >> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> >> < >> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> >> ;> >> > > > >> > >> > > > >> > >> > > > >> > So maybe it is time to make it schedulable, e.g. one per two >> years >> > > or >> > > > so. >> > > > >> > This one could help us to avoid such a discussion in the future, >> > > like >> > > > "We >> > > > >> > don't know when Airflow 4 is coming.". At the moment when the >> new >> > > > major >> > > > >> > version will be released new features wouldn't be added in the >> old >> > > > major >> > > > >> > version, however we would support bug / security for a while, >> > e.g. 1 >> > > > year >> > > > >> > for bug fixes, 3 years for security fixes with a total 5 year >> > > > lifecycle >> > > > >> per >> > > > >> > a major version. These just are approximate time periods for a >> > > > definition >> > > > >> > of current period, bugfix period and security fix period. >> > > > >> > >> > > > >> > In contributors' perspective it helps with dropping the >> deprecated >> > > > stuff >> > > > >> > which resolves some old problem: we have to support everything >> > > > including >> > > > >> > deprecated stuff and without schedulable lifecycle for the >> > > deprecated >> > > > >> stuff >> > > > >> > it could be showstopper for the new feature, because sometimes >> it >> > > > hard to >> > > > >> > support two different approaches for long period of time with no >> > > hope >> > > > >> that >> > > > >> > it will happen soon. For some fundamental stuff which do not >> > > require a >> > > > >> lot >> > > > >> > things time to support we could postponed removal for next after >> > the >> > > > next >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5 >> > > > >> > >> > > > >> > In the user perspective, they have at least bug fix support for >> a >> > > > while, >> > > > >> if >> > > > >> > someone want to use legacy version it their choice, however no >> new >> > > > >> > features, no new version of providers (after one year) >> > > > >> > >> > > > >> > >> > > > >> > ---- >> > > > >> > Best Wishes >> > > > >> > *Andrey Anshin* >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bdbr...@gmail.com >> <mailto:bdbr...@gmail.com> <mailto:bdbr...@gmail.com <mailto: >> bdbr...@gmail.com>>> >> > > > wrote: >> > > > >> > >> > > > >> > > I have left several comments :-). And on interactive dag runs >> > even >> > > > >> after >> > > > >> > > the explanation of Vikram I still don't have a clue what we >> want >> > > to >> > > > >> > > accomplish there :-P. >> > > > >> > > >> > > > >> > > I would like to see a mantra or team for Airflow 3. That helps >> > > > nudging >> > > > >> > > people in the same direction. Suggestions in the comments. >> > > > >> > > >> > > > >> > > Bolke >> > > > >> > > Sent from my iPhone >> > > > >> > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka >> > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> <mailto: >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>>lid >> > > > > >> > > > >> > > wrote: >> > > > >> > > > >> > > > >> > > > Good point Jed. >> > > > >> > > > I responded back to your comment in the doc as well and very >> > > open >> > > > to >> > > > >> > > > changing the term in the doc. >> > > > >> > > > >> > > > >> > > > Used the term "interactive DAG run" as the ability to invoke >> > or >> > > > >> > trigger a >> > > > >> > > > DAG run through the API, with the expectation of getting >> back >> > a >> > > > >> result >> > > > >> > > > immediately. An alternate term could be a "synchronous DAG >> > run". >> > > > >> > > > >> > > > >> > > > Regardless, this is a significant change so a good term to >> > > > indicate >> > > > >> the >> > > > >> > > > expansion from "batch runs only" is warranted. Very open to >> > > > different >> > > > >> > > terms >> > > > >> > > > here. >> > > > >> > > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham < >> > > > >> > jedcunning...@apache.org <mailto:jedcunning...@apache.org> >> <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org>> >> > > > >> > > > >> > > > >> > > >> wrote: >> > > > >> > > >> >> > > > >> > > >> Very exciting! Looks like we will have a busy period of >> time >> > > > ahead >> > > > >> of >> > > > >> > > us. >> > > > >> > > >> Overall I like the plan so far, especially using this >> year's >> > > > Airflow >> > > > >> > > Summit >> > > > >> > > >> as an opportunity to announce and gather feedback, and the >> > 2025 >> > > > >> > version >> > > > >> > > to >> > > > >> > > >> pitch upgrading. >> > > > >> > > >> >> > > > >> > > >> I left a comment in the doc, but we might want to iterate >> on >> > > the >> > > > >> > > >> terminology we use for high priority or "synchronous" DAG >> > runs >> > > to >> > > > >> > serve >> > > > >> > > LLM >> > > > >> > > >> responses - I find "interactive DAG runs" a bit confusing. >> > > > >> > > >> >> > > > >> > > >> > > > >> > > >> > > > --------------------------------------------------------------------- >> > > > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> <mailto:dev-unsubscr...@airflow.apache.org> <mailto: >> dev-unsubscr...@airflow.apache.org <mailto: >> dev-unsubscr...@airflow.apache.org>> >> > > > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org >> <mailto:dev-h...@airflow.apache.org> <mailto:dev-h...@airflow.apache.org >> <mailto:dev-h...@airflow.apache.org>> >> > > > >> > > >> > > > >> > > >> > > > >> > >> > > > >> >> > > > >> > > >> > >> >> >> >> >> >> >> >>
