Thanks to everyone for the constructive discussion - let's make it even more specific in the weekly syncs! I put a slot for us next Tuesday. I had some challenges with how some of the e-mails are displayed in this thread - please reach out to me if you'd like to be added.
In the meantime, I summarised challenges, opportunities and potential work streams of Airflow 3 from the Composer team perspective. Please have a look and chime in here: https://docs.google.com/document/d/1kfqq3YFtUjFuuiv7KVZRqYgwFu9jcYyW_Mz7UUZYnks/edit?usp=sharing The main themes are on-demand & event driven DAG parsing, performance, isolation and stability improvements, and a big concern of backwards compatibility (+1 to Shubham's point - let's try to make changes in DAGS unnecessary, as opposed to potential changes in Airflow configuration). Best, Michal On Sat, May 11, 2024 at 9:30 AM Mehta, Shubham <shu...@amazon.com.invalid> wrote: > Sorry for joining the discussion late - I was down under the weather for > the past two weeks. > First off, great discussion so far and appreciate the effort and thought > everyone has put into exploring Airflow 3. > > The list from Kaxil, et al looks solid and I'm really looking forward to > more extensive discussions on those and seeing them implemented in Airflow. > They will truly unlock some amazing use cases with Airflow. > > Regarding building for Airflow 2 vs Airflow 3, personally, I think we > should keep pushing critical features for Airflow 2 until the scope for > Airflow 3 is properly defined. For example, we shouldn't pause work on the > multi-team capabilities as there are quite a lot of enterprises rely on > Airflow today, and we have been saying that this feature is coming for > years now. Introducing this in Airflow 3 means users have to jump through > another non-backward compatible release to take advantage of it, which > doesn't feel user focused. Airflow 3 is still at least a year away from > anything substantial shipping. Not every change has to be breaking - we can > develop some things for Airflow 2 and potentially adopt them as-is in > Airflow 3 if it makes sense. Only if we're certain the dev effort will be > way less by starting fresh on Airflow 3, should we consider that path. > > On Jarek's point about being very opinionated in Airflow 3.0 and > potentially dropping certain features/options to start with a lean core - I > am very divided on this. In most cases, I would say that being opinionated > is great, but in this case, we risk alienating our users, especially those > who self-host as they have proportionately less representation among the > maintainers. Thousands of organizations rely on Airflow and breaking the > trust could be costly in the long run. We should of course be basing those > decisions on real user data versus assumptions, but not everyone is going > to jump to 2.10+ in a few months and data will be biased. I am not > concerned about MWAA customers here, but more about users who are > self-hosting. > > We've put together a Google doc (with writing access for all) capturing > some wish list items based on customer discussions: > https://docs.google.com/document/d/17KKjwmwHquL_laMpsqZaZD5wzruKAAyFeOsLhGNp7vc/edit?usp=sharing > . It's still a work-in-progress as we gather more feedback from our > customers on what could help increase Airflow adoption in their orgs. We > understand that not everything in this list will be implemented, but it is > essential to bring forward all ideas and discuss to make Airflow better for > everyone. > > I think it would be great to collaborate with others including Jarek, > Kaxil, Jens, Bolke, Amogh etc. who have already or would like to share > ideas, to brainstorm tenets, major themes, scope of Airflow 3, along with > prioritized list. Maybe we can have a dedicated sync, say next Thursday > 8:30am PT, to discuss this further? I can volunteer to organize these > meetings. Initial set of meetings will be to brainstorm to bring forward a > comprehensive and aligned proposal forward for the community. > > I'm excited about Airflow 3 and looking forward to shaping Airflow's > future together! > > Shubham > > > On 2024-05-10, 8:45 AM, "Pierre Jeambrun" <pierrejb...@gmail.com <mailto: > pierrejb...@gmail.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > Thank you for the work and proposal made around airflow 3. > > > I also strongly believe that those important changes need to happen for > airflow to stay relevant and it would be the right moment to start > discussing and planning for airflow 3. As mentioned some features are hard > to develop within airflow 2 with backward compatibility in mind: > * I would love to see a full react front end, without FAB. > * We could also take the opportunity to rework and clarify our RBAC code, > we had quite a number of CVEs related to it. > * Do the breaking change for api clients and (finally) upgrade the openapi > client generator to a more recent version > * Extract business service layers that could be shared across the entire > codebase instead of having a 'component wise' approach. Such that handling > of dag dates (execution, logical), serialization, resource access control > and such tools could be implemented once and shared everywhere. (easier > maintenance, more consistency between API, CLI, and other core components > behaviors). Inspired by some famous software architecture such as hexagonal > or clean. Even if I don't think that those would be relevant to airflow > codebase stricto sensu. > > > As mentioned stability for our users is key and supporting airflow 2.x for > a while after airflow 3 is a must have. I just want to bring to > your attention that this means more work for our release managers and also > for the security team (backporting security patches, CVEs handling for two > different major versions, etc.). All that to say that we should choose this > period wisely. (long enough, but not too long) > > > This sounds like a really exciting time for airflow. > Cheers > > > Le jeu. 9 mai 2024 à 11:23, Amogh Desai <amoghdesai....@gmail.com <mailto: > amoghdesai....@gmail.com>> a écrit : > > > > Thanks for initiating this discussion @Kaxil Naik <ka...@astronomer.io > <mailto:ka...@astronomer.io>>! > > > > > > Read through the document and the emails here and I like the direction in > > which > > we are proceeding. > > > > I also agree that removing code is almost as important as adding code. We > > need to have a clear checklist of what we want to > > remove and get a vote for the same from the community. > > > > I also see that we have multiple ways of doing a thing, which is not > needed > > (serialising for example) and removing > > which can lead to simpler interfaces for users, and contributors too. > > > > I also find the *secret masker* overly complex personally, why can't it > be > > simpler :) > > > > However, I have a small concern with the proportion of the users that > would > > like to migrate to an early version of > > Airflow 3 if we do not support various deployment modes. If we dont get > > enough initial feedback, we do not know how the new > > major is doing, no? > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <a...@apache.org > <mailto:a...@apache.org>> wrote: > > > > > Strong agree - I think one of the things I regret most about Airflow 2 > > was > > > that we didn't remove enough. It probably eased the upgrade process for > > > users though. > > > > > > > but I was hoping one day when we remove providers > > > > from main development, I can do it personally and beat Ash to it. > > > > > > Grrr ;l > > > > > > The other thing I want to look at (but don't have concrete ideas of > yet) > > > is some of the lesser used dag/task concurrency and dependency > features - > > > if we can remove a few things and get a faster scheduler than it's > worth > > > having a discussion over. > > > > > > On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com <mailto: > ja...@potiuk.com>> wrote: > > > >I hope more voices will be coming soon as well :). > > > > > > > >Constance - few words of explanation, for my "remove" things. I do not > > say > > > >we "have to" remove those things or even that we "should". > > > > > > > >My main motivation for mentioning the list is that we often forget > that > > > >when we have a new major version of software - removing stuff is as > > > >important as adding. So I think when it comes to final decisions about > > > >Airflow 3 we should be well aware about both: what we add and what we > > > >remove. So my main point is - "removal" of things should be as > > important a > > > >part in our future discussions as "adding", and I'd say with 10 years > of > > > >Airflow history, it's more important to remove things than to add > them. > > > > > > > >Side comment: Ash for one mentioned at multiple occasions how proud he > > is > > > >that his contributions to Airflow are net-negative - he removed more > > code > > > >than added. Unfortunately it's not shown any more here > > > >https://github.com/apache/airflow/graphs/contributors < > https://github.com/apache/airflow/graphs/contributors> - because we have > > > >more than 10.000 commits, but I was hoping one day when we remove > > > providers > > > >from main development, I can do it personally and beat Ash to it. > > > > > > > >Just wanted to stress a few things on why I think "discussing > potential > > > >removals" is important: > > > > > > > >* I think it is super important to deliver Airflow 3 really fast. It > > took > > > > > > > >1.5 year to release Airflow 2 and it was far too long and keeping > > Airflow > > > 1 > > > >and 2 in parallel was a huge pain and drain and it slowed us down > > > >enormously. We should absolutely limit the time when we actively > develop > > > >both Airflow 2 features and Airflow 3 features. > > > >* I think most of the current deprecations of the 100+ we have does > not > > > >slow us down AT ALL. Removal of those is mostly cosmetic change, a > > little > > > >bit clutter to remove but they have little-to-no impact on actual > speed > > of > > > >development, it's just an old code that keeps on being around > > > >* on the other hand - some of the things I mentioned ARE slowing us > > down A > > > >LOT and will continue to do so until we remove them. For example, I > > > believe > > > >"postgresql + mysql + sqlite complete versioning solution for all the > > > >possible variants of storage" will be quite a bit more complex and > > testing > > > >will take far more time than if we drop mysql and choose a single > > storage > > > >approach for Airflow 3.0. I have no hard data to back it up, but my > gut > > > >feeling is that it can take at least twice as long to get it out in > the > > > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 > for > > > >all the options we have there. > > > >* the telemetry will be cool - but even if we add it for 2.10, the > first > > > >time meaningful data will be available is maybe 2 years from now when > we > > > do > > > >Airflow 4. For now we can completely forget about it because it's only > > > >going to show us some early adopters of Airflow 2.10. But we have > > surveys > > > + > > > >we can collect data from Astronomer, Google. Amazon for some usage and > > > >identify who will be early adopters of Airflow 3 and target them > first. > > > >* the fact that we will drop something for Airflow 3.0, does not mean > > that > > > >we have to drop it "forever". We can safely assume that Airflow 3.0 > will > > > be > > > >only used by early adopters, and many of our users will wait for 3.1, > > 3.2 > > > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those > > "slower > > > >moving users" from the 3.0. We can re-add things in 3.1 building on > > > >foundations of 3.0 after it is in the hands of those early adopters. I > > > >personally think we should prioritize speed of delivery of 3.0 over > > > >"complete support of every deployment option of Airflow 2" - to allow > > > >**some** of our early adopter users to migrate quickly, and add what's > > > >missing for those who would like to move later in later versions. > > > >* I think - this is the most important "product management" decision > > here > > > - > > > >based on the data we have available. Which users do we target for > > Airflow > > > >3.0, and what we can drop to deliver it faster (and possibly re-add > more > > > >stuff for 3.1+). I think it's crucial to the success of the Airflow > 3.0 > > > >initiative and the most important decision from the Product management > > > side > > > >that will impact the timeline of Airflow 3.0. While a lot depends on > how > > > >much time and effort individuals will spend on implementing things, > the > > > >decision of what we drop will impact everyone's speed and overall > > delivery > > > >of Airflow 3.0. It's a decision we should not take lightly. > > > > > > > >So my main point is: let's add "what we remove" as a very important > > point > > > >in the product discussions we are going to have. > > > > > > > >J. > > > > > > > > > > > > > > > > > > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar > > > ><rbish...@amazon.com.inva <mailto:rbish...@amazon.com.inva>lid> > wrote: > > > > > > > >> Like Constance and many others, there was time needed to process > this > > > >> fantastical summary and approach to Airflow 3. __ > > > >> A lot of the points raised by Jarek make sense, ie being > prescriptive > > on > > > >> what features we want to be there on Day 1 vs launched on 3.x - this > > > does > > > >> allow us to move quickly for our users. > > > >> In line with Constance's concern, we need to be mindful of what we > > > decide > > > >> to keep and what we are willing to cut - which could be a > painstaking > > > >> process that could offset any gains of trying to be faster. I also > > > believe > > > >> that bringing all these new amazing features on Airflow 3 will peak > > the > > > >> interest of early adopters and eventually get others interested in > > > >> migration. However, I believe this migration will be a slow process > > and > > > >> will present a gap in certain functionalities that users may want > > before > > > >> entertaining any move to Airflow 3. There are still a lot of folks > > using > > > >> v1.10 today. There were several tactical initiatives in the past few > > > months > > > >> with intent on bringing new functionality, ie Multi-team, to Airflow > > 2.x > > > >> and I feel that while these efforts are not wasted, there should > still > > > be > > > >> an option to continue improving Airflow 2 to avoid alienating our > > users > > > on > > > >> the basis of a future promise in Airflow 3, that may not be easy to > > > migrate > > > >> towards. > > > >> > > > >> -- Rajesh > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > dev@airflow.apache.org <mailto:dev@airflow.apache.org>> < > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> > > > >> > > > >> > > > >> On 2024-05-07, 12:26 PM, "Constance Martineau" > > > >> <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> > <mailto:consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva > >> > > > >> <mailto:consta...@astronomer.io.inva <mailto: > consta...@astronomer.io.inva> <mailto: > > > consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva > >>>LID> > > > >> wrote: > > > >> > > > >> > > > >> > > > >> > > > >> CAUTION: This email originated from outside of the organization. Do > > not > > > >> click links or open attachments unless you can confirm the sender > and > > > know > > > >> the content is safe. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > > pouvez > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain > > > que > > > >> le contenu ne présente aucun risque. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Thank you Jarek for the detailed input. I've taken some time to > digest > > > your > > > >> points before responding. > > > >> > > > >> > > > >> > > > >> > > > >> You've outlined a bold vision for Airflow 3, and I agree that being > > > >> decisive about the features and architectures will be the key to > > > success. > > > >> However, before we make final decisions on what features to cut or > > > retain, > > > >> it would be beneficial to have a more comprehensive understanding of > > how > > > >> the current features are utilized by the open-source community. > > > >> > > > >> > > > >> > > > >> > > > >> @Kaxil Naik <ka...@astronomer.io <mailto:ka...@astronomer.io> > <mailto:ka...@astronomer.io <mailto:ka...@astronomer.io>> > > <mailto: > > > >> ka...@astronomer.io <mailto:ka...@astronomer.io> <mailto: > ka...@astronomer.io <mailto:ka...@astronomer.io>>>> recently > > initiated a > > > >> discussion on > > > >> collecting telemetry from open-source deployments: > > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>> < > > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>> < > > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> > <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt> > > ;>. > > > >> This data > > > >> could be critical in ensuring our decisions are well-informed and > > > reflect > > > >> the real-world usage patterns of our users, not just those from > > managed > > > >> environments like Astro, MWAA or GCC. > > > >> > > > >> > > > >> > > > >> > > > >> It's essential that we challenge our assumptions and base our > > decisions > > > on > > > >> a holistic view of feature usage. Identifying potential cuts is a > > > critical > > > >> step, but let's ensure our strategy aligns with the needs and > > > preferences > > > >> of the broader Airflow community. > > > >> > > > >> > > > >> > > > >> > > > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <ja...@potiuk.com > <mailto:ja...@potiuk.com> > > <mailto: > > > >> ja...@potiuk.com <mailto:ja...@potiuk.com>> <mailto: > ja...@potiuk.com <mailto:ja...@potiuk.com> <mailto:ja...@potiuk.com > <mailto:ja...@potiuk.com> > > >>> > > > >> wrote: > > > >> > > > >> > > > >> > > > >> > > > >> > I am currently on sick leave, and still recovering - hoping to be > > > able to > > > >> > travel next week to the US as planned, so I just wanted to break > out > > > of > > > >> it > > > >> > to make one comment here. > > > >> > > > > >> > I got a clearer head now a bit with medications hopefully > working. I > > > am > > > >> > still taking it that should help me to get over the current state, > > > and I > > > >> > wanted to take a look at this discussion unraveling first. Over > last > > > week > > > >> > I disconnected from "day-to-day" Airflow and put some thoughts (as > > > much > > > >> as > > > >> > I could in my current state) on it. The whole subject of this > thread > > > was > > > >> > started from that - how the current discussions on AIP-67 and > others > > > >> change > > > >> > if we consider Airflow 3 is "starting". > > > >> > > > > >> > The price for back-compat is speed of development and quality. > More > > > >> > combinations to test, more unexpected issues uncovered, necessity > to > > > keep > > > >> > parallel paths (old/new) while adding new features. All what > > Constance > > > >> > wrote about and what Ash explained. We already started to trip > over > > > our > > > >> own > > > >> > feet mutliple times in a few last releases. Have we tested all > > > >> combinations > > > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we > > already > > > see > > > >> > that in a number of "combos" of features things are not working in > > as > > > >> > stable a way as they did before. > > > >> > > > > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 > for a > > > long > > > >> > time (or even move out) as they will not want to move to Airflow > 3. > > A > > > lot > > > >> > of the work implemented in AIP-44 and design of AIP-67 was done > > around > > > >> > back-compatibility. but yes - > > > >> > it would have been way easier if designed anew without > > > back-compatibility > > > >> > in mind. And if we implement it and release it in Airflow 2 it > will > > > make > > > >> > new Airflow feature development even harder. That's why I wanted > to > > > treat > > > >> > it as "tactical" solution - hoping that in Airflow 3 we can make > it > > > >> > "properly" - and that's why I started the discussion here when I > > > sensed > > > >> > that we are "close" to Airflow 3 discussion, because I wanted to > see > > > what > > > >> > options we have there. This is why I have not yet concluded voting > > on > > > >> > AIP-67 waiting for the result of this discussion here. > > > >> > > > > >> > But if we are ready to go for Airflow.3 then I'd say there are two > > > >> > important things that should be part of the vision. > > > >> > > > > >> > 1) *We should be far more opinionated and have far fewer options > of > > > >> > running things in Airflow 3*. Even an order of magnitude more > > > >> opinionated. > > > >> > Make choices, stick to it, perfect those opinionated choices to > suit > > > >> 80/20 > > > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not > > > fitting > > > >> > the 20% that might choose to stay at Airflow 2. We can choose now > > > which > > > >> > ~20% of cases we do not want to handle deliberately. And we should > > be > > > >> very, > > > >> > very strict about it. Default should be "no choice". This will > > > radically > > > >> > simplify deployment and should make it easier to simplify Airflow > > > >> > development and DAG authoring experience because we will have less > > > cases > > > >> to > > > >> > support. Even if we plan to add more options in the future, the > > first > > > >> > version of Airflow 3 should support one deployment approach only. > > > This is > > > >> > the only way we can deliver it fast. And we should be very bold > > there. > > > >> > Choose one option and go for it in pretty much every place we have > > > >> choices > > > >> > now. We should Aim for Airflow 3.0 to support only a subset of > > current > > > >> > users - but those who are most likely to migrate first and those > > with > > > the > > > >> > biggest need for the new features. We can think 3.x to support > more > > > >> cases, > > > >> > but 3.0 should be as opinionated as humanly possible. > > > >> > > > > >> > And this deployment option should be also something ALL our > > > stakeholders > > > >> > will feel OK with as a way forward in their offering. > > > >> > > > > >> > My candidates (and yes, some are bold): > > > >> > > > > >> > * *Drop MySQL*. If we have a single thing that makes us avoid our > > > schema > > > >> > and DB migration - this is the case. Let's choose Postgres 15+ and > > use > > > >> some > > > >> > of the great features there. This will also enable much faster > async > > > SQL > > > >> > implementation and a number of other optimisations - not to > mention > > > >> cutting > > > >> > every single change in development and testing time by literally > > half. > > > >> And > > > >> > we should not look back to adding MySQL. > > > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S > only > > > (and > > > >> > AWS/Google others can continue developing theirs of course in > > parallel > > > >> and > > > >> > continue Hybrid executor work). Later - we figure out a better > > > solution > > > >> to > > > >> > support "small" tasks using some new K8S features and possibly > > non-k8s > > > >> > solutions (Ray-based?) > > > >> > * *Cut Connection and Variable Management from DB/UI*. Leave only > > > Secrets > > > >> > Management. Later when we have a 100% extensible React UI, we can > > add > > > a > > > >> > "local DB secrets manager" add-on > > > >> > * *Choose a single way for DAG storage that will support > versioning > > > from > > > >> > day one*. Bear in mind we can add others later. Bolke's idea of > > using > > > >> > FSspec is an interesting one, we should see if it is feasible. > > > >> > * *Drop FAB completely (including custom plugins) and invest in > > > >> > implementing Auth Manager based on a dedicated, external solution* > > > >> > (KeyCloak > > > >> > that we've discussed before as a likely candidate) > > > >> > * *Leave Providers with Airflow 2 and add tests to make sure they > > are > > > >> > Airflow 3 future-compatible *- develop a way where we continue > > > >> development > > > >> > and contributions for Providers with Airflow 2 and add complete > > tests > > > to > > > >> > run them with Airflow 3. This way we can continue developing > > Provider > > > >> > features independently, and make them work for Airflow 2 (and > > continue > > > >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes), > > > while > > > >> > also gradually fix any Airflow3 incompatibilities and instead of > > > >> > "back-compatibility" tests make provider "forward-compatibility" > > > tests so > > > >> > that future Providers are tested and work on Airflow 3. Also it > will > > > make > > > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested > > without > > > >> > investing in changing the current CI / test harness. > > > >> > * *Simplify Test Harness for Airflow 3 from the start *- without > > > >> providers > > > >> > and 790+ dependencies, we could vastly simplify Airflow3 testing > > > >> (basically > > > >> > make CI jobs from scratch) using mostly standard Python tooling > > > (while we > > > >> > can continue making use of the current test harness for Airflow 2 > + > > > >> > Providers and extend it with Airflow 3 future-compatibility > tests). > > > That > > > >> > means Breeze would be only staying in Airflow 2 + Providers repo > as > > we > > > >> > should be able to achieve most of what we have there with local > > venv/ > > > >> > tooling (especially with uv as underlying tooling). > > > >> > > > > >> > 2) *I think we only add very few new "important" features. > *Absolute > > > >> > minimum to make Airflow 3 appealing and add them only in Airflow > 3: > > > >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it > > > makes > > > >> no > > > >> > sense to invest into Airflow 2 if we already know Airflow 3 is > > coming > > > - > > > >> > that generally triples effort needed to get them out. We should > drop > > > new > > > >> > features development in Airflow 2. This will give users incentive > to > > > move > > > >> > to 3 if the new features will be worth it. Even paying > > > >> > compatibility/migration price. > > > >> > > > > >> > Versionig, for example: I believe if we decide to go only with > > > Airflow 3 > > > >> > and cut some of the above (Postgres only, Single versioning DAG > > > storage) > > > >> we > > > >> > can make bolder decisions in versioning and support simpler models > > > from > > > >> the > > > >> > get go (and deliver it faster). And we should add only a few - but > > > >> > important - features that our users clearly asked for and focus on > > > >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 > or > > > >> 2.11). > > > >> > Similarly - multi-team can be simplified if we cut things from the > > > list > > > >> > above and have Task isolation as first-class citizens in Airflow > > (and > > > the > > > >> > only option). > > > >> > > > > >> > My candidates very much concur with the list shared by Kaxil in > the > > > doc + > > > >> > I'd add multi-team (but simplified thanks to the cuts). But I also > > > here > > > >> > would mostly revert to Astronomer, Google. AWS team to define > > > >> collectively > > > >> > what is the absolute minimum set of features that would get the > > > "target" > > > >> > part of their customers happy. And ONLY do that. > > > >> > > > > >> > So in short - I think the big part of our discussion should be > what > > we > > > >> are > > > >> > ready to drop when we start airflow 3 and be very bold. Once we > know > > > we > > > >> > should figure out the absolute minimum of things that we can add > > that > > > >> will > > > >> > benefit a significant part of our users (and make use of increased > > > speed > > > >> > because we dropped things). > > > >> > > > > >> > J. > > > >> > > > > >> > > > > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau > > > >> > <consta...@astronomer.io.inva <mailto: > consta...@astronomer.io.inva> <mailto:consta...@astronomer.io.inva > <mailto:consta...@astronomer.io.inva>> > > > >> <mailto:consta...@astronomer.io.inva <mailto: > consta...@astronomer.io.inva> <mailto: > > > consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva > >>>lid> > > > >> wrote: > > > >> > > > > >> > > Hi Michal, > > > >> > > > > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate > > > your > > > >> > > concerns about the migration overhead for our users with a major > > new > > > >> > > version and see the appeal in your suggestion to integrate many > of > > > the > > > >> > > proposed changes into Airflow 2 through separate AIPs. It’s a > > valid > > > >> point > > > >> > > and certainly aligns with the value of making incremental > > > improvements. > > > >> > > > > > >> > > However, after looking closely at the enhancements outlined for > > > Airflow > > > >> > 3, > > > >> > > I'm convinced they warrant a new major release. Here’s why: > > > >> > > > > > >> > > 1. *Core Architectural Changes:* We’re looking at foundational > > > changes > > > >> > > with Airflow 3—like redefining task priorities, separating task > > > >> > > definition > > > >> > > and task execution, and new AIPs like DAG versioning. remote > > > execution > > > >> > > and restricting database access from workers. These aren’t just > > > >> > > incremental > > > >> > > improvements but major shifts that will set the stage for the > next > > > >> > > decade > > > >> > > of Airflow’s architecture. Grouping these changes into a major > > > release > > > >> > > will > > > >> > > help us make these transitions more cleanly and with fewer > > > constraints > > > >> > > from > > > >> > > past decisions. > > > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140 > > > >> > deprecated > > > >> > > issues, and this will only grow if we continue without a major > > > >> > cleanup. > > > >> > > This makes it increasingly difficult to implement new features > > > >> > > effectively > > > >> > > while maintaining backward compatibility. A major release allows > > us > > > to > > > >> > > address these issues head-on, reducing technical debt and paving > > the > > > >> > way > > > >> > > for a more robust platform. > > > >> > > 3. *Managing Breaking Changes:* Let’s take the example of > > > restricting > > > >> > > database access from workers. It’s a necessary move for better > > > >> > security > > > >> > > and > > > >> > > also potentially scalability reasons (reduces DB load). Many > users > > > >> > have > > > >> > > workflows that interact with the DB, either by using raw sql or > by > > > >> > > leveraging a session object. We could implement this feature in > > > >> > Airflow > > > >> > > 2 > > > >> > > and avoid breaking existing workflows by continuing to have the > > old > > > >> > > standard mode as default - much of the work is already done - > but > > > that > > > >> > > would mean supporting both the new secure mode and the old > > standard > > > >> > mode > > > >> > > indefinitely and design new features with the assumption that > most > > > >> > will > > > >> > > continue using the old standard mode. With Airflow 3, we can > make > > > >> > secure > > > >> > > mode the default or even the only option, simplifying > > implementation > > > >> > and > > > >> > > future development. This is just one example where it is > feasible > > to > > > >> > > implement in Airflow 2, but is better if we release it under the > > > >> > > context of > > > >> > > Airflow 3. > > > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up > > > >> > > possibilities for handling workflows beyond batch processing. > > > Features > > > >> > > like > > > >> > > real-time DAG execution through API and multi-language task > > support > > > >> > are > > > >> > > big > > > >> > > steps forward, significantly expanding Airflow’s utility. > > > >> > > > > > >> > > > > > >> > > While integrating these updates into Airflow 2 might look less > > > >> disruptive > > > >> > > initially, the scale and nature of the required changes really > > > support > > > >> a > > > >> > > move to Airflow 3. It’s not just about adding new features; it’s > > > about > > > >> > > setting up Airflow so that it continues to remain relevant for > the > > > next > > > >> > ten > > > >> > > years. > > > >> > > > > > >> > > Constance > > > >> > > > > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor < > a...@apache.org <mailto:a...@apache.org> > > > >> <mailto:a...@apache.org <mailto:a...@apache.org>> <mailto: > a...@apache.org <mailto:a...@apache.org> <mailto:a...@apache.org <mailto: > a...@apache.org> > > > >>> > > > >> wrote: > > > >> > > > > > >> > > > There's a lot of technical debt hiding in Airflow, especially > > the > > > >> > > > scheduler that makes it harder and harder to efficiently add > new > > > >> > > features. > > > >> > > > > > > >> > > > At some point, very soon, we are going to have to remove some > > very > > > >> > > > infrequently used back compat shims that negatively affect > > > >> performance. > > > >> > > > Without doing that the pace at which we can realistically add > > > some of > > > >> > the > > > >> > > > more exciting features tends towards zero. Developer speed of > > > >> > > contributors > > > >> > > > is a factor here too! > > > >> > > > > > > >> > > > So while we are still using SemVer, that necessitates v3. > > > >> > > > > > > >> > > > Ash > > > >> > > > > > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" < > > > michalmod...@google.com <mailto:michalmod...@google.com> > > > >> <mailto:michalmod...@google.com <mailto:michalmod...@google.com>> > <mailto:michalmod...@google.com <mailto:michalmod...@google.com> > > > <mailto: > > > >> michalmod...@google.com <mailto:michalmod...@google.com>>> > > > >> > > .INVALID> > > > >> > > > wrote: > > > >> > > > >+1 to Jens's & Bolke's points here and in the doc > > > >> > > > > > > > >> > > > >I agree we should work on clarifying the directions we would > > like > > > >> > > Airflow > > > >> > > > >to go. Introducing a new major Airflow version is a massive > > > overhead > > > >> > for > > > >> > > > >users, who would need to plan for migrations, onboarding the > > new > > > >> > Airflow > > > >> > > > >(with a slightly different architecture), etc., and > effectively > > > >> > Airflow > > > >> > > 2 > > > >> > > > >would live in parallel for a long time. > > > >> > > > > > > > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's > doc > > > are > > > >> > > > valuable > > > >> > > > >projects of their own, and I could imagine all of them being > > > >> delivered > > > >> > > as > > > >> > > > >separate AIPs within Airflow 2 (surely new minor versions of > > > Airflow > > > >> > > 2). I > > > >> > > > >am not sure if the scope of changes and the goal we want to > > > achieve > > > >> is > > > >> > > a) > > > >> > > > >clear enough b) broad enough to call for a new major version. > > > >> > > > > > > > >> > > > >Best, > > > >> > > > >Michal > > > >> > > > > > > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens > > (XC-AS/EAE-ADA-T) > > > >> > > > ><jens.scheff...@de.bosch.com.inva <mailto: > jens.scheff...@de.bosch.com.inva> <mailto: > > > >> jens.scheff...@de.bosch.com.inva <mailto: > jens.scheff...@de.bosch.com.inva>> <mailto: > > > jens.scheff...@de.bosch.com.inva <mailto: > jens.scheff...@de.bosch.com.inva> > > > >> <mailto:jens.scheff...@de.bosch.com.inva <mailto: > jens.scheff...@de.bosch.com.inva>>>lid> wrote: > > > >> > > > > > > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is > > > mostly a > > > >> > > > vision > > > >> > > > >> statement. > > > >> > > > >> > > > >> > > > >> Looking forward for a larger addendum where we can collect > > > things > > > >> > that > > > >> > > > we > > > >> > > > >> all can vote and agree on as targets. > > > >> > > > >> > > > >> > > > >> As I started earlier with a confluence page and it seems > this > > > is > > > >> not > > > >> > > > >> accessible to all, shall we convert this to a Google Doc > for > > > >> better > > > >> > > > >> collaboration and item collection? > > > >> > > > >> > > > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> < > https://aka.ms/o0ukef>> < > > > >> https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>> < > https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>> < > > > >> https://aka.ms/o0ukef&gt;>> < > https://aka.ms/o0ukef&amp;gt;&gt;>> > > > >> > > > >> ________________________________ > > > >> > > > >> From: Vikram Koka <vik...@astronomer.io.inva <mailto: > vik...@astronomer.io.inva> <mailto: > > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>> > <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > <mailto: > > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>>>LID> > > > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM > > > >> > > > >> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> > > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org> > <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> < > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > dev@airflow.apache.org <mailto:dev@airflow.apache.org>> <mailto: > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > dev@airflow.apache.org <mailto:dev@airflow.apache.org>>>> > > > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical > (Airflow > > > 2) > > > >> vs > > > >> > > > >> strategic (Airflow 3) approach > > > >> > > > >> > > > >> > > > >> Thank you for your feedback, Bolke and Andrey! > > > >> > > > >> > > > >> > > > >> Bolke, > > > >> > > > >> I have replied to some of your comments in the doc. > > > >> > > > >> I will provide a detailed write up on the "Interactive DAG > > run" > > > >> (or > > > >> > > > >> synchronous DAG run) capability, which has generated some > > early > > > >> > > > questions. > > > >> > > > >> I had intended to get an AIP published for that as a > > follow-up, > > > >> but > > > >> > I > > > >> > > > >> believe that a simpler write up would be useful ahead of > the > > > AIP. > > > >> > > > >> > > > >> > > > >> Andrey, > > > >> > > > >> You raise an interesting point. > > > >> > > > >> > > > >> > > > >> As part of the Airflow 2.0 release, we as a community had > > > decided > > > >> to > > > >> > > > >> strictly adhere to Semver as detailed in the document you > > > >> > referenced. > > > >> > > We > > > >> > > > >> also consciously split out the "Core Airflow" releases from > > the > > > >> > > > "Provider" > > > >> > > > >> releases at that time. We had a clear expectation then for > > the > > > >> > cadence > > > >> > > > of > > > >> > > > >> both minor and patch releases, which we have generally > > adhered > > > to > > > >> > > since > > > >> > > > >> then. > > > >> > > > >> > > > >> > > > >> Personally, I am more concerned about our Provider releases > > > right > > > >> > now, > > > >> > > > as > > > >> > > > >> compared to the cadence of our major releases. I believe > that > > > one > > > >> of > > > >> > > the > > > >> > > > >> proposed changes in the Airflow 3 document i.e. the clear > > > >> separation > > > >> > > for > > > >> > > > >> Task Execution will help here, but more may be needed. > > > >> > > > >> > > > >> > > > >> Definitely interested in more feedback on this as well. > > > >> > > > >> > > > >> > > > >> Vikram > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin < > > > >> > > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> > <mailto:andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>> > > <mailto: > > > >> andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> <mailto: > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>>> > > > >> > > > > > > > >> > > > >> wrote: > > > >> > > > >> > > > >> > > > >> > I would like to propose to change (at least discuss) > > release > > > >> > policy > > > >> > > > >> around > > > >> > > > >> > the Major version of Airflow. > > > >> > > > >> > > > > >> > > > >> > Right now it is described as "These releases do not > happen > > > with > > > >> > any > > > >> > > > >> regular > > > >> > > > >> > interval or on any predictable schedule." : > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > < > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > > > > >> < > > > >> > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > < > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0 > > > > > > > > > >> < > > > >> > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > < > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0 > > > > > > > > > >> < > > > >> > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> > < > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt > > > > > >> ;> > > > >> > > > >> < > > > >> > > > >> > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > > > >> < > > > >> > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > > > > > > > >> < > > > >> > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > > > > > > > >> < > > > >> > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt > > > > > >> ;> > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > So maybe it is time to make it schedulable, e.g. one per > > two > > > >> years > > > >> > > or > > > >> > > > so. > > > >> > > > >> > This one could help us to avoid such a discussion in the > > > future, > > > >> > > like > > > >> > > > "We > > > >> > > > >> > don't know when Airflow 4 is coming.". At the moment when > > the > > > >> new > > > >> > > > major > > > >> > > > >> > version will be released new features wouldn't be added > in > > > the > > > >> old > > > >> > > > major > > > >> > > > >> > version, however we would support bug / security for a > > while, > > > >> > e.g. 1 > > > >> > > > year > > > >> > > > >> > for bug fixes, 3 years for security fixes with a total 5 > > year > > > >> > > > lifecycle > > > >> > > > >> per > > > >> > > > >> > a major version. These just are approximate time periods > > for > > > a > > > >> > > > definition > > > >> > > > >> > of current period, bugfix period and security fix period. > > > >> > > > >> > > > > >> > > > >> > In contributors' perspective it helps with dropping the > > > >> deprecated > > > >> > > > stuff > > > >> > > > >> > which resolves some old problem: we have to support > > > everything > > > >> > > > including > > > >> > > > >> > deprecated stuff and without schedulable lifecycle for > the > > > >> > > deprecated > > > >> > > > >> stuff > > > >> > > > >> > it could be showstopper for the new feature, because > > > sometimes > > > >> it > > > >> > > > hard to > > > >> > > > >> > support two different approaches for long period of time > > > with no > > > >> > > hope > > > >> > > > >> that > > > >> > > > >> > it will happen soon. For some fundamental stuff which do > > not > > > >> > > require a > > > >> > > > >> lot > > > >> > > > >> > things time to support we could postponed removal for > next > > > after > > > >> > the > > > >> > > > next > > > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in > > > Airflow 5 > > > >> > > > >> > > > > >> > > > >> > In the user perspective, they have at least bug fix > support > > > for > > > >> a > > > >> > > > while, > > > >> > > > >> if > > > >> > > > >> > someone want to use legacy version it their choice, > however > > > no > > > >> new > > > >> > > > >> > features, no new version of providers (after one year) > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > ---- > > > >> > > > >> > Best Wishes > > > >> > > > >> > *Andrey Anshin* > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin < > > > bdbr...@gmail.com <mailto:bdbr...@gmail.com> > > > >> <mailto:bdbr...@gmail.com <mailto:bdbr...@gmail.com>> <mailto: > bdbr...@gmail.com <mailto:bdbr...@gmail.com> <mailto: > > > >> bdbr...@gmail.com <mailto:bdbr...@gmail.com>>>> > > > >> > > > wrote: > > > >> > > > >> > > > > >> > > > >> > > I have left several comments :-). And on interactive > dag > > > runs > > > >> > even > > > >> > > > >> after > > > >> > > > >> > > the explanation of Vikram I still don't have a clue > what > > we > > > >> want > > > >> > > to > > > >> > > > >> > > accomplish there :-P. > > > >> > > > >> > > > > > >> > > > >> > > I would like to see a mantra or team for Airflow 3. > That > > > helps > > > >> > > > nudging > > > >> > > > >> > > people in the same direction. Suggestions in the > > comments. > > > >> > > > >> > > > > > >> > > > >> > > Bolke > > > >> > > > >> > > Sent from my iPhone > > > >> > > > >> > > > > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka > > > >> > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>> > > > <mailto: > > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>>>lid > > > >> > > > > > > > >> > > > >> > > wrote: > > > >> > > > >> > > > > > > >> > > > >> > > > Good point Jed. > > > >> > > > >> > > > I responded back to your comment in the doc as well > and > > > very > > > >> > > open > > > >> > > > to > > > >> > > > >> > > > changing the term in the doc. > > > >> > > > >> > > > > > > >> > > > >> > > > Used the term "interactive DAG run" as the ability to > > > invoke > > > >> > or > > > >> > > > >> > trigger a > > > >> > > > >> > > > DAG run through the API, with the expectation of > > getting > > > >> back > > > >> > a > > > >> > > > >> result > > > >> > > > >> > > > immediately. An alternate term could be a > "synchronous > > > DAG > > > >> > run". > > > >> > > > >> > > > > > > >> > > > >> > > > Regardless, this is a significant change so a good > term > > > to > > > >> > > > indicate > > > >> > > > >> the > > > >> > > > >> > > > expansion from "batch runs only" is warranted. Very > > open > > > to > > > >> > > > different > > > >> > > > >> > > terms > > > >> > > > >> > > > here. > > > >> > > > >> > > > > > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham < > > > >> > > > >> > jedcunning...@apache.org <mailto: > jedcunning...@apache.org> <mailto:jedcunning...@apache.org <mailto: > jedcunning...@apache.org>> > > > >> <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org> > <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org>>> > > > >> > > > >> > > > > > > >> > > > >> > > >> wrote: > > > >> > > > >> > > >> > > > >> > > > >> > > >> Very exciting! Looks like we will have a busy period > > of > > > >> time > > > >> > > > ahead > > > >> > > > >> of > > > >> > > > >> > > us. > > > >> > > > >> > > >> Overall I like the plan so far, especially using > this > > > >> year's > > > >> > > > Airflow > > > >> > > > >> > > Summit > > > >> > > > >> > > >> as an opportunity to announce and gather feedback, > and > > > the > > > >> > 2025 > > > >> > > > >> > version > > > >> > > > >> > > to > > > >> > > > >> > > >> pitch upgrading. > > > >> > > > >> > > >> > > > >> > > > >> > > >> I left a comment in the doc, but we might want to > > > iterate > > > >> on > > > >> > > the > > > >> > > > >> > > >> terminology we use for high priority or > "synchronous" > > > DAG > > > >> > runs > > > >> > > to > > > >> > > > >> > serve > > > >> > > > >> > > LLM > > > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit > > > confusing. > > > >> > > > >> > > >> > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > > > > --------------------------------------------------------------------- > > > >> > > > >> > > To unsubscribe, e-mail: > > dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org> > > > >> <mailto:dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org>> <mailto: > > > >> dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org> <mailto: > > > >> dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org>>> > > > >> > > > >> > > For additional commands, e-mail: > > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > >> <mailto:dev-h...@airflow.apache.org <mailto: > dev-h...@airflow.apache.org>> <mailto: > > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > >> <mailto:dev-h...@airflow.apache.org <mailto: > dev-h...@airflow.apache.org>>> > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > > > > > > >
