Sorry for joining the discussion late - I was down under the weather for the past two weeks. First off, great discussion so far and appreciate the effort and thought everyone has put into exploring Airflow 3.
The list from Kaxil, et al looks solid and I'm really looking forward to more extensive discussions on those and seeing them implemented in Airflow. They will truly unlock some amazing use cases with Airflow. Regarding building for Airflow 2 vs Airflow 3, personally, I think we should keep pushing critical features for Airflow 2 until the scope for Airflow 3 is properly defined. For example, we shouldn't pause work on the multi-team capabilities as there are quite a lot of enterprises rely on Airflow today, and we have been saying that this feature is coming for years now. Introducing this in Airflow 3 means users have to jump through another non-backward compatible release to take advantage of it, which doesn't feel user focused. Airflow 3 is still at least a year away from anything substantial shipping. Not every change has to be breaking - we can develop some things for Airflow 2 and potentially adopt them as-is in Airflow 3 if it makes sense. Only if we're certain the dev effort will be way less by starting fresh on Airflow 3, should we consider that path. On Jarek's point about being very opinionated in Airflow 3.0 and potentially dropping certain features/options to start with a lean core - I am very divided on this. In most cases, I would say that being opinionated is great, but in this case, we risk alienating our users, especially those who self-host as they have proportionately less representation among the maintainers. Thousands of organizations rely on Airflow and breaking the trust could be costly in the long run. We should of course be basing those decisions on real user data versus assumptions, but not everyone is going to jump to 2.10+ in a few months and data will be biased. I am not concerned about MWAA customers here, but more about users who are self-hosting. We've put together a Google doc (with writing access for all) capturing some wish list items based on customer discussions: https://docs.google.com/document/d/17KKjwmwHquL_laMpsqZaZD5wzruKAAyFeOsLhGNp7vc/edit?usp=sharing . It's still a work-in-progress as we gather more feedback from our customers on what could help increase Airflow adoption in their orgs. We understand that not everything in this list will be implemented, but it is essential to bring forward all ideas and discuss to make Airflow better for everyone. I think it would be great to collaborate with others including Jarek, Kaxil, Jens, Bolke, Amogh etc. who have already or would like to share ideas, to brainstorm tenets, major themes, scope of Airflow 3, along with prioritized list. Maybe we can have a dedicated sync, say next Thursday 8:30am PT, to discuss this further? I can volunteer to organize these meetings. Initial set of meetings will be to brainstorm to bring forward a comprehensive and aligned proposal forward for the community. I'm excited about Airflow 3 and looking forward to shaping Airflow's future together! Shubham On 2024-05-10, 8:45 AM, "Pierre Jeambrun" <pierrejb...@gmail.com <mailto:pierrejb...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Thank you for the work and proposal made around airflow 3. I also strongly believe that those important changes need to happen for airflow to stay relevant and it would be the right moment to start discussing and planning for airflow 3. As mentioned some features are hard to develop within airflow 2 with backward compatibility in mind: * I would love to see a full react front end, without FAB. * We could also take the opportunity to rework and clarify our RBAC code, we had quite a number of CVEs related to it. * Do the breaking change for api clients and (finally) upgrade the openapi client generator to a more recent version * Extract business service layers that could be shared across the entire codebase instead of having a 'component wise' approach. Such that handling of dag dates (execution, logical), serialization, resource access control and such tools could be implemented once and shared everywhere. (easier maintenance, more consistency between API, CLI, and other core components behaviors). Inspired by some famous software architecture such as hexagonal or clean. Even if I don't think that those would be relevant to airflow codebase stricto sensu. As mentioned stability for our users is key and supporting airflow 2.x for a while after airflow 3 is a must have. I just want to bring to your attention that this means more work for our release managers and also for the security team (backporting security patches, CVEs handling for two different major versions, etc.). All that to say that we should choose this period wisely. (long enough, but not too long) This sounds like a really exciting time for airflow. Cheers Le jeu. 9 mai 2024 à 11:23, Amogh Desai <amoghdesai....@gmail.com <mailto:amoghdesai....@gmail.com>> a écrit : > Thanks for initiating this discussion @Kaxil Naik <ka...@astronomer.io > <mailto:ka...@astronomer.io>>! > > > Read through the document and the emails here and I like the direction in > which > we are proceeding. > > I also agree that removing code is almost as important as adding code. We > need to have a clear checklist of what we want to > remove and get a vote for the same from the community. > > I also see that we have multiple ways of doing a thing, which is not needed > (serialising for example) and removing > which can lead to simpler interfaces for users, and contributors too. > > I also find the *secret masker* overly complex personally, why can't it be > simpler :) > > However, I have a small concern with the proportion of the users that would > like to migrate to an early version of > Airflow 3 if we do not support various deployment modes. If we dont get > enough initial feedback, we do not know how the new > major is doing, no? > > Thanks & Regards, > Amogh Desai > > > On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <a...@apache.org > <mailto:a...@apache.org>> wrote: > > > Strong agree - I think one of the things I regret most about Airflow 2 > was > > that we didn't remove enough. It probably eased the upgrade process for > > users though. > > > > > but I was hoping one day when we remove providers > > > from main development, I can do it personally and beat Ash to it. > > > > Grrr ;l > > > > The other thing I want to look at (but don't have concrete ideas of yet) > > is some of the lesser used dag/task concurrency and dependency features - > > if we can remove a few things and get a faster scheduler than it's worth > > having a discussion over. > > > > On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com > > <mailto:ja...@potiuk.com>> wrote: > > >I hope more voices will be coming soon as well :). > > > > > >Constance - few words of explanation, for my "remove" things. I do not > say > > >we "have to" remove those things or even that we "should". > > > > > >My main motivation for mentioning the list is that we often forget that > > >when we have a new major version of software - removing stuff is as > > >important as adding. So I think when it comes to final decisions about > > >Airflow 3 we should be well aware about both: what we add and what we > > >remove. So my main point is - "removal" of things should be as > important a > > >part in our future discussions as "adding", and I'd say with 10 years of > > >Airflow history, it's more important to remove things than to add them. > > > > > >Side comment: Ash for one mentioned at multiple occasions how proud he > is > > >that his contributions to Airflow are net-negative - he removed more > code > > >than added. Unfortunately it's not shown any more here > > >https://github.com/apache/airflow/graphs/contributors > > ><https://github.com/apache/airflow/graphs/contributors> - because we have > > >more than 10.000 commits, but I was hoping one day when we remove > > providers > > >from main development, I can do it personally and beat Ash to it. > > > > > >Just wanted to stress a few things on why I think "discussing potential > > >removals" is important: > > > > > >* I think it is super important to deliver Airflow 3 really fast. It > took > > > > > >1.5 year to release Airflow 2 and it was far too long and keeping > Airflow > > 1 > > >and 2 in parallel was a huge pain and drain and it slowed us down > > >enormously. We should absolutely limit the time when we actively develop > > >both Airflow 2 features and Airflow 3 features. > > >* I think most of the current deprecations of the 100+ we have does not > > >slow us down AT ALL. Removal of those is mostly cosmetic change, a > little > > >bit clutter to remove but they have little-to-no impact on actual speed > of > > >development, it's just an old code that keeps on being around > > >* on the other hand - some of the things I mentioned ARE slowing us > down A > > >LOT and will continue to do so until we remove them. For example, I > > believe > > >"postgresql + mysql + sqlite complete versioning solution for all the > > >possible variants of storage" will be quite a bit more complex and > testing > > >will take far more time than if we drop mysql and choose a single > storage > > >approach for Airflow 3.0. I have no hard data to back it up, but my gut > > >feeling is that it can take at least twice as long to get it out in the > > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for > > >all the options we have there. > > >* the telemetry will be cool - but even if we add it for 2.10, the first > > >time meaningful data will be available is maybe 2 years from now when we > > do > > >Airflow 4. For now we can completely forget about it because it's only > > >going to show us some early adopters of Airflow 2.10. But we have > surveys > > + > > >we can collect data from Astronomer, Google. Amazon for some usage and > > >identify who will be early adopters of Airflow 3 and target them first. > > >* the fact that we will drop something for Airflow 3.0, does not mean > that > > >we have to drop it "forever". We can safely assume that Airflow 3.0 will > > be > > >only used by early adopters, and many of our users will wait for 3.1, > 3.2 > > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those > "slower > > >moving users" from the 3.0. We can re-add things in 3.1 building on > > >foundations of 3.0 after it is in the hands of those early adopters. I > > >personally think we should prioritize speed of delivery of 3.0 over > > >"complete support of every deployment option of Airflow 2" - to allow > > >**some** of our early adopter users to migrate quickly, and add what's > > >missing for those who would like to move later in later versions. > > >* I think - this is the most important "product management" decision > here > > - > > >based on the data we have available. Which users do we target for > Airflow > > >3.0, and what we can drop to deliver it faster (and possibly re-add more > > >stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0 > > >initiative and the most important decision from the Product management > > side > > >that will impact the timeline of Airflow 3.0. While a lot depends on how > > >much time and effort individuals will spend on implementing things, the > > >decision of what we drop will impact everyone's speed and overall > delivery > > >of Airflow 3.0. It's a decision we should not take lightly. > > > > > >So my main point is: let's add "what we remove" as a very important > point > > >in the product discussions we are going to have. > > > > > >J. > > > > > > > > > > > > > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar > > ><rbish...@amazon.com.inva <mailto:rbish...@amazon.com.inva>lid> wrote: > > > > > >> Like Constance and many others, there was time needed to process this > > >> fantastical summary and approach to Airflow 3. __ > > >> A lot of the points raised by Jarek make sense, ie being prescriptive > on > > >> what features we want to be there on Day 1 vs launched on 3.x - this > > does > > >> allow us to move quickly for our users. > > >> In line with Constance's concern, we need to be mindful of what we > > decide > > >> to keep and what we are willing to cut - which could be a painstaking > > >> process that could offset any gains of trying to be faster. I also > > believe > > >> that bringing all these new amazing features on Airflow 3 will peak > the > > >> interest of early adopters and eventually get others interested in > > >> migration. However, I believe this migration will be a slow process > and > > >> will present a gap in certain functionalities that users may want > before > > >> entertaining any move to Airflow 3. There are still a lot of folks > using > > >> v1.10 today. There were several tactical initiatives in the past few > > months > > >> with intent on bringing new functionality, ie Multi-team, to Airflow > 2.x > > >> and I feel that while these efforts are not wasted, there should still > > be > > >> an option to continue improving Airflow 2 to avoid alienating our > users > > on > > >> the basis of a future promise in Airflow 3, that may not be easy to > > migrate > > >> towards. > > >> > > >> -- Rajesh > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> < > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> > > >> > > >> > > >> On 2024-05-07, 12:26 PM, "Constance Martineau" > > >> <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> > > >> <mailto:consta...@astronomer.io.inva > > >> <mailto:consta...@astronomer.io.inva>> > > >> <mailto:consta...@astronomer.io.inva > > >> <mailto:consta...@astronomer.io.inva> <mailto: > > consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva>>>LID> > > >> wrote: > > >> > > >> > > >> > > >> > > >> CAUTION: This email originated from outside of the organization. Do > not > > >> click links or open attachments unless you can confirm the sender and > > know > > >> the content is safe. > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > externe. > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > >> le contenu ne présente aucun risque. > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> Thank you Jarek for the detailed input. I've taken some time to digest > > your > > >> points before responding. > > >> > > >> > > >> > > >> > > >> You've outlined a bold vision for Airflow 3, and I agree that being > > >> decisive about the features and architectures will be the key to > > success. > > >> However, before we make final decisions on what features to cut or > > retain, > > >> it would be beneficial to have a more comprehensive understanding of > how > > >> the current features are utilized by the open-source community. > > >> > > >> > > >> > > >> > > >> @Kaxil Naik <ka...@astronomer.io <mailto:ka...@astronomer.io> > > >> <mailto:ka...@astronomer.io <mailto:ka...@astronomer.io>> > <mailto: > > >> ka...@astronomer.io <mailto:ka...@astronomer.io> > > >> <mailto:ka...@astronomer.io <mailto:ka...@astronomer.io>>>> recently > initiated a > > >> discussion on > > >> collecting telemetry from open-source deployments: > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m > > >> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> > > >> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>> < > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> > > >> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>> < > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> > > >> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt> > ;>. > > >> This data > > >> could be critical in ensuring our decisions are well-informed and > > reflect > > >> the real-world usage patterns of our users, not just those from > managed > > >> environments like Astro, MWAA or GCC. > > >> > > >> > > >> > > >> > > >> It's essential that we challenge our assumptions and base our > decisions > > on > > >> a holistic view of feature usage. Identifying potential cuts is a > > critical > > >> step, but let's ensure our strategy aligns with the needs and > > preferences > > >> of the broader Airflow community. > > >> > > >> > > >> > > >> > > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <ja...@potiuk.com > > >> <mailto:ja...@potiuk.com> > <mailto: > > >> ja...@potiuk.com <mailto:ja...@potiuk.com>> <mailto:ja...@potiuk.com > > >> <mailto:ja...@potiuk.com> <mailto:ja...@potiuk.com > > >> <mailto:ja...@potiuk.com> > >>> > > >> wrote: > > >> > > >> > > >> > > >> > > >> > I am currently on sick leave, and still recovering - hoping to be > > able to > > >> > travel next week to the US as planned, so I just wanted to break out > > of > > >> it > > >> > to make one comment here. > > >> > > > >> > I got a clearer head now a bit with medications hopefully working. I > > am > > >> > still taking it that should help me to get over the current state, > > and I > > >> > wanted to take a look at this discussion unraveling first. Over last > > week > > >> > I disconnected from "day-to-day" Airflow and put some thoughts (as > > much > > >> as > > >> > I could in my current state) on it. The whole subject of this thread > > was > > >> > started from that - how the current discussions on AIP-67 and others > > >> change > > >> > if we consider Airflow 3 is "starting". > > >> > > > >> > The price for back-compat is speed of development and quality. More > > >> > combinations to test, more unexpected issues uncovered, necessity to > > keep > > >> > parallel paths (old/new) while adding new features. All what > Constance > > >> > wrote about and what Ash explained. We already started to trip over > > our > > >> own > > >> > feet mutliple times in a few last releases. Have we tested all > > >> combinations > > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we > already > > see > > >> > that in a number of "combos" of features things are not working in > as > > >> > stable a way as they did before. > > >> > > > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a > > long > > >> > time (or even move out) as they will not want to move to Airflow 3. > A > > lot > > >> > of the work implemented in AIP-44 and design of AIP-67 was done > around > > >> > back-compatibility. but yes - > > >> > it would have been way easier if designed anew without > > back-compatibility > > >> > in mind. And if we implement it and release it in Airflow 2 it will > > make > > >> > new Airflow feature development even harder. That's why I wanted to > > treat > > >> > it as "tactical" solution - hoping that in Airflow 3 we can make it > > >> > "properly" - and that's why I started the discussion here when I > > sensed > > >> > that we are "close" to Airflow 3 discussion, because I wanted to see > > what > > >> > options we have there. This is why I have not yet concluded voting > on > > >> > AIP-67 waiting for the result of this discussion here. > > >> > > > >> > But if we are ready to go for Airflow.3 then I'd say there are two > > >> > important things that should be part of the vision. > > >> > > > >> > 1) *We should be far more opinionated and have far fewer options of > > >> > running things in Airflow 3*. Even an order of magnitude more > > >> opinionated. > > >> > Make choices, stick to it, perfect those opinionated choices to suit > > >> 80/20 > > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not > > fitting > > >> > the 20% that might choose to stay at Airflow 2. We can choose now > > which > > >> > ~20% of cases we do not want to handle deliberately. And we should > be > > >> very, > > >> > very strict about it. Default should be "no choice". This will > > radically > > >> > simplify deployment and should make it easier to simplify Airflow > > >> > development and DAG authoring experience because we will have less > > cases > > >> to > > >> > support. Even if we plan to add more options in the future, the > first > > >> > version of Airflow 3 should support one deployment approach only. > > This is > > >> > the only way we can deliver it fast. And we should be very bold > there. > > >> > Choose one option and go for it in pretty much every place we have > > >> choices > > >> > now. We should Aim for Airflow 3.0 to support only a subset of > current > > >> > users - but those who are most likely to migrate first and those > with > > the > > >> > biggest need for the new features. We can think 3.x to support more > > >> cases, > > >> > but 3.0 should be as opinionated as humanly possible. > > >> > > > >> > And this deployment option should be also something ALL our > > stakeholders > > >> > will feel OK with as a way forward in their offering. > > >> > > > >> > My candidates (and yes, some are bold): > > >> > > > >> > * *Drop MySQL*. If we have a single thing that makes us avoid our > > schema > > >> > and DB migration - this is the case. Let's choose Postgres 15+ and > use > > >> some > > >> > of the great features there. This will also enable much faster async > > SQL > > >> > implementation and a number of other optimisations - not to mention > > >> cutting > > >> > every single change in development and testing time by literally > half. > > >> And > > >> > we should not look back to adding MySQL. > > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S only > > (and > > >> > AWS/Google others can continue developing theirs of course in > parallel > > >> and > > >> > continue Hybrid executor work). Later - we figure out a better > > solution > > >> to > > >> > support "small" tasks using some new K8S features and possibly > non-k8s > > >> > solutions (Ray-based?) > > >> > * *Cut Connection and Variable Management from DB/UI*. Leave only > > Secrets > > >> > Management. Later when we have a 100% extensible React UI, we can > add > > a > > >> > "local DB secrets manager" add-on > > >> > * *Choose a single way for DAG storage that will support versioning > > from > > >> > day one*. Bear in mind we can add others later. Bolke's idea of > using > > >> > FSspec is an interesting one, we should see if it is feasible. > > >> > * *Drop FAB completely (including custom plugins) and invest in > > >> > implementing Auth Manager based on a dedicated, external solution* > > >> > (KeyCloak > > >> > that we've discussed before as a likely candidate) > > >> > * *Leave Providers with Airflow 2 and add tests to make sure they > are > > >> > Airflow 3 future-compatible *- develop a way where we continue > > >> development > > >> > and contributions for Providers with Airflow 2 and add complete > tests > > to > > >> > run them with Airflow 3. This way we can continue developing > Provider > > >> > features independently, and make them work for Airflow 2 (and > continue > > >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes), > > while > > >> > also gradually fix any Airflow3 incompatibilities and instead of > > >> > "back-compatibility" tests make provider "forward-compatibility" > > tests so > > >> > that future Providers are tested and work on Airflow 3. Also it will > > make > > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested > without > > >> > investing in changing the current CI / test harness. > > >> > * *Simplify Test Harness for Airflow 3 from the start *- without > > >> providers > > >> > and 790+ dependencies, we could vastly simplify Airflow3 testing > > >> (basically > > >> > make CI jobs from scratch) using mostly standard Python tooling > > (while we > > >> > can continue making use of the current test harness for Airflow 2 + > > >> > Providers and extend it with Airflow 3 future-compatibility tests). > > That > > >> > means Breeze would be only staying in Airflow 2 + Providers repo as > we > > >> > should be able to achieve most of what we have there with local > venv/ > > >> > tooling (especially with uv as underlying tooling). > > >> > > > >> > 2) *I think we only add very few new "important" features. *Absolute > > >> > minimum to make Airflow 3 appealing and add them only in Airflow 3: > > >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it > > makes > > >> no > > >> > sense to invest into Airflow 2 if we already know Airflow 3 is > coming > > - > > >> > that generally triples effort needed to get them out. We should drop > > new > > >> > features development in Airflow 2. This will give users incentive to > > move > > >> > to 3 if the new features will be worth it. Even paying > > >> > compatibility/migration price. > > >> > > > >> > Versionig, for example: I believe if we decide to go only with > > Airflow 3 > > >> > and cut some of the above (Postgres only, Single versioning DAG > > storage) > > >> we > > >> > can make bolder decisions in versioning and support simpler models > > from > > >> the > > >> > get go (and deliver it faster). And we should add only a few - but > > >> > important - features that our users clearly asked for and focus on > > >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or > > >> 2.11). > > >> > Similarly - multi-team can be simplified if we cut things from the > > list > > >> > above and have Task isolation as first-class citizens in Airflow > (and > > the > > >> > only option). > > >> > > > >> > My candidates very much concur with the list shared by Kaxil in the > > doc + > > >> > I'd add multi-team (but simplified thanks to the cuts). But I also > > here > > >> > would mostly revert to Astronomer, Google. AWS team to define > > >> collectively > > >> > what is the absolute minimum set of features that would get the > > "target" > > >> > part of their customers happy. And ONLY do that. > > >> > > > >> > So in short - I think the big part of our discussion should be what > we > > >> are > > >> > ready to drop when we start airflow 3 and be very bold. Once we know > > we > > >> > should figure out the absolute minimum of things that we can add > that > > >> will > > >> > benefit a significant part of our users (and make use of increased > > speed > > >> > because we dropped things). > > >> > > > >> > J. > > >> > > > >> > > > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau > > >> > <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> > > >> > <mailto:consta...@astronomer.io.inva > > >> > <mailto:consta...@astronomer.io.inva>> > > >> <mailto:consta...@astronomer.io.inva > > >> <mailto:consta...@astronomer.io.inva> <mailto: > > consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva>>>lid> > > >> wrote: > > >> > > > >> > > Hi Michal, > > >> > > > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate > > your > > >> > > concerns about the migration overhead for our users with a major > new > > >> > > version and see the appeal in your suggestion to integrate many of > > the > > >> > > proposed changes into Airflow 2 through separate AIPs. It’s a > valid > > >> point > > >> > > and certainly aligns with the value of making incremental > > improvements. > > >> > > > > >> > > However, after looking closely at the enhancements outlined for > > Airflow > > >> > 3, > > >> > > I'm convinced they warrant a new major release. Here’s why: > > >> > > > > >> > > 1. *Core Architectural Changes:* We’re looking at foundational > > changes > > >> > > with Airflow 3—like redefining task priorities, separating task > > >> > > definition > > >> > > and task execution, and new AIPs like DAG versioning. remote > > execution > > >> > > and restricting database access from workers. These aren’t just > > >> > > incremental > > >> > > improvements but major shifts that will set the stage for the next > > >> > > decade > > >> > > of Airflow’s architecture. Grouping these changes into a major > > release > > >> > > will > > >> > > help us make these transitions more cleanly and with fewer > > constraints > > >> > > from > > >> > > past decisions. > > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140 > > >> > deprecated > > >> > > issues, and this will only grow if we continue without a major > > >> > cleanup. > > >> > > This makes it increasingly difficult to implement new features > > >> > > effectively > > >> > > while maintaining backward compatibility. A major release allows > us > > to > > >> > > address these issues head-on, reducing technical debt and paving > the > > >> > way > > >> > > for a more robust platform. > > >> > > 3. *Managing Breaking Changes:* Let’s take the example of > > restricting > > >> > > database access from workers. It’s a necessary move for better > > >> > security > > >> > > and > > >> > > also potentially scalability reasons (reduces DB load). Many users > > >> > have > > >> > > workflows that interact with the DB, either by using raw sql or by > > >> > > leveraging a session object. We could implement this feature in > > >> > Airflow > > >> > > 2 > > >> > > and avoid breaking existing workflows by continuing to have the > old > > >> > > standard mode as default - much of the work is already done - but > > that > > >> > > would mean supporting both the new secure mode and the old > standard > > >> > mode > > >> > > indefinitely and design new features with the assumption that most > > >> > will > > >> > > continue using the old standard mode. With Airflow 3, we can make > > >> > secure > > >> > > mode the default or even the only option, simplifying > implementation > > >> > and > > >> > > future development. This is just one example where it is feasible > to > > >> > > implement in Airflow 2, but is better if we release it under the > > >> > > context of > > >> > > Airflow 3. > > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up > > >> > > possibilities for handling workflows beyond batch processing. > > Features > > >> > > like > > >> > > real-time DAG execution through API and multi-language task > support > > >> > are > > >> > > big > > >> > > steps forward, significantly expanding Airflow’s utility. > > >> > > > > >> > > > > >> > > While integrating these updates into Airflow 2 might look less > > >> disruptive > > >> > > initially, the scale and nature of the required changes really > > support > > >> a > > >> > > move to Airflow 3. It’s not just about adding new features; it’s > > about > > >> > > setting up Airflow so that it continues to remain relevant for the > > next > > >> > ten > > >> > > years. > > >> > > > > >> > > Constance > > >> > > > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <a...@apache.org > > >> > > <mailto:a...@apache.org> > > >> <mailto:a...@apache.org <mailto:a...@apache.org>> > > >> <mailto:a...@apache.org <mailto:a...@apache.org> <mailto:a...@apache.org > > >> <mailto:a...@apache.org> > > >>> > > >> wrote: > > >> > > > > >> > > > There's a lot of technical debt hiding in Airflow, especially > the > > >> > > > scheduler that makes it harder and harder to efficiently add new > > >> > > features. > > >> > > > > > >> > > > At some point, very soon, we are going to have to remove some > very > > >> > > > infrequently used back compat shims that negatively affect > > >> performance. > > >> > > > Without doing that the pace at which we can realistically add > > some of > > >> > the > > >> > > > more exciting features tends towards zero. Developer speed of > > >> > > contributors > > >> > > > is a factor here too! > > >> > > > > > >> > > > So while we are still using SemVer, that necessitates v3. > > >> > > > > > >> > > > Ash > > >> > > > > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" < > > michalmod...@google.com <mailto:michalmod...@google.com> > > >> <mailto:michalmod...@google.com <mailto:michalmod...@google.com>> > > >> <mailto:michalmod...@google.com <mailto:michalmod...@google.com> > > <mailto: > > >> michalmod...@google.com <mailto:michalmod...@google.com>>> > > >> > > .INVALID> > > >> > > > wrote: > > >> > > > >+1 to Jens's & Bolke's points here and in the doc > > >> > > > > > > >> > > > >I agree we should work on clarifying the directions we would > like > > >> > > Airflow > > >> > > > >to go. Introducing a new major Airflow version is a massive > > overhead > > >> > for > > >> > > > >users, who would need to plan for migrations, onboarding the > new > > >> > Airflow > > >> > > > >(with a slightly different architecture), etc., and effectively > > >> > Airflow > > >> > > 2 > > >> > > > >would live in parallel for a long time. > > >> > > > > > > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc > > are > > >> > > > valuable > > >> > > > >projects of their own, and I could imagine all of them being > > >> delivered > > >> > > as > > >> > > > >separate AIPs within Airflow 2 (surely new minor versions of > > Airflow > > >> > > 2). I > > >> > > > >am not sure if the scope of changes and the goal we want to > > achieve > > >> is > > >> > > a) > > >> > > > >clear enough b) broad enough to call for a new major version. > > >> > > > > > > >> > > > >Best, > > >> > > > >Michal > > >> > > > > > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens > (XC-AS/EAE-ADA-T) > > >> > > > ><jens.scheff...@de.bosch.com.inva > > >> > > > ><mailto:jens.scheff...@de.bosch.com.inva> <mailto: > > >> jens.scheff...@de.bosch.com.inva > > >> <mailto:jens.scheff...@de.bosch.com.inva>> <mailto: > > jens.scheff...@de.bosch.com.inva <mailto:jens.scheff...@de.bosch.com.inva> > > >> <mailto:jens.scheff...@de.bosch.com.inva > > >> <mailto:jens.scheff...@de.bosch.com.inva>>>lid> wrote: > > >> > > > > > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is > > mostly a > > >> > > > vision > > >> > > > >> statement. > > >> > > > >> > > >> > > > >> Looking forward for a larger addendum where we can collect > > things > > >> > that > > >> > > > we > > >> > > > >> all can vote and agree on as targets. > > >> > > > >> > > >> > > > >> As I started earlier with a confluence page and it seems this > > is > > >> not > > >> > > > >> accessible to all, shall we convert this to a Google Doc for > > >> better > > >> > > > >> collaboration and item collection? > > >> > > > >> > > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> > > >> > > > >> <https://aka.ms/o0ukef>> < > > >> https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>> > > >> <https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>> < > > >> https://aka.ms/o0ukef&gt;>> > > >> <https://aka.ms/o0ukef&amp;gt;&gt;>> > > >> > > > >> ________________________________ > > >> > > > >> From: Vikram Koka <vik...@astronomer.io.inva > > >> > > > >> <mailto:vik...@astronomer.io.inva> <mailto: > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>> > > >> <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > > >> <mailto: > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>>>LID> > > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM > > >> > > > >> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > >> > > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> < > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> <mailto: > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>>>> > > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow > > 2) > > >> vs > > >> > > > >> strategic (Airflow 3) approach > > >> > > > >> > > >> > > > >> Thank you for your feedback, Bolke and Andrey! > > >> > > > >> > > >> > > > >> Bolke, > > >> > > > >> I have replied to some of your comments in the doc. > > >> > > > >> I will provide a detailed write up on the "Interactive DAG > run" > > >> (or > > >> > > > >> synchronous DAG run) capability, which has generated some > early > > >> > > > questions. > > >> > > > >> I had intended to get an AIP published for that as a > follow-up, > > >> but > > >> > I > > >> > > > >> believe that a simpler write up would be useful ahead of the > > AIP. > > >> > > > >> > > >> > > > >> Andrey, > > >> > > > >> You raise an interesting point. > > >> > > > >> > > >> > > > >> As part of the Airflow 2.0 release, we as a community had > > decided > > >> to > > >> > > > >> strictly adhere to Semver as detailed in the document you > > >> > referenced. > > >> > > We > > >> > > > >> also consciously split out the "Core Airflow" releases from > the > > >> > > > "Provider" > > >> > > > >> releases at that time. We had a clear expectation then for > the > > >> > cadence > > >> > > > of > > >> > > > >> both minor and patch releases, which we have generally > adhered > > to > > >> > > since > > >> > > > >> then. > > >> > > > >> > > >> > > > >> Personally, I am more concerned about our Provider releases > > right > > >> > now, > > >> > > > as > > >> > > > >> compared to the cadence of our major releases. I believe that > > one > > >> of > > >> > > the > > >> > > > >> proposed changes in the Airflow 3 document i.e. the clear > > >> separation > > >> > > for > > >> > > > >> Task Execution will help here, but more may be needed. > > >> > > > >> > > >> > > > >> Definitely interested in more feedback on this as well. > > >> > > > >> > > >> > > > >> Vikram > > >> > > > >> > > >> > > > >> > > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin < > > >> > > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> > > >> > > <mailto:andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>> > <mailto: > > >> andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> > > >> <mailto:andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>>> > > >> > > > > > > >> > > > >> wrote: > > >> > > > >> > > >> > > > >> > I would like to propose to change (at least discuss) > release > > >> > policy > > >> > > > >> around > > >> > > > >> > the Major version of Airflow. > > >> > > > >> > > > >> > > > >> > Right now it is described as "These releases do not happen > > with > > >> > any > > >> > > > >> regular > > >> > > > >> > interval or on any predictable schedule." : > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > > > > >> > > > > >> > > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0> > > >> < > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> > > > > > >> < > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> > > > > > >> < > > >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> > > <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt> > > >> ;> > > >> > > > >> < > > >> > > > >> > > >> > > > > > >> > > > > >> > > > >> > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > > >> < > > >> > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > > > > > >> < > > >> > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > > > > > >> < > > >> > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > > <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt> > > >> ;> > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > So maybe it is time to make it schedulable, e.g. one per > two > > >> years > > >> > > or > > >> > > > so. > > >> > > > >> > This one could help us to avoid such a discussion in the > > future, > > >> > > like > > >> > > > "We > > >> > > > >> > don't know when Airflow 4 is coming.". At the moment when > the > > >> new > > >> > > > major > > >> > > > >> > version will be released new features wouldn't be added in > > the > > >> old > > >> > > > major > > >> > > > >> > version, however we would support bug / security for a > while, > > >> > e.g. 1 > > >> > > > year > > >> > > > >> > for bug fixes, 3 years for security fixes with a total 5 > year > > >> > > > lifecycle > > >> > > > >> per > > >> > > > >> > a major version. These just are approximate time periods > for > > a > > >> > > > definition > > >> > > > >> > of current period, bugfix period and security fix period. > > >> > > > >> > > > >> > > > >> > In contributors' perspective it helps with dropping the > > >> deprecated > > >> > > > stuff > > >> > > > >> > which resolves some old problem: we have to support > > everything > > >> > > > including > > >> > > > >> > deprecated stuff and without schedulable lifecycle for the > > >> > > deprecated > > >> > > > >> stuff > > >> > > > >> > it could be showstopper for the new feature, because > > sometimes > > >> it > > >> > > > hard to > > >> > > > >> > support two different approaches for long period of time > > with no > > >> > > hope > > >> > > > >> that > > >> > > > >> > it will happen soon. For some fundamental stuff which do > not > > >> > > require a > > >> > > > >> lot > > >> > > > >> > things time to support we could postponed removal for next > > after > > >> > the > > >> > > > next > > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in > > Airflow 5 > > >> > > > >> > > > >> > > > >> > In the user perspective, they have at least bug fix support > > for > > >> a > > >> > > > while, > > >> > > > >> if > > >> > > > >> > someone want to use legacy version it their choice, however > > no > > >> new > > >> > > > >> > features, no new version of providers (after one year) > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > ---- > > >> > > > >> > Best Wishes > > >> > > > >> > *Andrey Anshin* > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin < > > bdbr...@gmail.com <mailto:bdbr...@gmail.com> > > >> <mailto:bdbr...@gmail.com <mailto:bdbr...@gmail.com>> > > >> <mailto:bdbr...@gmail.com <mailto:bdbr...@gmail.com> <mailto: > > >> bdbr...@gmail.com <mailto:bdbr...@gmail.com>>>> > > >> > > > wrote: > > >> > > > >> > > > >> > > > >> > > I have left several comments :-). And on interactive dag > > runs > > >> > even > > >> > > > >> after > > >> > > > >> > > the explanation of Vikram I still don't have a clue what > we > > >> want > > >> > > to > > >> > > > >> > > accomplish there :-P. > > >> > > > >> > > > > >> > > > >> > > I would like to see a mantra or team for Airflow 3. That > > helps > > >> > > > nudging > > >> > > > >> > > people in the same direction. Suggestions in the > comments. > > >> > > > >> > > > > >> > > > >> > > Bolke > > >> > > > >> > > Sent from my iPhone > > >> > > > >> > > > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka > > >> > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > > >> > > <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>> > > <mailto: > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > > >> <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>>>lid > > >> > > > > > > >> > > > >> > > wrote: > > >> > > > >> > > > > > >> > > > >> > > > Good point Jed. > > >> > > > >> > > > I responded back to your comment in the doc as well and > > very > > >> > > open > > >> > > > to > > >> > > > >> > > > changing the term in the doc. > > >> > > > >> > > > > > >> > > > >> > > > Used the term "interactive DAG run" as the ability to > > invoke > > >> > or > > >> > > > >> > trigger a > > >> > > > >> > > > DAG run through the API, with the expectation of > getting > > >> back > > >> > a > > >> > > > >> result > > >> > > > >> > > > immediately. An alternate term could be a "synchronous > > DAG > > >> > run". > > >> > > > >> > > > > > >> > > > >> > > > Regardless, this is a significant change so a good term > > to > > >> > > > indicate > > >> > > > >> the > > >> > > > >> > > > expansion from "batch runs only" is warranted. Very > open > > to > > >> > > > different > > >> > > > >> > > terms > > >> > > > >> > > > here. > > >> > > > >> > > > > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham < > > >> > > > >> > jedcunning...@apache.org <mailto:jedcunning...@apache.org> > > >> > > > >> > <mailto:jedcunning...@apache.org > > >> > > > >> > <mailto:jedcunning...@apache.org>> > > >> <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org> > > >> <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org>>> > > >> > > > >> > > > > > >> > > > >> > > >> wrote: > > >> > > > >> > > >> > > >> > > > >> > > >> Very exciting! Looks like we will have a busy period > of > > >> time > > >> > > > ahead > > >> > > > >> of > > >> > > > >> > > us. > > >> > > > >> > > >> Overall I like the plan so far, especially using this > > >> year's > > >> > > > Airflow > > >> > > > >> > > Summit > > >> > > > >> > > >> as an opportunity to announce and gather feedback, and > > the > > >> > 2025 > > >> > > > >> > version > > >> > > > >> > > to > > >> > > > >> > > >> pitch upgrading. > > >> > > > >> > > >> > > >> > > > >> > > >> I left a comment in the doc, but we might want to > > iterate > > >> on > > >> > > the > > >> > > > >> > > >> terminology we use for high priority or "synchronous" > > DAG > > >> > runs > > >> > > to > > >> > > > >> > serve > > >> > > > >> > > LLM > > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit > > confusing. > > >> > > > >> > > >> > > >> > > > >> > > > > >> > > > >> > > > > >> > > > > > --------------------------------------------------------------------- > > >> > > > >> > > To unsubscribe, e-mail: > dev-unsubscr...@airflow.apache.org <mailto:dev-unsubscr...@airflow.apache.org> > > >> <mailto:dev-unsubscr...@airflow.apache.org > > >> <mailto:dev-unsubscr...@airflow.apache.org>> <mailto: > > >> dev-unsubscr...@airflow.apache.org > > >> <mailto:dev-unsubscr...@airflow.apache.org> <mailto: > > >> dev-unsubscr...@airflow.apache.org > > >> <mailto:dev-unsubscr...@airflow.apache.org>>> > > >> > > > >> > > For additional commands, e-mail: > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > >> <mailto:dev-h...@airflow.apache.org > > >> <mailto:dev-h...@airflow.apache.org>> <mailto: > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > >> <mailto:dev-h...@airflow.apache.org > > >> <mailto:dev-h...@airflow.apache.org>>> > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >
