I think relying on connections wrapped in the task execution context makes sense as a way to retrieve the connections to be used by OpenLineage. Similarly, lineage backend relied on the task instance context in the past ( https://github.com/apache/airflow/blob/dc61da2509536bb509a4a2e1781ff47aa36b62b4/airflow/lineage/backend.py#L44; not sure if connections were included in it though). It also has the benefit of the appropriate connections (and authorized for the task) being present in the task context, and not querying db for connections, having access to all that are defined. One thing we'd need to verify is how that context is serialised/persisted (if it is at all), so that we don't proliferate any connection data.
On Mon, May 13, 2024 at 10:53 AM Maciej Obuchowski < obuchowski.mac...@gmail.com> wrote: > I would like to discuss the role of the Listener API and the OpenLineage > provider that utilizes it, > as there are current discussions that impact this integration. > > For example, there is a consensus on (and I strongly support) removing > direct database access from > workers. However, currently, OpenLineage integration depends on it. I want > to explain what we do > and how we can proceed with removing database access from workers while not > only retaining but > possibly enhancing lineage functionality. > > Currently, OpenLineage uses not only the data that the task directly has > but also utilizes > external connections in three ways: > > - We send events to the configured OpenLineage backend. This does not > depend on any external > connection or Airflow database but merely on configuration, so it's > acceptable. > - We use the Airflow database, primarily to get connections - for example, > when gathering lineage > from SQLExecuteQueryOperator, to obtain information about the default > database or schema. > - We use external APIs - sometimes directly (like asking BigQuery about job > table dependencies) or > using Airflow Connections received from the Airflow database, like in > SQLExecuteQueryOperator to > get table schemas. > > The issue arises if we are unable to perform the second or third methods - > not being able to > understand on which database a SELECT * from table operates would render > the potential lineage > data gathered from the statement itself meaningless. > > As tasks require connection access, I assume connection data will somehow > be passed as part of the > metadata to task execution - whether it's part of the executor protocol or > in some other way (I'm > not an expert on that part of Airflow). Then, provided it's accessible as > part of some execution > context, and not only passed to the task's execute method, OpenLineage > could utilize it. > > If the task execution metadata were standardized, we could take a step > further and perhaps move > from a model where TaskInstance listeners are executed on the worker to one > where they simply > depend on that metadata rather than the TaskInstance object. They could be > executed elsewhere - on > some specialized component working asynchronously, like the triggerer - > which would not be a fitting > name anymore. > > pon., 13 maj 2024 o 08:55 Michał Modras <michalmod...@google.com.invalid> > napisał(a): > > > Thanks to everyone for the constructive discussion - let's make it even > > more specific in the weekly syncs! I put a slot for us next Tuesday. I > had > > some challenges with how some of the e-mails are displayed in this > thread - > > please reach out to me if you'd like to be added. > > > > In the meantime, I summarised challenges, opportunities and potential > > work streams of Airflow 3 from the Composer team perspective. Please > have a > > look and chime in here: > > > > > https://docs.google.com/document/d/1kfqq3YFtUjFuuiv7KVZRqYgwFu9jcYyW_Mz7UUZYnks/edit?usp=sharing > > > > The main themes are on-demand & event driven DAG parsing, performance, > > isolation and stability improvements, and a big concern of backwards > > compatibility (+1 to Shubham's point - let's try to make changes in DAGS > > unnecessary, as opposed to potential changes in Airflow configuration). > > > > Best, > > Michal > > > > On Sat, May 11, 2024 at 9:30 AM Mehta, Shubham <shu...@amazon.com.invalid > > > > wrote: > > > > > Sorry for joining the discussion late - I was down under the weather > for > > > the past two weeks. > > > First off, great discussion so far and appreciate the effort and > thought > > > everyone has put into exploring Airflow 3. > > > > > > The list from Kaxil, et al looks solid and I'm really looking forward > to > > > more extensive discussions on those and seeing them implemented in > > Airflow. > > > They will truly unlock some amazing use cases with Airflow. > > > > > > Regarding building for Airflow 2 vs Airflow 3, personally, I think we > > > should keep pushing critical features for Airflow 2 until the scope for > > > Airflow 3 is properly defined. For example, we shouldn't pause work on > > the > > > multi-team capabilities as there are quite a lot of enterprises rely on > > > Airflow today, and we have been saying that this feature is coming for > > > years now. Introducing this in Airflow 3 means users have to jump > through > > > another non-backward compatible release to take advantage of it, which > > > doesn't feel user focused. Airflow 3 is still at least a year away from > > > anything substantial shipping. Not every change has to be breaking - we > > can > > > develop some things for Airflow 2 and potentially adopt them as-is in > > > Airflow 3 if it makes sense. Only if we're certain the dev effort will > be > > > way less by starting fresh on Airflow 3, should we consider that path. > > > > > > On Jarek's point about being very opinionated in Airflow 3.0 and > > > potentially dropping certain features/options to start with a lean core > > - I > > > am very divided on this. In most cases, I would say that being > > opinionated > > > is great, but in this case, we risk alienating our users, especially > > those > > > who self-host as they have proportionately less representation among > the > > > maintainers. Thousands of organizations rely on Airflow and breaking > the > > > trust could be costly in the long run. We should of course be basing > > those > > > decisions on real user data versus assumptions, but not everyone is > going > > > to jump to 2.10+ in a few months and data will be biased. I am not > > > concerned about MWAA customers here, but more about users who are > > > self-hosting. > > > > > > We've put together a Google doc (with writing access for all) capturing > > > some wish list items based on customer discussions: > > > > > > https://docs.google.com/document/d/17KKjwmwHquL_laMpsqZaZD5wzruKAAyFeOsLhGNp7vc/edit?usp=sharing > > > . It's still a work-in-progress as we gather more feedback from our > > > customers on what could help increase Airflow adoption in their orgs. > We > > > understand that not everything in this list will be implemented, but it > > is > > > essential to bring forward all ideas and discuss to make Airflow better > > for > > > everyone. > > > > > > I think it would be great to collaborate with others including Jarek, > > > Kaxil, Jens, Bolke, Amogh etc. who have already or would like to share > > > ideas, to brainstorm tenets, major themes, scope of Airflow 3, along > with > > > prioritized list. Maybe we can have a dedicated sync, say next Thursday > > > 8:30am PT, to discuss this further? I can volunteer to organize these > > > meetings. Initial set of meetings will be to brainstorm to bring > forward > > a > > > comprehensive and aligned proposal forward for the community. > > > > > > I'm excited about Airflow 3 and looking forward to shaping Airflow's > > > future together! > > > > > > Shubham > > > > > > > > > On 2024-05-10, 8:45 AM, "Pierre Jeambrun" <pierrejb...@gmail.com > > <mailto: > > > pierrejb...@gmail.com>> wrote: > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > pouvez > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > > que > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > Thank you for the work and proposal made around airflow 3. > > > > > > > > > I also strongly believe that those important changes need to happen for > > > airflow to stay relevant and it would be the right moment to start > > > discussing and planning for airflow 3. As mentioned some features are > > hard > > > to develop within airflow 2 with backward compatibility in mind: > > > * I would love to see a full react front end, without FAB. > > > * We could also take the opportunity to rework and clarify our RBAC > code, > > > we had quite a number of CVEs related to it. > > > * Do the breaking change for api clients and (finally) upgrade the > > openapi > > > client generator to a more recent version > > > * Extract business service layers that could be shared across the > entire > > > codebase instead of having a 'component wise' approach. Such that > > handling > > > of dag dates (execution, logical), serialization, resource access > control > > > and such tools could be implemented once and shared everywhere. (easier > > > maintenance, more consistency between API, CLI, and other core > components > > > behaviors). Inspired by some famous software architecture such as > > hexagonal > > > or clean. Even if I don't think that those would be relevant to airflow > > > codebase stricto sensu. > > > > > > > > > As mentioned stability for our users is key and supporting airflow 2.x > > for > > > a while after airflow 3 is a must have. I just want to bring to > > > your attention that this means more work for our release managers and > > also > > > for the security team (backporting security patches, CVEs handling for > > two > > > different major versions, etc.). All that to say that we should choose > > this > > > period wisely. (long enough, but not too long) > > > > > > > > > This sounds like a really exciting time for airflow. > > > Cheers > > > > > > > > > Le jeu. 9 mai 2024 à 11:23, Amogh Desai <amoghdesai....@gmail.com > > <mailto: > > > amoghdesai....@gmail.com>> a écrit : > > > > > > > > > > Thanks for initiating this discussion @Kaxil Naik < > ka...@astronomer.io > > > <mailto:ka...@astronomer.io>>! > > > > > > > > > > > > Read through the document and the emails here and I like the > direction > > in > > > > which > > > > we are proceeding. > > > > > > > > I also agree that removing code is almost as important as adding > code. > > We > > > > need to have a clear checklist of what we want to > > > > remove and get a vote for the same from the community. > > > > > > > > I also see that we have multiple ways of doing a thing, which is not > > > needed > > > > (serialising for example) and removing > > > > which can lead to simpler interfaces for users, and contributors too. > > > > > > > > I also find the *secret masker* overly complex personally, why can't > it > > > be > > > > simpler :) > > > > > > > > However, I have a small concern with the proportion of the users that > > > would > > > > like to migrate to an early version of > > > > Airflow 3 if we do not support various deployment modes. If we dont > get > > > > enough initial feedback, we do not know how the new > > > > major is doing, no? > > > > > > > > Thanks & Regards, > > > > Amogh Desai > > > > > > > > > > > > On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <a...@apache.org > > > <mailto:a...@apache.org>> wrote: > > > > > > > > > Strong agree - I think one of the things I regret most about > Airflow > > 2 > > > > was > > > > > that we didn't remove enough. It probably eased the upgrade process > > for > > > > > users though. > > > > > > > > > > > but I was hoping one day when we remove providers > > > > > > from main development, I can do it personally and beat Ash to it. > > > > > > > > > > Grrr ;l > > > > > > > > > > The other thing I want to look at (but don't have concrete ideas of > > > yet) > > > > > is some of the lesser used dag/task concurrency and dependency > > > features - > > > > > if we can remove a few things and get a faster scheduler than it's > > > worth > > > > > having a discussion over. > > > > > > > > > > On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com > <mailto: > > > ja...@potiuk.com>> wrote: > > > > > >I hope more voices will be coming soon as well :). > > > > > > > > > > > >Constance - few words of explanation, for my "remove" things. I do > > not > > > > say > > > > > >we "have to" remove those things or even that we "should". > > > > > > > > > > > >My main motivation for mentioning the list is that we often forget > > > that > > > > > >when we have a new major version of software - removing stuff is > as > > > > > >important as adding. So I think when it comes to final decisions > > about > > > > > >Airflow 3 we should be well aware about both: what we add and what > > we > > > > > >remove. So my main point is - "removal" of things should be as > > > > important a > > > > > >part in our future discussions as "adding", and I'd say with 10 > > years > > > of > > > > > >Airflow history, it's more important to remove things than to add > > > them. > > > > > > > > > > > >Side comment: Ash for one mentioned at multiple occasions how > proud > > he > > > > is > > > > > >that his contributions to Airflow are net-negative - he removed > more > > > > code > > > > > >than added. Unfortunately it's not shown any more here > > > > > >https://github.com/apache/airflow/graphs/contributors < > > > https://github.com/apache/airflow/graphs/contributors> - because we > have > > > > > >more than 10.000 commits, but I was hoping one day when we remove > > > > > providers > > > > > >from main development, I can do it personally and beat Ash to it. > > > > > > > > > > > >Just wanted to stress a few things on why I think "discussing > > > potential > > > > > >removals" is important: > > > > > > > > > > > >* I think it is super important to deliver Airflow 3 really fast. > It > > > > took > > > > > > > > > > > >1.5 year to release Airflow 2 and it was far too long and keeping > > > > Airflow > > > > > 1 > > > > > >and 2 in parallel was a huge pain and drain and it slowed us down > > > > > >enormously. We should absolutely limit the time when we actively > > > develop > > > > > >both Airflow 2 features and Airflow 3 features. > > > > > >* I think most of the current deprecations of the 100+ we have > does > > > not > > > > > >slow us down AT ALL. Removal of those is mostly cosmetic change, a > > > > little > > > > > >bit clutter to remove but they have little-to-no impact on actual > > > speed > > > > of > > > > > >development, it's just an old code that keeps on being around > > > > > >* on the other hand - some of the things I mentioned ARE slowing > us > > > > down A > > > > > >LOT and will continue to do so until we remove them. For example, > I > > > > > believe > > > > > >"postgresql + mysql + sqlite complete versioning solution for all > > the > > > > > >possible variants of storage" will be quite a bit more complex and > > > > testing > > > > > >will take far more time than if we drop mysql and choose a single > > > > storage > > > > > >approach for Airflow 3.0. I have no hard data to back it up, but > my > > > gut > > > > > >feeling is that it can take at least twice as long to get it out > in > > > the > > > > > >hands of our users if we try to have Airflow 2 parity in Airflow > 3.0 > > > for > > > > > >all the options we have there. > > > > > >* the telemetry will be cool - but even if we add it for 2.10, the > > > first > > > > > >time meaningful data will be available is maybe 2 years from now > > when > > > we > > > > > do > > > > > >Airflow 4. For now we can completely forget about it because it's > > only > > > > > >going to show us some early adopters of Airflow 2.10. But we have > > > > surveys > > > > > + > > > > > >we can collect data from Astronomer, Google. Amazon for some usage > > and > > > > > >identify who will be early adopters of Airflow 3 and target them > > > first. > > > > > >* the fact that we will drop something for Airflow 3.0, does not > > mean > > > > that > > > > > >we have to drop it "forever". We can safely assume that Airflow > 3.0 > > > will > > > > > be > > > > > >only used by early adopters, and many of our users will wait for > > 3.1, > > > > 3.2 > > > > > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those > > > > "slower > > > > > >moving users" from the 3.0. We can re-add things in 3.1 building > on > > > > > >foundations of 3.0 after it is in the hands of those early > > adopters. I > > > > > >personally think we should prioritize speed of delivery of 3.0 > over > > > > > >"complete support of every deployment option of Airflow 2" - to > > allow > > > > > >**some** of our early adopter users to migrate quickly, and add > > what's > > > > > >missing for those who would like to move later in later versions. > > > > > >* I think - this is the most important "product management" > decision > > > > here > > > > > - > > > > > >based on the data we have available. Which users do we target for > > > > Airflow > > > > > >3.0, and what we can drop to deliver it faster (and possibly > re-add > > > more > > > > > >stuff for 3.1+). I think it's crucial to the success of the > Airflow > > > 3.0 > > > > > >initiative and the most important decision from the Product > > management > > > > > side > > > > > >that will impact the timeline of Airflow 3.0. While a lot depends > on > > > how > > > > > >much time and effort individuals will spend on implementing > things, > > > the > > > > > >decision of what we drop will impact everyone's speed and overall > > > > delivery > > > > > >of Airflow 3.0. It's a decision we should not take lightly. > > > > > > > > > > > >So my main point is: let's add "what we remove" as a very > important > > > > point > > > > > >in the product discussions we are going to have. > > > > > > > > > > > >J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar > > > > > ><rbish...@amazon.com.inva <mailto:rbish...@amazon.com.inva>lid> > > > wrote: > > > > > > > > > > > >> Like Constance and many others, there was time needed to process > > > this > > > > > >> fantastical summary and approach to Airflow 3. __ > > > > > >> A lot of the points raised by Jarek make sense, ie being > > > prescriptive > > > > on > > > > > >> what features we want to be there on Day 1 vs launched on 3.x - > > this > > > > > does > > > > > >> allow us to move quickly for our users. > > > > > >> In line with Constance's concern, we need to be mindful of what > we > > > > > decide > > > > > >> to keep and what we are willing to cut - which could be a > > > painstaking > > > > > >> process that could offset any gains of trying to be faster. I > also > > > > > believe > > > > > >> that bringing all these new amazing features on Airflow 3 will > > peak > > > > the > > > > > >> interest of early adopters and eventually get others interested > in > > > > > >> migration. However, I believe this migration will be a slow > > process > > > > and > > > > > >> will present a gap in certain functionalities that users may > want > > > > before > > > > > >> entertaining any move to Airflow 3. There are still a lot of > folks > > > > using > > > > > >> v1.10 today. There were several tactical initiatives in the past > > few > > > > > months > > > > > >> with intent on bringing new functionality, ie Multi-team, to > > Airflow > > > > 2.x > > > > > >> and I feel that while these efforts are not wasted, there should > > > still > > > > > be > > > > > >> an option to continue improving Airflow 2 to avoid alienating > our > > > > users > > > > > on > > > > > >> the basis of a future promise in Airflow 3, that may not be easy > > to > > > > > migrate > > > > > >> towards. > > > > > >> > > > > > >> -- Rajesh > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org>> < > > > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> > > > > > >> > > > > > >> > > > > > >> On 2024-05-07, 12:26 PM, "Constance Martineau" > > > > > >> <consta...@astronomer.io.inva <mailto: > > consta...@astronomer.io.inva> > > > <mailto:consta...@astronomer.io.inva <mailto: > > consta...@astronomer.io.inva > > > >> > > > > > >> <mailto:consta...@astronomer.io.inva <mailto: > > > consta...@astronomer.io.inva> <mailto: > > > > > consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva > > > >>>LID> > > > > > >> wrote: > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> CAUTION: This email originated from outside of the organization. > > Do > > > > not > > > > > >> click links or open attachments unless you can confirm the > sender > > > and > > > > > know > > > > > >> the content is safe. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > > externe. > > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > > ne > > > > > pouvez > > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > certain > > > > > que > > > > > >> le contenu ne présente aucun risque. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> Thank you Jarek for the detailed input. I've taken some time to > > > digest > > > > > your > > > > > >> points before responding. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> You've outlined a bold vision for Airflow 3, and I agree that > > being > > > > > >> decisive about the features and architectures will be the key to > > > > > success. > > > > > >> However, before we make final decisions on what features to cut > or > > > > > retain, > > > > > >> it would be beneficial to have a more comprehensive > understanding > > of > > > > how > > > > > >> the current features are utilized by the open-source community. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> @Kaxil Naik <ka...@astronomer.io <mailto:ka...@astronomer.io> > > > <mailto:ka...@astronomer.io <mailto:ka...@astronomer.io>> > > > > <mailto: > > > > > >> ka...@astronomer.io <mailto:ka...@astronomer.io> <mailto: > > > ka...@astronomer.io <mailto:ka...@astronomer.io>>>> recently > > > > initiated a > > > > > >> discussion on > > > > > >> collecting telemetry from open-source deployments: > > > > > >> > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m > > < > > > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > > > > > >> > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> > > < > > > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>> > < > > > > > >> > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> > > < > > > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>> > < > > > > > >> > > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> > > > < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt > > > > > > > ;>. > > > > > >> This data > > > > > >> could be critical in ensuring our decisions are well-informed > and > > > > > reflect > > > > > >> the real-world usage patterns of our users, not just those from > > > > managed > > > > > >> environments like Astro, MWAA or GCC. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> It's essential that we challenge our assumptions and base our > > > > decisions > > > > > on > > > > > >> a holistic view of feature usage. Identifying potential cuts is > a > > > > > critical > > > > > >> step, but let's ensure our strategy aligns with the needs and > > > > > preferences > > > > > >> of the broader Airflow community. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <ja...@potiuk.com > > > <mailto:ja...@potiuk.com> > > > > <mailto: > > > > > >> ja...@potiuk.com <mailto:ja...@potiuk.com>> <mailto: > > > ja...@potiuk.com <mailto:ja...@potiuk.com> <mailto:ja...@potiuk.com > > > <mailto:ja...@potiuk.com> > > > > >>> > > > > > >> wrote: > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > I am currently on sick leave, and still recovering - hoping to > > be > > > > > able to > > > > > >> > travel next week to the US as planned, so I just wanted to > break > > > out > > > > > of > > > > > >> it > > > > > >> > to make one comment here. > > > > > >> > > > > > > >> > I got a clearer head now a bit with medications hopefully > > > working. I > > > > > am > > > > > >> > still taking it that should help me to get over the current > > state, > > > > > and I > > > > > >> > wanted to take a look at this discussion unraveling first. > Over > > > last > > > > > week > > > > > >> > I disconnected from "day-to-day" Airflow and put some thoughts > > (as > > > > > much > > > > > >> as > > > > > >> > I could in my current state) on it. The whole subject of this > > > thread > > > > > was > > > > > >> > started from that - how the current discussions on AIP-67 and > > > others > > > > > >> change > > > > > >> > if we consider Airflow 3 is "starting". > > > > > >> > > > > > > >> > The price for back-compat is speed of development and quality. > > > More > > > > > >> > combinations to test, more unexpected issues uncovered, > > necessity > > > to > > > > > keep > > > > > >> > parallel paths (old/new) while adding new features. All what > > > > Constance > > > > > >> > wrote about and what Ash explained. We already started to trip > > > over > > > > > our > > > > > >> own > > > > > >> > feet mutliple times in a few last releases. Have we tested all > > > > > >> combinations > > > > > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we > > > > already > > > > > see > > > > > >> > that in a number of "combos" of features things are not > working > > in > > > > as > > > > > >> > stable a way as they did before. > > > > > >> > > > > > > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 > > > for a > > > > > long > > > > > >> > time (or even move out) as they will not want to move to > Airflow > > > 3. > > > > A > > > > > lot > > > > > >> > of the work implemented in AIP-44 and design of AIP-67 was > done > > > > around > > > > > >> > back-compatibility. but yes - > > > > > >> > it would have been way easier if designed anew without > > > > > back-compatibility > > > > > >> > in mind. And if we implement it and release it in Airflow 2 it > > > will > > > > > make > > > > > >> > new Airflow feature development even harder. That's why I > wanted > > > to > > > > > treat > > > > > >> > it as "tactical" solution - hoping that in Airflow 3 we can > make > > > it > > > > > >> > "properly" - and that's why I started the discussion here > when I > > > > > sensed > > > > > >> > that we are "close" to Airflow 3 discussion, because I wanted > to > > > see > > > > > what > > > > > >> > options we have there. This is why I have not yet concluded > > voting > > > > on > > > > > >> > AIP-67 waiting for the result of this discussion here. > > > > > >> > > > > > > >> > But if we are ready to go for Airflow.3 then I'd say there are > > two > > > > > >> > important things that should be part of the vision. > > > > > >> > > > > > > >> > 1) *We should be far more opinionated and have far fewer > options > > > of > > > > > >> > running things in Airflow 3*. Even an order of magnitude more > > > > > >> opinionated. > > > > > >> > Make choices, stick to it, perfect those opinionated choices > to > > > suit > > > > > >> 80/20 > > > > > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking > > not > > > > > fitting > > > > > >> > the 20% that might choose to stay at Airflow 2. We can choose > > now > > > > > which > > > > > >> > ~20% of cases we do not want to handle deliberately. And we > > should > > > > be > > > > > >> very, > > > > > >> > very strict about it. Default should be "no choice". This will > > > > > radically > > > > > >> > simplify deployment and should make it easier to simplify > > Airflow > > > > > >> > development and DAG authoring experience because we will have > > less > > > > > cases > > > > > >> to > > > > > >> > support. Even if we plan to add more options in the future, > the > > > > first > > > > > >> > version of Airflow 3 should support one deployment approach > > only. > > > > > This is > > > > > >> > the only way we can deliver it fast. And we should be very > bold > > > > there. > > > > > >> > Choose one option and go for it in pretty much every place we > > have > > > > > >> choices > > > > > >> > now. We should Aim for Airflow 3.0 to support only a subset of > > > > current > > > > > >> > users - but those who are most likely to migrate first and > those > > > > with > > > > > the > > > > > >> > biggest need for the new features. We can think 3.x to support > > > more > > > > > >> cases, > > > > > >> > but 3.0 should be as opinionated as humanly possible. > > > > > >> > > > > > > >> > And this deployment option should be also something ALL our > > > > > stakeholders > > > > > >> > will feel OK with as a way forward in their offering. > > > > > >> > > > > > > >> > My candidates (and yes, some are bold): > > > > > >> > > > > > > >> > * *Drop MySQL*. If we have a single thing that makes us avoid > > our > > > > > schema > > > > > >> > and DB migration - this is the case. Let's choose Postgres 15+ > > and > > > > use > > > > > >> some > > > > > >> > of the great features there. This will also enable much faster > > > async > > > > > SQL > > > > > >> > implementation and a number of other optimisations - not to > > > mention > > > > > >> cutting > > > > > >> > every single change in development and testing time by > literally > > > > half. > > > > > >> And > > > > > >> > we should not look back to adding MySQL. > > > > > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S > > > only > > > > > (and > > > > > >> > AWS/Google others can continue developing theirs of course in > > > > parallel > > > > > >> and > > > > > >> > continue Hybrid executor work). Later - we figure out a better > > > > > solution > > > > > >> to > > > > > >> > support "small" tasks using some new K8S features and possibly > > > > non-k8s > > > > > >> > solutions (Ray-based?) > > > > > >> > * *Cut Connection and Variable Management from DB/UI*. Leave > > only > > > > > Secrets > > > > > >> > Management. Later when we have a 100% extensible React UI, we > > can > > > > add > > > > > a > > > > > >> > "local DB secrets manager" add-on > > > > > >> > * *Choose a single way for DAG storage that will support > > > versioning > > > > > from > > > > > >> > day one*. Bear in mind we can add others later. Bolke's idea > of > > > > using > > > > > >> > FSspec is an interesting one, we should see if it is feasible. > > > > > >> > * *Drop FAB completely (including custom plugins) and invest > in > > > > > >> > implementing Auth Manager based on a dedicated, external > > solution* > > > > > >> > (KeyCloak > > > > > >> > that we've discussed before as a likely candidate) > > > > > >> > * *Leave Providers with Airflow 2 and add tests to make sure > > they > > > > are > > > > > >> > Airflow 3 future-compatible *- develop a way where we continue > > > > > >> development > > > > > >> > and contributions for Providers with Airflow 2 and add > complete > > > > tests > > > > > to > > > > > >> > run them with Airflow 3. This way we can continue developing > > > > Provider > > > > > >> > features independently, and make them work for Airflow 2 (and > > > > continue > > > > > >> > adding features for Airflow 2 users alongside Airflow 2 > > bugfixes), > > > > > while > > > > > >> > also gradually fix any Airflow3 incompatibilities and instead > of > > > > > >> > "back-compatibility" tests make provider > "forward-compatibility" > > > > > tests so > > > > > >> > that future Providers are tested and work on Airflow 3. Also > it > > > will > > > > > make > > > > > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested > > > > without > > > > > >> > investing in changing the current CI / test harness. > > > > > >> > * *Simplify Test Harness for Airflow 3 from the start *- > without > > > > > >> providers > > > > > >> > and 790+ dependencies, we could vastly simplify Airflow3 > testing > > > > > >> (basically > > > > > >> > make CI jobs from scratch) using mostly standard Python > tooling > > > > > (while we > > > > > >> > can continue making use of the current test harness for > Airflow > > 2 > > > + > > > > > >> > Providers and extend it with Airflow 3 future-compatibility > > > tests). > > > > > That > > > > > >> > means Breeze would be only staying in Airflow 2 + Providers > repo > > > as > > > > we > > > > > >> > should be able to achieve most of what we have there with > local > > > > venv/ > > > > > >> > tooling (especially with uv as underlying tooling). > > > > > >> > > > > > > >> > 2) *I think we only add very few new "important" features. > > > *Absolute > > > > > >> > minimum to make Airflow 3 appealing and add them only in > Airflow > > > 3: > > > > > >> > versioning, multi-team, pluggable UI should only be Airflow 3 > - > > it > > > > > makes > > > > > >> no > > > > > >> > sense to invest into Airflow 2 if we already know Airflow 3 is > > > > coming > > > > > - > > > > > >> > that generally triples effort needed to get them out. We > should > > > drop > > > > > new > > > > > >> > features development in Airflow 2. This will give users > > incentive > > > to > > > > > move > > > > > >> > to 3 if the new features will be worth it. Even paying > > > > > >> > compatibility/migration price. > > > > > >> > > > > > > >> > Versionig, for example: I believe if we decide to go only with > > > > > Airflow 3 > > > > > >> > and cut some of the above (Postgres only, Single versioning > DAG > > > > > storage) > > > > > >> we > > > > > >> > can make bolder decisions in versioning and support simpler > > models > > > > > from > > > > > >> the > > > > > >> > get go (and deliver it faster). And we should add only a few - > > but > > > > > >> > important - features that our users clearly asked for and > focus > > on > > > > > >> > delivering Airflow 3 as soon as possible (instead of Airflow > > 2.10 > > > or > > > > > >> 2.11). > > > > > >> > Similarly - multi-team can be simplified if we cut things from > > the > > > > > list > > > > > >> > above and have Task isolation as first-class citizens in > Airflow > > > > (and > > > > > the > > > > > >> > only option). > > > > > >> > > > > > > >> > My candidates very much concur with the list shared by Kaxil > in > > > the > > > > > doc + > > > > > >> > I'd add multi-team (but simplified thanks to the cuts). But I > > also > > > > > here > > > > > >> > would mostly revert to Astronomer, Google. AWS team to define > > > > > >> collectively > > > > > >> > what is the absolute minimum set of features that would get > the > > > > > "target" > > > > > >> > part of their customers happy. And ONLY do that. > > > > > >> > > > > > > >> > So in short - I think the big part of our discussion should be > > > what > > > > we > > > > > >> are > > > > > >> > ready to drop when we start airflow 3 and be very bold. Once > we > > > know > > > > > we > > > > > >> > should figure out the absolute minimum of things that we can > add > > > > that > > > > > >> will > > > > > >> > benefit a significant part of our users (and make use of > > increased > > > > > speed > > > > > >> > because we dropped things). > > > > > >> > > > > > > >> > J. > > > > > >> > > > > > > >> > > > > > > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau > > > > > >> > <consta...@astronomer.io.inva <mailto: > > > consta...@astronomer.io.inva> <mailto:consta...@astronomer.io.inva > > > <mailto:consta...@astronomer.io.inva>> > > > > > >> <mailto:consta...@astronomer.io.inva <mailto: > > > consta...@astronomer.io.inva> <mailto: > > > > > consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva > > > >>>lid> > > > > > >> wrote: > > > > > >> > > > > > > >> > > Hi Michal, > > > > > >> > > > > > > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I > > appreciate > > > > > your > > > > > >> > > concerns about the migration overhead for our users with a > > major > > > > new > > > > > >> > > version and see the appeal in your suggestion to integrate > > many > > > of > > > > > the > > > > > >> > > proposed changes into Airflow 2 through separate AIPs. It’s > a > > > > valid > > > > > >> point > > > > > >> > > and certainly aligns with the value of making incremental > > > > > improvements. > > > > > >> > > > > > > > >> > > However, after looking closely at the enhancements outlined > > for > > > > > Airflow > > > > > >> > 3, > > > > > >> > > I'm convinced they warrant a new major release. Here’s why: > > > > > >> > > > > > > > >> > > 1. *Core Architectural Changes:* We’re looking at > foundational > > > > > changes > > > > > >> > > with Airflow 3—like redefining task priorities, separating > > task > > > > > >> > > definition > > > > > >> > > and task execution, and new AIPs like DAG versioning. remote > > > > > execution > > > > > >> > > and restricting database access from workers. These aren’t > > just > > > > > >> > > incremental > > > > > >> > > improvements but major shifts that will set the stage for > the > > > next > > > > > >> > > decade > > > > > >> > > of Airflow’s architecture. Grouping these changes into a > major > > > > > release > > > > > >> > > will > > > > > >> > > help us make these transitions more cleanly and with fewer > > > > > constraints > > > > > >> > > from > > > > > >> > > past decisions. > > > > > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140 > > > > > >> > deprecated > > > > > >> > > issues, and this will only grow if we continue without a > major > > > > > >> > cleanup. > > > > > >> > > This makes it increasingly difficult to implement new > features > > > > > >> > > effectively > > > > > >> > > while maintaining backward compatibility. A major release > > allows > > > > us > > > > > to > > > > > >> > > address these issues head-on, reducing technical debt and > > paving > > > > the > > > > > >> > way > > > > > >> > > for a more robust platform. > > > > > >> > > 3. *Managing Breaking Changes:* Let’s take the example of > > > > > restricting > > > > > >> > > database access from workers. It’s a necessary move for > better > > > > > >> > security > > > > > >> > > and > > > > > >> > > also potentially scalability reasons (reduces DB load). Many > > > users > > > > > >> > have > > > > > >> > > workflows that interact with the DB, either by using raw sql > > or > > > by > > > > > >> > > leveraging a session object. We could implement this feature > > in > > > > > >> > Airflow > > > > > >> > > 2 > > > > > >> > > and avoid breaking existing workflows by continuing to have > > the > > > > old > > > > > >> > > standard mode as default - much of the work is already done > - > > > but > > > > > that > > > > > >> > > would mean supporting both the new secure mode and the old > > > > standard > > > > > >> > mode > > > > > >> > > indefinitely and design new features with the assumption > that > > > most > > > > > >> > will > > > > > >> > > continue using the old standard mode. With Airflow 3, we can > > > make > > > > > >> > secure > > > > > >> > > mode the default or even the only option, simplifying > > > > implementation > > > > > >> > and > > > > > >> > > future development. This is just one example where it is > > > feasible > > > > to > > > > > >> > > implement in Airflow 2, but is better if we release it under > > the > > > > > >> > > context of > > > > > >> > > Airflow 3. > > > > > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open > up > > > > > >> > > possibilities for handling workflows beyond batch > processing. > > > > > Features > > > > > >> > > like > > > > > >> > > real-time DAG execution through API and multi-language task > > > > support > > > > > >> > are > > > > > >> > > big > > > > > >> > > steps forward, significantly expanding Airflow’s utility. > > > > > >> > > > > > > > >> > > > > > > > >> > > While integrating these updates into Airflow 2 might look > less > > > > > >> disruptive > > > > > >> > > initially, the scale and nature of the required changes > really > > > > > support > > > > > >> a > > > > > >> > > move to Airflow 3. It’s not just about adding new features; > > it’s > > > > > about > > > > > >> > > setting up Airflow so that it continues to remain relevant > for > > > the > > > > > next > > > > > >> > ten > > > > > >> > > years. > > > > > >> > > > > > > > >> > > Constance > > > > > >> > > > > > > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor < > > > a...@apache.org <mailto:a...@apache.org> > > > > > >> <mailto:a...@apache.org <mailto:a...@apache.org>> <mailto: > > > a...@apache.org <mailto:a...@apache.org> <mailto:a...@apache.org <mailto: > > > a...@apache.org> > > > > > >>> > > > > > >> wrote: > > > > > >> > > > > > > > >> > > > There's a lot of technical debt hiding in Airflow, > > especially > > > > the > > > > > >> > > > scheduler that makes it harder and harder to efficiently > add > > > new > > > > > >> > > features. > > > > > >> > > > > > > > > >> > > > At some point, very soon, we are going to have to remove > > some > > > > very > > > > > >> > > > infrequently used back compat shims that negatively affect > > > > > >> performance. > > > > > >> > > > Without doing that the pace at which we can realistically > > add > > > > > some of > > > > > >> > the > > > > > >> > > > more exciting features tends towards zero. Developer speed > > of > > > > > >> > > contributors > > > > > >> > > > is a factor here too! > > > > > >> > > > > > > > > >> > > > So while we are still using SemVer, that necessitates v3. > > > > > >> > > > > > > > > >> > > > Ash > > > > > >> > > > > > > > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" < > > > > > michalmod...@google.com <mailto:michalmod...@google.com> > > > > > >> <mailto:michalmod...@google.com <mailto:michalmod...@google.com > >> > > > <mailto:michalmod...@google.com <mailto:michalmod...@google.com> > > > > > <mailto: > > > > > >> michalmod...@google.com <mailto:michalmod...@google.com>>> > > > > > >> > > .INVALID> > > > > > >> > > > wrote: > > > > > >> > > > >+1 to Jens's & Bolke's points here and in the doc > > > > > >> > > > > > > > > > >> > > > >I agree we should work on clarifying the directions we > > would > > > > like > > > > > >> > > Airflow > > > > > >> > > > >to go. Introducing a new major Airflow version is a > massive > > > > > overhead > > > > > >> > for > > > > > >> > > > >users, who would need to plan for migrations, onboarding > > the > > > > new > > > > > >> > Airflow > > > > > >> > > > >(with a slightly different architecture), etc., and > > > effectively > > > > > >> > Airflow > > > > > >> > > 2 > > > > > >> > > > >would live in parallel for a long time. > > > > > >> > > > > > > > > > >> > > > >Personally, I think most of the points in > Kaxil's/Vikram's > > > doc > > > > > are > > > > > >> > > > valuable > > > > > >> > > > >projects of their own, and I could imagine all of them > > being > > > > > >> delivered > > > > > >> > > as > > > > > >> > > > >separate AIPs within Airflow 2 (surely new minor versions > > of > > > > > Airflow > > > > > >> > > 2). I > > > > > >> > > > >am not sure if the scope of changes and the goal we want > to > > > > > achieve > > > > > >> is > > > > > >> > > a) > > > > > >> > > > >clear enough b) broad enough to call for a new major > > version. > > > > > >> > > > > > > > > > >> > > > >Best, > > > > > >> > > > >Michal > > > > > >> > > > > > > > > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens > > > > (XC-AS/EAE-ADA-T) > > > > > >> > > > ><jens.scheff...@de.bosch.com.inva <mailto: > > > jens.scheff...@de.bosch.com.inva> <mailto: > > > > > >> jens.scheff...@de.bosch.com.inva <mailto: > > > jens.scheff...@de.bosch.com.inva>> <mailto: > > > > > jens.scheff...@de.bosch.com.inva <mailto: > > > jens.scheff...@de.bosch.com.inva> > > > > > >> <mailto:jens.scheff...@de.bosch.com.inva <mailto: > > > jens.scheff...@de.bosch.com.inva>>>lid> wrote: > > > > > >> > > > > > > > > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this > is > > > > > mostly a > > > > > >> > > > vision > > > > > >> > > > >> statement. > > > > > >> > > > >> > > > > > >> > > > >> Looking forward for a larger addendum where we can > > collect > > > > > things > > > > > >> > that > > > > > >> > > > we > > > > > >> > > > >> all can vote and agree on as targets. > > > > > >> > > > >> > > > > > >> > > > >> As I started earlier with a confluence page and it > seems > > > this > > > > > is > > > > > >> not > > > > > >> > > > >> accessible to all, shall we convert this to a Google > Doc > > > for > > > > > >> better > > > > > >> > > > >> collaboration and item collection? > > > > > >> > > > >> > > > > > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> < > > > https://aka.ms/o0ukef>> < > > > > > >> https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>> > < > > > https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>> < > > > > > >> https://aka.ms/o0ukef&gt;>> < > > > https://aka.ms/o0ukef&amp;gt;&gt;>> > > > > > >> > > > >> ________________________________ > > > > > >> > > > >> From: Vikram Koka <vik...@astronomer.io.inva <mailto: > > > vik...@astronomer.io.inva> <mailto: > > > > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>> > > > <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > > > <mailto: > > > > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva > > >>>LID> > > > > > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM > > > > > >> > > > >> To: dev@airflow.apache.org <mailto: > > dev@airflow.apache.org> > > > <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> > > > > > >> <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > > <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> < > > > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org>> <mailto: > > > > > >> dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org>>>> > > > > > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical > > > (Airflow > > > > > 2) > > > > > >> vs > > > > > >> > > > >> strategic (Airflow 3) approach > > > > > >> > > > >> > > > > > >> > > > >> Thank you for your feedback, Bolke and Andrey! > > > > > >> > > > >> > > > > > >> > > > >> Bolke, > > > > > >> > > > >> I have replied to some of your comments in the doc. > > > > > >> > > > >> I will provide a detailed write up on the "Interactive > > DAG > > > > run" > > > > > >> (or > > > > > >> > > > >> synchronous DAG run) capability, which has generated > some > > > > early > > > > > >> > > > questions. > > > > > >> > > > >> I had intended to get an AIP published for that as a > > > > follow-up, > > > > > >> but > > > > > >> > I > > > > > >> > > > >> believe that a simpler write up would be useful ahead > of > > > the > > > > > AIP. > > > > > >> > > > >> > > > > > >> > > > >> Andrey, > > > > > >> > > > >> You raise an interesting point. > > > > > >> > > > >> > > > > > >> > > > >> As part of the Airflow 2.0 release, we as a community > had > > > > > decided > > > > > >> to > > > > > >> > > > >> strictly adhere to Semver as detailed in the document > you > > > > > >> > referenced. > > > > > >> > > We > > > > > >> > > > >> also consciously split out the "Core Airflow" releases > > from > > > > the > > > > > >> > > > "Provider" > > > > > >> > > > >> releases at that time. We had a clear expectation then > > for > > > > the > > > > > >> > cadence > > > > > >> > > > of > > > > > >> > > > >> both minor and patch releases, which we have generally > > > > adhered > > > > > to > > > > > >> > > since > > > > > >> > > > >> then. > > > > > >> > > > >> > > > > > >> > > > >> Personally, I am more concerned about our Provider > > releases > > > > > right > > > > > >> > now, > > > > > >> > > > as > > > > > >> > > > >> compared to the cadence of our major releases. I > believe > > > that > > > > > one > > > > > >> of > > > > > >> > > the > > > > > >> > > > >> proposed changes in the Airflow 3 document i.e. the > clear > > > > > >> separation > > > > > >> > > for > > > > > >> > > > >> Task Execution will help here, but more may be needed. > > > > > >> > > > >> > > > > > >> > > > >> Definitely interested in more feedback on this as well. > > > > > >> > > > >> > > > > > >> > > > >> Vikram > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin < > > > > > >> > > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> > > > <mailto:andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>> > > > > <mailto: > > > > > >> andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> > > <mailto: > > > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>>> > > > > > >> > > > > > > > > > >> > > > >> wrote: > > > > > >> > > > >> > > > > > >> > > > >> > I would like to propose to change (at least discuss) > > > > release > > > > > >> > policy > > > > > >> > > > >> around > > > > > >> > > > >> > the Major version of Airflow. > > > > > >> > > > >> > > > > > > >> > > > >> > Right now it is described as "These releases do not > > > happen > > > > > with > > > > > >> > any > > > > > >> > > > >> regular > > > > > >> > > > >> > interval or on any predictable schedule." : > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > > < > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > > > > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > > < > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0 > > > > > > > > > > > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > > > < > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0 > > > > > > > > > > > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> > > > < > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt > > > > > > > > > >> ;> > > > > > >> > > > >> < > > > > > >> > > > >> > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > > > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > > > > > > > > > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > > > > > > > > > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > > > < > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt > > > > > > > > > >> ;> > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > So maybe it is time to make it schedulable, e.g. one > > per > > > > two > > > > > >> years > > > > > >> > > or > > > > > >> > > > so. > > > > > >> > > > >> > This one could help us to avoid such a discussion in > > the > > > > > future, > > > > > >> > > like > > > > > >> > > > "We > > > > > >> > > > >> > don't know when Airflow 4 is coming.". At the moment > > when > > > > the > > > > > >> new > > > > > >> > > > major > > > > > >> > > > >> > version will be released new features wouldn't be > added > > > in > > > > > the > > > > > >> old > > > > > >> > > > major > > > > > >> > > > >> > version, however we would support bug / security for > a > > > > while, > > > > > >> > e.g. 1 > > > > > >> > > > year > > > > > >> > > > >> > for bug fixes, 3 years for security fixes with a > total > > 5 > > > > year > > > > > >> > > > lifecycle > > > > > >> > > > >> per > > > > > >> > > > >> > a major version. These just are approximate time > > periods > > > > for > > > > > a > > > > > >> > > > definition > > > > > >> > > > >> > of current period, bugfix period and security fix > > period. > > > > > >> > > > >> > > > > > > >> > > > >> > In contributors' perspective it helps with dropping > the > > > > > >> deprecated > > > > > >> > > > stuff > > > > > >> > > > >> > which resolves some old problem: we have to support > > > > > everything > > > > > >> > > > including > > > > > >> > > > >> > deprecated stuff and without schedulable lifecycle > for > > > the > > > > > >> > > deprecated > > > > > >> > > > >> stuff > > > > > >> > > > >> > it could be showstopper for the new feature, because > > > > > sometimes > > > > > >> it > > > > > >> > > > hard to > > > > > >> > > > >> > support two different approaches for long period of > > time > > > > > with no > > > > > >> > > hope > > > > > >> > > > >> that > > > > > >> > > > >> > it will happen soon. For some fundamental stuff which > > do > > > > not > > > > > >> > > require a > > > > > >> > > > >> lot > > > > > >> > > > >> > things time to support we could postponed removal for > > > next > > > > > after > > > > > >> > the > > > > > >> > > > next > > > > > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it > in > > > > > Airflow 5 > > > > > >> > > > >> > > > > > > >> > > > >> > In the user perspective, they have at least bug fix > > > support > > > > > for > > > > > >> a > > > > > >> > > > while, > > > > > >> > > > >> if > > > > > >> > > > >> > someone want to use legacy version it their choice, > > > however > > > > > no > > > > > >> new > > > > > >> > > > >> > features, no new version of providers (after one > year) > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > ---- > > > > > >> > > > >> > Best Wishes > > > > > >> > > > >> > *Andrey Anshin* > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin < > > > > > bdbr...@gmail.com <mailto:bdbr...@gmail.com> > > > > > >> <mailto:bdbr...@gmail.com <mailto:bdbr...@gmail.com>> <mailto: > > > bdbr...@gmail.com <mailto:bdbr...@gmail.com> <mailto: > > > > > >> bdbr...@gmail.com <mailto:bdbr...@gmail.com>>>> > > > > > >> > > > wrote: > > > > > >> > > > >> > > > > > > >> > > > >> > > I have left several comments :-). And on > interactive > > > dag > > > > > runs > > > > > >> > even > > > > > >> > > > >> after > > > > > >> > > > >> > > the explanation of Vikram I still don't have a clue > > > what > > > > we > > > > > >> want > > > > > >> > > to > > > > > >> > > > >> > > accomplish there :-P. > > > > > >> > > > >> > > > > > > > >> > > > >> > > I would like to see a mantra or team for Airflow 3. > > > That > > > > > helps > > > > > >> > > > nudging > > > > > >> > > > >> > > people in the same direction. Suggestions in the > > > > comments. > > > > > >> > > > >> > > > > > > > >> > > > >> > > Bolke > > > > > >> > > > >> > > Sent from my iPhone > > > > > >> > > > >> > > > > > > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka > > > > > >> > > <vik...@astronomer.io.inva <mailto: > vik...@astronomer.io.inva> > > > <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>> > > > > > <mailto: > > > > > >> vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> > > > <mailto:vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva > > >>>lid > > > > > >> > > > > > > > > > >> > > > >> > > wrote: > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > Good point Jed. > > > > > >> > > > >> > > > I responded back to your comment in the doc as > well > > > and > > > > > very > > > > > >> > > open > > > > > >> > > > to > > > > > >> > > > >> > > > changing the term in the doc. > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > Used the term "interactive DAG run" as the > ability > > to > > > > > invoke > > > > > >> > or > > > > > >> > > > >> > trigger a > > > > > >> > > > >> > > > DAG run through the API, with the expectation of > > > > getting > > > > > >> back > > > > > >> > a > > > > > >> > > > >> result > > > > > >> > > > >> > > > immediately. An alternate term could be a > > > "synchronous > > > > > DAG > > > > > >> > run". > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > Regardless, this is a significant change so a > good > > > term > > > > > to > > > > > >> > > > indicate > > > > > >> > > > >> the > > > > > >> > > > >> > > > expansion from "batch runs only" is warranted. > Very > > > > open > > > > > to > > > > > >> > > > different > > > > > >> > > > >> > > terms > > > > > >> > > > >> > > > here. > > > > > >> > > > >> > > > > > > > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham < > > > > > >> > > > >> > jedcunning...@apache.org <mailto: > > > jedcunning...@apache.org> <mailto:jedcunning...@apache.org <mailto: > > > jedcunning...@apache.org>> > > > > > >> <mailto:jedcunning...@apache.org <mailto: > jedcunning...@apache.org > > > > > > <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org>>> > > > > > >> > > > >> > > > > > > > > >> > > > >> > > >> wrote: > > > > > >> > > > >> > > >> > > > > > >> > > > >> > > >> Very exciting! Looks like we will have a busy > > period > > > > of > > > > > >> time > > > > > >> > > > ahead > > > > > >> > > > >> of > > > > > >> > > > >> > > us. > > > > > >> > > > >> > > >> Overall I like the plan so far, especially using > > > this > > > > > >> year's > > > > > >> > > > Airflow > > > > > >> > > > >> > > Summit > > > > > >> > > > >> > > >> as an opportunity to announce and gather > feedback, > > > and > > > > > the > > > > > >> > 2025 > > > > > >> > > > >> > version > > > > > >> > > > >> > > to > > > > > >> > > > >> > > >> pitch upgrading. > > > > > >> > > > >> > > >> > > > > > >> > > > >> > > >> I left a comment in the doc, but we might want > to > > > > > iterate > > > > > >> on > > > > > >> > > the > > > > > >> > > > >> > > >> terminology we use for high priority or > > > "synchronous" > > > > > DAG > > > > > >> > runs > > > > > >> > > to > > > > > >> > > > >> > serve > > > > > >> > > > >> > > LLM > > > > > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit > > > > > confusing. > > > > > >> > > > >> > > >> > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > > >> > > > > > > > > > --------------------------------------------------------------------- > > > > > >> > > > >> > > To unsubscribe, e-mail: > > > > dev-unsubscr...@airflow.apache.org <mailto: > > > dev-unsubscr...@airflow.apache.org> > > > > > >> <mailto:dev-unsubscr...@airflow.apache.org <mailto: > > > dev-unsubscr...@airflow.apache.org>> <mailto: > > > > > >> dev-unsubscr...@airflow.apache.org <mailto: > > > dev-unsubscr...@airflow.apache.org> <mailto: > > > > > >> dev-unsubscr...@airflow.apache.org <mailto: > > > dev-unsubscr...@airflow.apache.org>>> > > > > > >> > > > >> > > For additional commands, e-mail: > > > > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > > > >> <mailto:dev-h...@airflow.apache.org <mailto: > > > dev-h...@airflow.apache.org>> <mailto: > > > > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > > > >> <mailto:dev-h...@airflow.apache.org <mailto: > > > dev-h...@airflow.apache.org>>> > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >
