+1 to Jens's & Bolke's points here and in the doc I agree we should work on clarifying the directions we would like Airflow to go. Introducing a new major Airflow version is a massive overhead for users, who would need to plan for migrations, onboarding the new Airflow (with a slightly different architecture), etc., and effectively Airflow 2 would live in parallel for a long time.
Personally, I think most of the points in Kaxil's/Vikram's doc are valuable projects of their own, and I could imagine all of them being delivered as separate AIPs within Airflow 2 (surely new minor versions of Airflow 2). I am not sure if the scope of changes and the goal we want to achieve is a) clear enough b) broad enough to call for a new major version. Best, Michal On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com.invalid> wrote: > Thanks for the document write-up, Kaxil. I assume this is mostly a vision > statement. > > Looking forward for a larger addendum where we can collect things that we > all can vote and agree on as targets. > > As I started earlier with a confluence page and it seems this is not > accessible to all, shall we convert this to a Google Doc for better > collaboration and item collection? > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > ________________________________ > From: Vikram Koka <vik...@astronomer.io.INVALID> > Sent: Sunday, May 5, 2024 3:34:33 AM > To: dev@airflow.apache.org <dev@airflow.apache.org> > Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs > strategic (Airflow 3) approach > > Thank you for your feedback, Bolke and Andrey! > > Bolke, > I have replied to some of your comments in the doc. > I will provide a detailed write up on the "Interactive DAG run" (or > synchronous DAG run) capability, which has generated some early questions. > I had intended to get an AIP published for that as a follow-up, but I > believe that a simpler write up would be useful ahead of the AIP. > > Andrey, > You raise an interesting point. > > As part of the Airflow 2.0 release, we as a community had decided to > strictly adhere to Semver as detailed in the document you referenced. We > also consciously split out the "Core Airflow" releases from the "Provider" > releases at that time. We had a clear expectation then for the cadence of > both minor and patch releases, which we have generally adhered to since > then. > > Personally, I am more concerned about our Provider releases right now, as > compared to the cadence of our major releases. I believe that one of the > proposed changes in the Airflow 3 document i.e. the clear separation for > Task Execution will help here, but more may be needed. > > Definitely interested in more feedback on this as well. > > Vikram > > > On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <andrey.ans...@taragol.is> > wrote: > > > I would like to propose to change (at least discuss) release policy > around > > the Major version of Airflow. > > > > Right now it is described as "These releases do not happen with any > regular > > interval or on any predictable schedule." : > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > > > > > > So maybe it is time to make it schedulable, e.g. one per two years or so. > > This one could help us to avoid such a discussion in the future, like "We > > don't know when Airflow 4 is coming.". At the moment when the new major > > version will be released new features wouldn't be added in the old major > > version, however we would support bug / security for a while, e.g. 1 year > > for bug fixes, 3 years for security fixes with a total 5 year lifecycle > per > > a major version. These just are approximate time periods for a definition > > of current period, bugfix period and security fix period. > > > > In contributors' perspective it helps with dropping the deprecated stuff > > which resolves some old problem: we have to support everything including > > deprecated stuff and without schedulable lifecycle for the deprecated > stuff > > it could be showstopper for the new feature, because sometimes it hard to > > support two different approaches for long period of time with no hope > that > > it will happen soon. For some fundamental stuff which do not require a > lot > > things time to support we could postponed removal for next after the next > > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5 > > > > In the user perspective, they have at least bug fix support for a while, > if > > someone want to use legacy version it their choice, however no new > > features, no new version of providers (after one year) > > > > > > ---- > > Best Wishes > > *Andrey Anshin* > > > > > > > > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > > > I have left several comments :-). And on interactive dag runs even > after > > > the explanation of Vikram I still don't have a clue what we want to > > > accomplish there :-P. > > > > > > I would like to see a mantra or team for Airflow 3. That helps nudging > > > people in the same direction. Suggestions in the comments. > > > > > > Bolke > > > Sent from my iPhone > > > > > > > On 4 May 2024, at 01:14, Vikram Koka <vik...@astronomer.io.invalid> > > > wrote: > > > > > > > > Good point Jed. > > > > I responded back to your comment in the doc as well and very open to > > > > changing the term in the doc. > > > > > > > > Used the term "interactive DAG run" as the ability to invoke or > > trigger a > > > > DAG run through the API, with the expectation of getting back a > result > > > > immediately. An alternate term could be a "synchronous DAG run". > > > > > > > > Regardless, this is a significant change so a good term to indicate > the > > > > expansion from "batch runs only" is warranted. Very open to different > > > terms > > > > here. > > > > > > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham < > > jedcunning...@apache.org > > > > > > > >> wrote: > > > >> > > > >> Very exciting! Looks like we will have a busy period of time ahead > of > > > us. > > > >> Overall I like the plan so far, especially using this year's Airflow > > > Summit > > > >> as an opportunity to announce and gather feedback, and the 2025 > > version > > > to > > > >> pitch upgrading. > > > >> > > > >> I left a comment in the doc, but we might want to iterate on the > > > >> terminology we use for high priority or "synchronous" DAG runs to > > serve > > > LLM > > > >> responses - I find "interactive DAG runs" a bit confusing. > > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
