I understand that the compatibility layer mentioned by TP would allow providers to work with both Airflow 2.x and 3.x. I could see it working in the following ways: 1) You can use DAGs with Airflow 3.x syntax in Airflow 2.x, so you can migrate your DAGs gradually to 3.x syntax before upgrading to it (until then you stay on Airflow 2.x). 2) You can use DAGs with Airflow 2.x syntax in Airflow 3.x, so you can upgrade to 3.x and then gradually migrate your DAGs to 3.x syntax.
Which one do you have in mind? I think 2) would ease the upgrade path to Airflow 3.x and encourage users to try it out, while 1) would have the effect of keeping users at Airflow 2.x, as they cannot upgrade before modifying their DAGs. On Mon, Aug 5, 2024 at 9:15 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Also I have a feeling that with AST / smart conversion, we should be able > to get an almost 100% accurate automated conversion of that one. It is > "almost" (of-course there will be edge cases), but it looks like a pretty > automate'able conversion. > > Maybe - I am not sure how possible it is due to contracts etc. - but maybe > also the managed service teams could run such conversion on the vast number > of DAGs people have out there and "perfect" the conversion mechanism. > > I think - again - what's really crucial here is that it does not have to be > a big-bang. If we will have prolonged period of time, where compatible > approach will happily coexist with new approach - for Airflow 2.11 and > Airflow 3 and when you will still be able to run part of your DAGS with > "old ways" and gradually converting your DAGs to "new ways", ability to > track the progress and "nagging warnings" - and you have a converter that > works "out-of-the-box" for "almost" everything - this is the maximum > support we can give to our users. > > J. > > > On Mon, Aug 5, 2024 at 8:52 AM Tzu-ping Chung <t...@astronomer.io.invalid> > wrote: > > > I don’t know how to compare the efforts since honestly I have very little > > idea how many templated fields people tend to have. I can outline what > each > > group needs to do though. > > > > Airflow maintainers > > - Develop the compatibility layer > > - One time work before 3.0 > > - Maintenance afterwards, at least until we drop 2.x support from all > > providers > > - Apply compatibility layer to all providers > > - Drop compatibility layer from providers when they drop 2.x support > > - Not a hard requirement > > > > Third-party operator authors > > - Apply compatibility layer to all operators > > - Drop compatibility layer from operators when dropping 2.x support > > - Not a hard requirement > > > > DAG authors (that want to upgrade to Airflow 3) > > - Same as third-party operator authors if the real authors don’t want to > > upgrade > > - Convert DAGs to use the new syntax > > > > > > > On 30 Jul 2024, at 04:55, Scheffler Jens (XC-AS/EAE-ADA-T) < > > jens.scheff...@de.bosch.com.INVALID> wrote: > > > > > > Thanks TP for the rework. > > > > > > I added some comments (iteration 2) on the migration ideas. I think > > these details make it clearer, still i have partial doubts how much > burden > > we add to users to migrate DAGs to get to version 3. I very much favor > the > > new templating but am not sure how many DAG authors we leave with > migration > > problems behind. > > > Do we have a guess or estimation how much burden we as airflow > > developers need to keep as compatability compared to the amount of DAG > > templates that people neet to adjust? > > > > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> > > > ________________________________ > > > From: Tzu-ping Chung <t...@astronomer.io.INVALID> > > > Sent: Monday, July 29, 2024 8:54:58 PM > > > To: dev@airflow.apache.org <dev@airflow.apache.org> > > > Cc: michalmod...@google.com <michalmod...@google.com> > > > Subject: Re: [VOTE] AIP-80: Explicit Template Fields in Operator > > Arguments > > > > > > I have updated the AIP to include the additional compatibility > > discussions in this thread. Please take a look again. > > > > > > Specifically (although by no means exclusively) it would be awesome if > > Michał you could have a look and see if it addresses more of the concerns > > and could be viable for you. Although the vote is non-binding, I still > > would like to be more confident I tried to address the real concerns from > > the community, which is the real problem when it comes to migration. > > > > > > Link to document for convenience > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fx%2F2grOEg&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9318251bd2fe4490b64308dcafffffe5%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638578761240102546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=vVICot9p06VQJ56QmP9dwvEaWfpR7nCcMVqUexl6B3w%3D&reserved=0 > > <https://cwiki.apache.org/confluence/x/2grOEg> > > > > > > TP > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
