I think ensuring smooth migration should be the goal of every AIP that makes it part of Airflow 3, as we discussed in the dev call on 4th of June: >Ensure a smoother migration path between Airflow 2 and 3, particularly for DAG authors using the existing official Airflow providers. Breaking things and throwing the support of code mitigating it over the fence should not be the approach we take if we value the user experience.
On Wed, Aug 7, 2024 at 10:32 AM Tzu-ping Chung <t...@astronomer.io.invalid> wrote: > I think I’m fine with having this as a provider that if someone wants to > maintain it. Not every provider needs to be maintained by every Airflow > maintainer anyway. I’m not making it a goal for the AIP, but there’s also > nothing in there that would prevent it from happening. While I don’t see > myself maintaining the provider, I’ll be happy to tweak things if it makes > the implementation easier too. > > TP > > > On 7 Aug 2024, at 16:19, Michał Modras <michalmod...@google.com.INVALID> > wrote: > > > > I don't think (obviously) is so obvious. While 2) might be seen as not > > incentivising to change anything (btw software forcing people to change > > something that works just fine does not sound great to me, but I guess > > that's a philosophical thing - if software is meant to be for users or > > maintainers :)), it actually will discentivise some users from moving to > > Airflow 3 entirely. If we care for Airflow 3 adoption and not putting > > blockers for users for it, we should go with 2). I believe it should be > at > > least a provider and maintained. > > > > > > On Wed, Aug 7, 2024 at 10:18 AM Michał Modras <michalmod...@google.com> > > wrote: > > > >> I don't think (obviously) is so obvious. While 1) might be seen as not > >> incentivising to change anything (btw software forcing people to change > >> something that works just fine does not sound great to me, but I guess > >> that's a philosophical thing - if software is meant to be for users or > >> maintainers :)), it actually will discentivise some users from moving to > >> Airflow 3 entirely. If we care for Airflow 3 adoption and not putting > >> blockers for users for it, we should go with 2). I believe it should be > at > >> least a provider and maintained. > >> > >> > >> On Wed, Aug 7, 2024 at 10:09 AM Tzu-ping Chung <t...@astronomer.io.invalid > > > >> wrote: > >> > >>> The canonical upgrade path would be 1) since we want people to use the > >>> new syntax (obviously), and allowing 2) would encourage people to never > >>> change anything, bringing on additional maintenance burden on Airflow > >>> maintainers indefinitely. > >>> > >>> However, since the compatibility layer needs to support both syntaxes > on > >>> Airflow 2, it would also contain the actual mechanism to make 2) work. > I > >>> don’t want to include it in Airflow (neither core nor provider), but a > >>> third-party package can be created for this (I can do that when > working on > >>> the canonical compatibility layer). But it will not be actively > maintained. > >>> > >>> TP > >>> > >>> > >>>> On 6 Aug 2024, at 04:55, Michał Modras <michalmod...@google.com > .INVALID> > >>> wrote: > >>>> > >>>> I understand that the compatibility layer mentioned by TP would allow > >>>> providers to work with both Airflow 2.x and 3.x. I could see it > working > >>> in > >>>> the following ways: > >>>> 1) You can use DAGs with Airflow 3.x syntax in Airflow 2.x, so you can > >>>> migrate your DAGs gradually to 3.x syntax before upgrading to it > (until > >>>> then you stay on Airflow 2.x). > >>>> 2) You can use DAGs with Airflow 2.x syntax in Airflow 3.x, so you can > >>>> upgrade to 3.x and then gradually migrate your DAGs to 3.x syntax. > >>>> > >>>> Which one do you have in mind? I think 2) would ease the upgrade path > to > >>>> Airflow 3.x and encourage users to try it out, while 1) would have the > >>>> effect of keeping users at Airflow 2.x, as they cannot upgrade before > >>>> modifying their DAGs. > >>>> > >>>> On Mon, Aug 5, 2024 at 9:15 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >>>> > >>>>> Also I have a feeling that with AST / smart conversion, we should be > >>> able > >>>>> to get an almost 100% accurate automated conversion of that one. It > is > >>>>> "almost" (of-course there will be edge cases), but it looks like a > >>> pretty > >>>>> automate'able conversion. > >>>>> > >>>>> Maybe - I am not sure how possible it is due to contracts etc. - but > >>> maybe > >>>>> also the managed service teams could run such conversion on the vast > >>> number > >>>>> of DAGs people have out there and "perfect" the conversion mechanism. > >>>>> > >>>>> I think - again - what's really crucial here is that it does not have > >>> to be > >>>>> a big-bang. If we will have prolonged period of time, where > compatible > >>>>> approach will happily coexist with new approach - for Airflow 2.11 > and > >>>>> Airflow 3 and when you will still be able to run part of your DAGS > with > >>>>> "old ways" and gradually converting your DAGs to "new ways", ability > to > >>>>> track the progress and "nagging warnings" - and you have a converter > >>> that > >>>>> works "out-of-the-box" for "almost" everything - this is the maximum > >>>>> support we can give to our users. > >>>>> > >>>>> J. > >>>>> > >>>>> > >>>>> On Mon, Aug 5, 2024 at 8:52 AM Tzu-ping Chung > <t...@astronomer.io.invalid > >>>> > >>>>> wrote: > >>>>> > >>>>>> I don’t know how to compare the efforts since honestly I have very > >>> little > >>>>>> idea how many templated fields people tend to have. I can outline > what > >>>>> each > >>>>>> group needs to do though. > >>>>>> > >>>>>> Airflow maintainers > >>>>>> - Develop the compatibility layer > >>>>>> - One time work before 3.0 > >>>>>> - Maintenance afterwards, at least until we drop 2.x support from > >>> all > >>>>>> providers > >>>>>> - Apply compatibility layer to all providers > >>>>>> - Drop compatibility layer from providers when they drop 2.x support > >>>>>> - Not a hard requirement > >>>>>> > >>>>>> Third-party operator authors > >>>>>> - Apply compatibility layer to all operators > >>>>>> - Drop compatibility layer from operators when dropping 2.x support > >>>>>> - Not a hard requirement > >>>>>> > >>>>>> DAG authors (that want to upgrade to Airflow 3) > >>>>>> - Same as third-party operator authors if the real authors don’t > want > >>> to > >>>>>> upgrade > >>>>>> - Convert DAGs to use the new syntax > >>>>>> > >>>>>> > >>>>>>> On 30 Jul 2024, at 04:55, Scheffler Jens (XC-AS/EAE-ADA-T) < > >>>>>> jens.scheff...@de.bosch.com.INVALID> wrote: > >>>>>>> > >>>>>>> Thanks TP for the rework. > >>>>>>> > >>>>>>> I added some comments (iteration 2) on the migration ideas. I think > >>>>>> these details make it clearer, still i have partial doubts how much > >>>>> burden > >>>>>> we add to users to migrate DAGs to get to version 3. I very much > favor > >>>>> the > >>>>>> new templating but am not sure how many DAG authors we leave with > >>>>> migration > >>>>>> problems behind. > >>>>>>> Do we have a guess or estimation how much burden we as airflow > >>>>>> developers need to keep as compatability compared to the amount of > DAG > >>>>>> templates that people neet to adjust? > >>>>>>> > >>>>>>> Sent from Outlook for iOS<https://aka.ms/o0ukef> > >>>>>>> ________________________________ > >>>>>>> From: Tzu-ping Chung <t...@astronomer.io.INVALID> > >>>>>>> Sent: Monday, July 29, 2024 8:54:58 PM > >>>>>>> To: dev@airflow.apache.org <dev@airflow.apache.org> > >>>>>>> Cc: michalmod...@google.com <michalmod...@google.com> > >>>>>>> Subject: Re: [VOTE] AIP-80: Explicit Template Fields in Operator > >>>>>> Arguments > >>>>>>> > >>>>>>> I have updated the AIP to include the additional compatibility > >>>>>> discussions in this thread. Please take a look again. > >>>>>>> > >>>>>>> Specifically (although by no means exclusively) it would be awesome > >>> if > >>>>>> Michał you could have a look and see if it addresses more of the > >>> concerns > >>>>>> and could be viable for you. Although the vote is non-binding, I > still > >>>>>> would like to be more confident I tried to address the real concerns > >>> from > >>>>>> the community, which is the real problem when it comes to migration. > >>>>>>> > >>>>>>> Link to document for convenience > >>>>>> > >>>>> > >>> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fx%2F2grOEg&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9318251bd2fe4490b64308dcafffffe5%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638578761240102546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=vVICot9p06VQJ56QmP9dwvEaWfpR7nCcMVqUexl6B3w%3D&reserved=0 > >>>>>> <https://cwiki.apache.org/confluence/x/2grOEg> > >>>>>>> > >>>>>>> TP > >>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>>>>> > >>>>>> > >>>>> > >>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>> > >>> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
