I think I’m fine with having this as a provider that if someone wants to maintain it. Not every provider needs to be maintained by every Airflow maintainer anyway. I’m not making it a goal for the AIP, but there’s also nothing in there that would prevent it from happening. While I don’t see myself maintaining the provider, I’ll be happy to tweak things if it makes the implementation easier too.
TP > On 7 Aug 2024, at 16:19, Michał Modras <michalmod...@google.com.INVALID> > wrote: > > I don't think (obviously) is so obvious. While 2) might be seen as not > incentivising to change anything (btw software forcing people to change > something that works just fine does not sound great to me, but I guess > that's a philosophical thing - if software is meant to be for users or > maintainers :)), it actually will discentivise some users from moving to > Airflow 3 entirely. If we care for Airflow 3 adoption and not putting > blockers for users for it, we should go with 2). I believe it should be at > least a provider and maintained. > > > On Wed, Aug 7, 2024 at 10:18 AM Michał Modras <michalmod...@google.com> > wrote: > >> I don't think (obviously) is so obvious. While 1) might be seen as not >> incentivising to change anything (btw software forcing people to change >> something that works just fine does not sound great to me, but I guess >> that's a philosophical thing - if software is meant to be for users or >> maintainers :)), it actually will discentivise some users from moving to >> Airflow 3 entirely. If we care for Airflow 3 adoption and not putting >> blockers for users for it, we should go with 2). I believe it should be at >> least a provider and maintained. >> >> >> On Wed, Aug 7, 2024 at 10:09 AM Tzu-ping Chung <t...@astronomer.io.invalid> >> wrote: >> >>> The canonical upgrade path would be 1) since we want people to use the >>> new syntax (obviously), and allowing 2) would encourage people to never >>> change anything, bringing on additional maintenance burden on Airflow >>> maintainers indefinitely. >>> >>> However, since the compatibility layer needs to support both syntaxes on >>> Airflow 2, it would also contain the actual mechanism to make 2) work. I >>> don’t want to include it in Airflow (neither core nor provider), but a >>> third-party package can be created for this (I can do that when working on >>> the canonical compatibility layer). But it will not be actively maintained. >>> >>> TP >>> >>> >>>> On 6 Aug 2024, at 04:55, Michał Modras <michalmod...@google.com.INVALID> >>> wrote: >>>> >>>> I understand that the compatibility layer mentioned by TP would allow >>>> providers to work with both Airflow 2.x and 3.x. I could see it working >>> in >>>> the following ways: >>>> 1) You can use DAGs with Airflow 3.x syntax in Airflow 2.x, so you can >>>> migrate your DAGs gradually to 3.x syntax before upgrading to it (until >>>> then you stay on Airflow 2.x). >>>> 2) You can use DAGs with Airflow 2.x syntax in Airflow 3.x, so you can >>>> upgrade to 3.x and then gradually migrate your DAGs to 3.x syntax. >>>> >>>> Which one do you have in mind? I think 2) would ease the upgrade path to >>>> Airflow 3.x and encourage users to try it out, while 1) would have the >>>> effect of keeping users at Airflow 2.x, as they cannot upgrade before >>>> modifying their DAGs. >>>> >>>> On Mon, Aug 5, 2024 at 9:15 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>>> >>>>> Also I have a feeling that with AST / smart conversion, we should be >>> able >>>>> to get an almost 100% accurate automated conversion of that one. It is >>>>> "almost" (of-course there will be edge cases), but it looks like a >>> pretty >>>>> automate'able conversion. >>>>> >>>>> Maybe - I am not sure how possible it is due to contracts etc. - but >>> maybe >>>>> also the managed service teams could run such conversion on the vast >>> number >>>>> of DAGs people have out there and "perfect" the conversion mechanism. >>>>> >>>>> I think - again - what's really crucial here is that it does not have >>> to be >>>>> a big-bang. If we will have prolonged period of time, where compatible >>>>> approach will happily coexist with new approach - for Airflow 2.11 and >>>>> Airflow 3 and when you will still be able to run part of your DAGS with >>>>> "old ways" and gradually converting your DAGs to "new ways", ability to >>>>> track the progress and "nagging warnings" - and you have a converter >>> that >>>>> works "out-of-the-box" for "almost" everything - this is the maximum >>>>> support we can give to our users. >>>>> >>>>> J. >>>>> >>>>> >>>>> On Mon, Aug 5, 2024 at 8:52 AM Tzu-ping Chung <t...@astronomer.io.invalid >>>> >>>>> wrote: >>>>> >>>>>> I don’t know how to compare the efforts since honestly I have very >>> little >>>>>> idea how many templated fields people tend to have. I can outline what >>>>> each >>>>>> group needs to do though. >>>>>> >>>>>> Airflow maintainers >>>>>> - Develop the compatibility layer >>>>>> - One time work before 3.0 >>>>>> - Maintenance afterwards, at least until we drop 2.x support from >>> all >>>>>> providers >>>>>> - Apply compatibility layer to all providers >>>>>> - Drop compatibility layer from providers when they drop 2.x support >>>>>> - Not a hard requirement >>>>>> >>>>>> Third-party operator authors >>>>>> - Apply compatibility layer to all operators >>>>>> - Drop compatibility layer from operators when dropping 2.x support >>>>>> - Not a hard requirement >>>>>> >>>>>> DAG authors (that want to upgrade to Airflow 3) >>>>>> - Same as third-party operator authors if the real authors don’t want >>> to >>>>>> upgrade >>>>>> - Convert DAGs to use the new syntax >>>>>> >>>>>> >>>>>>> On 30 Jul 2024, at 04:55, Scheffler Jens (XC-AS/EAE-ADA-T) < >>>>>> jens.scheff...@de.bosch.com.INVALID> wrote: >>>>>>> >>>>>>> Thanks TP for the rework. >>>>>>> >>>>>>> I added some comments (iteration 2) on the migration ideas. I think >>>>>> these details make it clearer, still i have partial doubts how much >>>>> burden >>>>>> we add to users to migrate DAGs to get to version 3. I very much favor >>>>> the >>>>>> new templating but am not sure how many DAG authors we leave with >>>>> migration >>>>>> problems behind. >>>>>>> Do we have a guess or estimation how much burden we as airflow >>>>>> developers need to keep as compatability compared to the amount of DAG >>>>>> templates that people neet to adjust? >>>>>>> >>>>>>> Sent from Outlook for iOS<https://aka.ms/o0ukef> >>>>>>> ________________________________ >>>>>>> From: Tzu-ping Chung <t...@astronomer.io.INVALID> >>>>>>> Sent: Monday, July 29, 2024 8:54:58 PM >>>>>>> To: dev@airflow.apache.org <dev@airflow.apache.org> >>>>>>> Cc: michalmod...@google.com <michalmod...@google.com> >>>>>>> Subject: Re: [VOTE] AIP-80: Explicit Template Fields in Operator >>>>>> Arguments >>>>>>> >>>>>>> I have updated the AIP to include the additional compatibility >>>>>> discussions in this thread. Please take a look again. >>>>>>> >>>>>>> Specifically (although by no means exclusively) it would be awesome >>> if >>>>>> Michał you could have a look and see if it addresses more of the >>> concerns >>>>>> and could be viable for you. Although the vote is non-binding, I still >>>>>> would like to be more confident I tried to address the real concerns >>> from >>>>>> the community, which is the real problem when it comes to migration. >>>>>>> >>>>>>> Link to document for convenience >>>>>> >>>>> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fx%2F2grOEg&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9318251bd2fe4490b64308dcafffffe5%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638578761240102546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=vVICot9p06VQJ56QmP9dwvEaWfpR7nCcMVqUexl6B3w%3D&reserved=0 >>>>>> <https://cwiki.apache.org/confluence/x/2grOEg> >>>>>>> >>>>>>> TP >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org >>>>>> >>>>>> >>>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> For additional commands, e-mail: dev-h...@airflow.apache.org >>> >>> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
