Yep. It could work with symbolic links. Tested it and with flit - both wheel and sdist packaged code such symbolically linked file is dereferenced and copy of the file is added there. It could be a nice way of doing it.
Maybe then worth trying next time if someone has a need? J On Thu, Feb 22, 2024 at 12:39 AM Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com.invalid> wrote: > >>> As of additional dependency complexity between providers actually the > additional dependency I think creates more problems than the benefit… > would be cool if there would be an option to „inline“ common code from a > single place but keep individual providers fully independent… > > >Well, we already do a lot of inlining, so if we think we should do more, > we have mechanisms for that. We have pre-commits and release commands that > do a lot of that. Pre commits are inlining scripts in Dockerfiles, > shortening PyPI readme . The providers __init__.py files and changelogs and > index documentation .rst (partially) are generated at release documentation > preparation time, pyproject.toml for providers are generated from common > templates at package building time and so on and so on :). So we can do > more of that and generate common code, it's just a matter of adding > pre-commits or breeze scripts. But again "can't have and eat cake" - this > has the drawback that there are extra steps involved and even if it's > automated it does add friction when you have to regenerate the code every > time you change it and when you change it in another place than where you > use it. > > Yes, also thought a moment about pre-commit. I#d be okay if we really > in-line and have a pre-commit aligning the redundancy of python in folders. > Might need to be an opt-in if only 10 of 85 providers are using common > stuff and if we change a common line we probably do not need to affect all > providers. > > As long as no Windows users trying to code for airflow (do we need to > consider?) would it also work to use symlinks? Git can cope with this, I > don't know if the python toolchain can de-reference a copy and are not > packaging a symlink? Would be worth a test... would save the pre-commit and > we even could selectively include common bla into providers as needed :-D > > Mit freundlichen Grüßen / Best regards > > Jens Scheffler > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > GERMANY | www.bosch.com > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > jens.scheff...@de.bosch.com > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > Forschner, > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > > -----Original Message----- > From: Jarek Potiuk <ja...@potiuk.com> > Sent: Mittwoch, 21. Februar 2024 21:18 > To: dev@airflow.apache.org > Subject: Re: [DISCUSS] Common.util provider? > > > if we have a common piece then we are locking all depending providers > (potentially) together if common code changes > > Yes, coupling in this case is the drawback of this solution. You can't > have cake and eat it too and in this case you trade DRY with coupling. > > > As of additional dependency complexity between providers actually the > additional dependency I think creates more problems than the benefit… > would be cool if there would be an option to „inline“ common code from a > single place but keep individual providers fully independent… > > Well, we already do a lot of inlining, so if we think we should do more, > we have mechanisms for that. We have pre-commits and release commands that > do a lot of that. Pre commits are inlining scripts in Dockerfiles, > shortening PyPI readme . The providers __init__.py files and changelogs and > index documentation .rst (partially) are generated at release documentation > preparation time, pyproject.toml for providers are generated from common > templates at package building time and so on and so on :). So we can do > more of that and generate common code, it's just a matter of adding > pre-commits or breeze scripts. But again "can't have and eat cake" - this > has the drawback that there are extra steps involved and even if it's > automated it does add friction when you have to regenerate the code every > time you change it and when you change it in another place than where you > use it. > > J. > > On Wed, Feb 21, 2024 at 9:02 PM Scheffler Jens (XC-AS/EAE-ADA-T) < > jens.scheff...@de.bosch.com.invalid> wrote: > > > Hi Jarek, > > > > At reviewing the PR from uranusjr for AIP-60 I also had the feeling > > that a lot of very similar code is repeated in all the providers. But > > during review yesterday I dropped the ides because if we have a common > > piece then we are locking all depending providers (potentially) > > together if common code changes. > > As of additional dependency complexity between providers actually the > > additional dependency I think creates more prblems than the benefit… > > would be cool if tehere would be an option to „inline“ common code > > from a single place but keep individual providers fully independent… > > > > Jens > > > > Sent from Outlook for > > iOS<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > aka.ms%2Fo0ukef&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C98c8897 > > 195d944d483ab08dc331a49bb%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7 > > C638441435197193656%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj > > oiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=n6gk9fNnWB > > SJOPYEgJ9WbriZ3H4id3RhLr16SguOuFA%3D&reserved=0> > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Wednesday, February 21, 2024 5:42:20 PM > > To: dev@airflow.apache.org <dev@airflow.apache.org> > > Subject: [DISCUSS] Common.util provider? > > > > Hello everyone, > > > > How do we feel about introducing a common.util provider? > > > > I know it's not been the original idea behind providers, but - after > > testing common.sql and now also having common.io, seems like more and > > more we would like to extract some common code that we would like > > providers to use, but we refrain from it, because it will only be > > actually usable 6 months after we introduce some common code. > > > > However, if we introduce common.util, this problem is generally gone - > > at the expense of more complex maintenance and cross-provider > dependencies. > > We should be able to add a common util method and use it in a provider > > at the same time. > > > > Think Amazon provider using a new feature released in common.util > > >=1.2.0 and google provider >= 1.1.0. All manageable and we do it > > already for common.sql. We know how to do it, we know what to avoid, > > we know we cannot introduce backwards-incompatible changes, so we have > > to be very clear what is and what is not a public API there, We could > > rather easily add tests to prevent such backwards-incompatible > > changes. We even have a solution for chicken-egg providers where we > > need to release two providers at the same time if they depend on each > > other. Generally speaking it's quite workable but adds a bit of overhead. > > > > Examples that we could implement as "common.util": > > > > - common versioning check with cache - where multiple providers could > > reuse "do we have pendulum 2" > > - more complex - some date management features (we have a few like > > date_ranges/round_time). But there are many more. > > > > I generally do not love the common "util" approach. It has a tendency > > to become a bag of everything over time. but if we limit it to a set > > of small, fully decoupled modules where each module is independent - > > it's OK. And we already have it in "airflow.util" and we seem to be > doing well. > > > > WDYT? Is it worth it ? > > > > J. > > >
