The Astral folks also seem very focused on it being a drop-in/compliant replacement for pip. So I think it's definitely worth dropping it in and seeing if we get the expected performance improvements. If tests still pass and user facing constraints and install instructions remain unchanged I don't see why not, if someone is willing to spend the time on it. Never mind the extra features it would give us (I, like others, am also very excited about --resolution=lowest, ability).
________________________________ From: Andrey Anshin <andrey.ans...@taragol.is> Sent: Tuesday, February 20, 2024 12:26:56 AM To: dev@airflow.apache.org Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Considering trying out uv for our CI workflows CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. > I share Andrey's skepticism. It's just yet another tool which has an unclear development strategy. My point was more about a matter of presentation. If someone told you "this is a new tool, like a killer of previous tools" then you might think "Yeah...yeah...yeah.. yet another replacement to tool X... not really interesting". On the other hand if someone told you what in cases you might solve, then this might be a mind changer. Especially the promising `--resolution=lowest` option. We always want to test something with minimal dependencies because we are not sure that it might work with pretty old dependencies, and recently I've started to work on POC to collect minimal versions of the Airflow and Providers. And at the moment when I almost finished it the uv was released. Well sometimes it is better to wait a bit and maybe someone would invent the same solution 😁 and you don't have to spend a personal time. So as POC I'm on it, we still need a `pip` and validate some stuff by a pip because it is only one officially supported way to install Airflow but if something could be improved in the CI then I'm on it, in most cases it would be behind of Breeze and many of the contributors might be even not noticed that something changed. On Tue, 20 Feb 2024 at 09:56, Jarek Potiuk <ja...@potiuk.com> wrote: > Actually - of you read that blog post, the strategy is clear - they aim to > create a comprehensive packaging tooling and improvnts are measured (80-100 > times they claim - I using caching - they (unlike pip) use a lot of local > caching including resolving dependencies). > > So I think both arguments are not valid if you ask me. > > wt., 20 lut 2024, 02:37 użytkownik Alexander Shorin <kxe...@apache.org> > napisał: > > > I share Andrey's skepticism. It's just yet another tool which has an > > unclear development strategy. Should you make it a free testing suite? > What > > project would receive in exchange? A lot of words about being faster, but > > how much? Are these milliseconds worth to change the stable tool with a > new > > one? And will it notably improve something? > > > > I think it's worth to try it just for fun and provide feedback, but it'll > > have to pass a long road to become such stable as pip. > > > > -- > > ,,,^..^,,, > > > > > > On Tue, Feb 20, 2024 at 3:06 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > My opinion: > > > > > > I think there is a place for a number of such tools. For a long time > the > > > packaging team and `pip` team have been working not only on `pip` > > > implementation but also (and most importantly) to make sure that what > > `pip` > > > does is to be the beacon of standardisation of packaging APIs and PEPs. > > It > > > will never IMHO have a lot of the fancy features that other tools might > > > provide (like the ones I mentioned). It will always be there to provide > > the > > > robust and solid CLI to run all packaging things, but there are plenty > of > > > opportunities to provide improved or modified, or more (or less) > > > opinionated ways of doing things that are addressing some cases that > > `pip` > > > team simply will not be able or willing to handle, preferring "pure" > > > standard approach vs. implement all the optional things. For example > the > > > way how pre-releases are handled can be improved to be more selective. > > The > > > PEP describing it gives the tools an option to add more fancy > behaviours > > > (some of which we could find useful in our CI tooling). Should `pip` > > > implement those - I don't think so. It would distract maintainers from > > > other more important things. It is quite ok to use other tooling in > > places > > > like our CI, where they do some parts of the installation better. > > > > > > For me `pip` is going more into the direction of `usable reference > > > implementation of package installed` - any standard/ PEP will not > matter > > if > > > `pip` does not implement it. But others might go in different > directions > > > and implement some less popular features and do it better, faster, with > > > greater flexibility. IMHO it's a win-win. > > > > > > J. > > > > > > > > > On Mon, Feb 19, 2024 at 11:40 PM Andrey Anshin < > andrey.ans...@taragol.is > > > > > > wrote: > > > > > > > Yesterday my friend shared with me that tool and I've been told that > > more > > > > presumably it would be a niche tool. I've been told "who needs yet > > > another > > > > installer which stands to resolve all your problems' '. > > > > I guess I was wrong? > > > > > > > > On Tue, 20 Feb 2024 at 00:53, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > > > Hey everyone, > > > > > > > > > > Few days ago the ruff creators have released a new tool uv - which > is > > > an > > > > > extremely fast (written in rust) and fully featured tool generally > > > fully > > > > > compatible with `pip`. > > > > > > > > > > Blog post here: https://astral.sh/blog/uv > > > > > > > > > > It looks like It has a number of things that would make our CI > cases > > > and > > > > > tooling quite a bit faster and better including a few things that I > > > have > > > > > implemented some workarounds for and some that I have not > > > > > implemented because `pip` had no good solution. > > > > > > > > > > I looked at the docs and it solves some problems that are currently > > > > > difficult or impossible to handle with `pip`: > > > > > > > > > > * ability to use overrides (which are constraints on steroids - > > > allowing > > > > to > > > > > override limits specified by the packages - this will be very > useful > > to > > > > > better handle our cases with "chicken-egg" providers (for example > > like > > > we > > > > > had in FAB) where we have pre-release packages depending on each > > other > > > > > > > > > > * different resolution strategies including --resolution=lowest > which > > > > will > > > > > finally allow us to see whether airflow's lower bounds are still > > > holding > > > > > (i.e. - will our test still pass if we use the lowest supported > > version > > > > of > > > > > our dependencies? this is something i wanted to do for quite some > > time > > > > and > > > > > recorded an issue for that - > > > > > https://github.com/apache/airflow/issues/35549 > > > > > but lack of tooling support made it a wish, with > > `--resolution=lowest` > > > it > > > > > seems like super-easy thing to do. > > > > > > > > > > * It is said to be many, many times faster - with better caching > and > > > > > resolution speeds (similarly like with ruff they claim orders of > > > > magnitude > > > > > speedups in a number of cases). We can likely make very good use of > > it > > > > and > > > > > speed up some parts of our CI workflow significantly. > > > > > > > > > > I might likely do some experimenting with uv in our toolchain, but > > > wanted > > > > > to make sure we are all aware of it - and ask if someone has > > something > > > > > against it (and maybe someone would like to do some work there > trying > > > it > > > > > out - I will be happy to guide others with the dev/tooling mindset > > and > > > > > incline to do some changes there/review PRs and cooperate on > testing > > > > those > > > > > things. > > > > > > > > > > It's not a user-facing change, and I do not think we want to get > rid > > of > > > > > `pip` as an installation tool in general (in our images and user > > facing > > > > > side) - it's mostly an internal CI tooling improvement I am > thinking > > > of. > > > > > Maybe at some point in time we can recommend it also for > development > > > > > workflows, and maybe someday it will gain enough popularity to > think > > > > about > > > > > recommending it to our users, but definitely not now nor in even > > > mid-term > > > > > future. > > > > > > > > > > Let me know what you think. > > > > > > > > > > Repo here: https://github.com/astral-sh/uv > > > > > > > > > > J. > > > > > > > > > > > > > > >
