> Btw, I've been testing how well the resolver works for uv based on many real world examples I have built up over the years improving the pip resolver, it still can't resolve apache-airflow[all]: https://github.com/astral-sh/uv/issues/1560. I have recommended heuristics that uv could implement which I think would help here.
Strange with [all] - because with we are now using uv nicely to install `[devel-ci]` which is way bigger than all (though all has one more level of indirection via released providers (all has dependencies to providers and then only those - various versions of providers have other dependencies, where devel-ci is has all the dependencies of latest dependencies which is quite a bit simpler resolution to happen). Curious to see the progress on it. > But one interesting idea that airflow could look at is periodically running test cases against either "--resolution=lowest" or "--resolution=lowest-direct" to try and flush out bad old dependencies? Look at the beginning of the discussion :) . This is definitely one of the features that we are going to use (and soon). And we have an issue already planned to do so: https://github.com/apache/airflow/issues/35549 The (little) difficulty here is that we won't be able to weed all the weeds out this way - because with 90 optional providers/ extras we have a lot of overlapping dependencies so just "lowest" does not mean that our lower-bound criteria for particular provider are "good enough" - because the same transitive dependency from one provider might be limited by another - so in order to **really** find out the right lower bounds we will literally have to get <lowest> for a single provider and run tests for it for that provider. So this is going to be a little harder than just running --lowest on everything. That might be a good start but is not nearly enough. But - we have all the right tooling with breeze, test parallelism that we can employ for that. J. On Mon, Feb 26, 2024 at 6:22 PM Damian Shaw <ds...@striketechnologies.com> wrote: > Btw, I've been testing how well the resolver works for uv based on many > real world examples I have built up over the years improving the pip > resolver, it still can't resolve apache-airflow[all]: > https://github.com/astral-sh/uv/issues/1560. I have recommended > heuristics that uv could implement which I think would help here. > > But one interesting idea that airflow could look at is periodically > running test cases against either "--resolution=lowest" or > "--resolution=lowest-direct" to try and flush out bad old dependencies? > > Damian > > -----Original Message----- > From: Jarek Potiuk <ja...@potiuk.com> > Sent: Monday, February 26, 2024 7:45 AM > To: dev@airflow.apache.org > Subject: Re: [DISCUSS] Considering trying out uv for our CI workflows > > And merged. I will keep an eye on it for the next few days. > > On Mon, Feb 26, 2024 at 11:47 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Yes. The difference was because of caching. I forgot to mention that. > > This is due to the way our CI image docker optimisation works > > > > The way how the image is constructed is that it first installs > > dependencies from https://github.com/apache/airflow to save on any > > future re-installs - using docker caching mechanism (it works the same > > for `pip` and `uv pip`. > > > > It works roughly like this: > > > > LAYER 1: pip install "apache-airflow[devel-ci] @ > > https://github.com/apache/airflow/archive/main.tar.gz <- here we > > install all airflow dependencies in the main AT THE MOMENT THIS LAYER > > WAS BUILT LAYER 2: COPY pyproject.toml LAYER 3: pip install > > ".[devel-ci]" --constraints https://..... > > > > It is a very nice way of caching and speeding up adding new > > dependencies I introduced years ago that works very nicely for us for > > remote builds (so local breeze builds are making use of it as well) - > > this means that ALL dependencies of airflow will be pre-installed as a > > cached layer in the image, regardless of modification in > > pyproject.toml. So whenever someone modifies pyproject.toml and adds > > new dependencies or modifies the existing ones - LAYER 1 will NOT be > > invalidated, but LAYER 2 (and LAYER 3) will - which means that the > > LAYER 3 pip install will only install new dependencies (and it will use > latest constraints for that). > > > > LAYER 1 is only modified when: > > > > a) Python base image changes (every few weeks) > > b) Docker scripts change (those scripts that are COPIED before that > > layer > > - so for example airlfow installation scripts). > > c) DEPENDENCY_EPOCH change -> we can manually bump it to force the > > reinstallation after we remove some dependency, to make sure it is > > regenerated > > > > This has the side effect that when you add or modify a dependency, it > > is very fast - instead of reinstalling all 600+ dependencies, they are > > already installed and you only get dif Another side effect of it that > > the image (between Python base image updates or epoch/docker script > > changes) is that the image gets ever so bigger - every time new > > constraints are update and cache rebuilt and constraints updated, the > > image gets updated with new dependencies in main, incrementally adding > > changed (and only those changed ones) dependencies in LAYER 3. So I > > was comparing an image where LAYER 1 was created some time ago with > > pip (and LAYER 3 got bigger) with pretty much "From the scratch" > > image where LAYER 1 was "latest deps" and LAYER 3 has almost no > > updated/ new dependencies. > > > > That explains the difference. The new image will also get slightly > > bigger in the next few days or weeks, until a new Python base image is > > released or we will update the scripts. > > > > Also - all tests pass and that's most important. The CI image is > > exclusively used to run tests, it's not used in production. The > > production image is still using `pip` (I had some problems with PROD > > image building with uv - because it expects virtualenv. rather than > > --user installation of > > ours) . We might want to fix it some time in the future (and uv might > > add it as a feature in the meantime) - let's give `uv` some time to > settle :). > > > > So - it has no impact on the user-facing side (at all). > > > > Re: dependencies after `uv` was successful have been updated here: > > https://github.com/apache/airflow/commit/fd64235a481adb4aaff1b2f432eac > > eb9d0b5c53c > > in our constraints 2 hrs ago. > > > > As you can see - the changes are "as expected" - there are a few > > dependencies bumped since yesterday (correctly picked up by uv > > --highest resolution mechanism). The `uv pip freeze` command of uv for > > now uses original, non-canonical names of packages - but the original > > now (underscores instead of dashes) - but that's perfectly fine, those > > packages get canonical names. This will likely get changed in the future. > > > > J. > > > > > > > > > > > > > > On Mon, Feb 26, 2024 at 10:18 AM Scheffler Jens (XC-AS/EAE-ADA-T) > > <jens.scheff...@de.bosch.com.invalid> wrote: > > > >> @Jarek, had no time to review PR. > >> If the Docker image is ~400MB smaller, I fear there is a diff. Were > >> you able to dump a file list to inspect the diff? > >> If not I would propose to make it in the PR to understand "why". If > >> there care cache files (only) then in general it would make sense to > >> think about if "cache/garbage" is anyway left in pip/uv which we > >> should clean to shrink images. > >> > >> Mit freundlichen Grüßen / Best regards > >> > >> Jens Scheffler > >> > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch GmbH | > >> Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | > >> http://www.bosch.com/ Tel. +49 711 811-91508 | Mobil +49 160 90417410 > >> | jens.scheff...@de.bosch.com > >> > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. > >> Markus Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, > >> Dr. Tanja Rückert > >> > >> -----Original Message----- > >> From: Jarek Potiuk <ja...@potiuk.com> > >> Sent: Montag, 26. Februar 2024 08:54 > >> To: Amogh Desai <amoghdesai....@gmail.com> > >> Cc: dev@airflow.apache.org > >> Subject: Re: [DISCUSS] Considering trying out uv for our CI workflows > >> > >> Yep. It all looks good now and I re-ran last intermittently failing job: > >> Final effect of it: > >> > >> * CI image (uncompressed) with uv is slightly smaller (3.5 GB vs. 3.9 > >> GB) > >> * regular code only PRs: same time to incrementally build image ~ 1m > >> * adding/modifying dependency in the PR:: 12 m -> 6m : 50% > >> improvement > >> * removing dependency/rebuilding things from scratch -> 27m -> 12 m : > >> 55% improvement > >> > >> Depending on the speed of your network, also locally rebuilding your > >> image should be generally much faster in all cases once we merge it > >> and update cache. > >> > >> Also the flaky test turned out to be really just "sometimes running > >> much slower than expected" case - I increased the number of retries > >> and gave the test a bit more time and added better message, so > >> hopefully the flaky test will stop happening now. > >> > >> I think it's a no-brainer :). > >> https://github.com/apache/airflow/pull/37692 waiting for reviews > >> > >> J. > >> > >> > >> > >> On Mon, Feb 26, 2024 at 4:50 AM Amogh Desai > >> <amoghdesai....@gmail.com> > >> wrote: > >> > >> > Thanks for the superb investigation and effort @Jarek Potiuk > >> > <ja...@potiuk.com>! > >> > > >> > I quite like the performance improvement numbers uv brings in > >> > compared to pip. > >> > I see no reason not to switch to UV in prod images as well. > >> > > >> > I will take a look at the pull request soon. > >> > > >> > Thanks & Regards, > >> > Amogh Desai > >> > > >> > On Mon, Feb 26, 2024 at 5:29 AM Jarek Potiuk <ja...@potiuk.com> > wrote: > >> > > >> >> I think I will get it green finally: > >> >> https://github.com/apache/airflow/pull/37692. > >> >> > >> >> I know where the test flakiness was from. Generally speaking it > >> >> turned out that there is no free lunch and - of course - cache > >> >> from uv increased our CI image size significantly (by around 1.5G) > >> >> - and it caused much slower test execution (and test became more > >> >> flaky because of that). So after looking at that I decided to > >> >> disable the cache - it's definitely not worth it to increase the > >> >> size of our images that much. We still have significant (50% - 60% > >> >> improvements - not the 60% - 70% like we had with cache), but it's > >> >> still significant enough. Without cache the "upgrade scenario is ~ > >> >> 40s (so no 4s any > >> >> more) instead of 7m with pip - so this is still a huge improvement > >> >> (image size is even smaller than the one with `pip`). > >> >> > >> >> > >> >> J, > >> >> > >> >> > >> >> > >> >> On Sun, Feb 25, 2024 at 9:17 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > >> >> > >> >> > Some more findings. > >> >> > > >> >> > Overall, I can confirm that with `uv` we will get significant - > >> >> > 60 > >> >> > - 70% on build image times. This will impact both CI but also > >> >> > `breeze` local rebuilds. > >> >> > > >> >> > I am getting closer to a mergeable state. I switched to > >> >> > https://g/ ithub.com > >> %2Fapache%2Fairflow%2Fpull%2F37692&data=05%7C02%7CJens.Scheffler% > >> 40de.bosch.com%7Cda57d392cf6a4799ce6608dc36a01ff7%7C0ae51e1907c84e4bb > >> b6d648ee58410f4%7C0%7C0%7C638445308555397453%7CUnknown%7CTWFpbGZsb3d8 > >> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C > >> 0%7C%7C%7C&sdata=QD6nJwgUu5Hncvb3Vu%2F0WCbL2iYXzxu3z8N7xYhcjAg%3D&res > >> erved=0 to test "upgrade to latest dependencies" workflow and canary > >> build impact. > >> >> > > >> >> > The PR is getting greener and greener. I have a few last things > >> >> > to address. > >> >> > > >> >> > An interesting story is that a flaky test in CLI > >> >> > > >> >> (tests/cli/commands/test_webserver_command.py::TestCliWebServer::t > >> >> est > >> >> _cli_webserver_background) > >> >> > we had is suddenly significantly more flaky, so I will have to > >> >> > take a > >> >> look > >> >> > at how to finally remove the flakiness from it. > >> >> > This is a good thing because this test had been flaky for quite > >> >> > a while but it was very difficult to reproduce and seems that > >> >> > for some reason > >> >> it is > >> >> > now much easier to reproduce (which also means we will know when > >> >> > we fix > >> >> it0. > >> >> > > >> >> > Looking at stats it seems that a lot (but not all) of the speed > >> >> > improvement might come with Parallel downloading of dependencies > >> >> > - which are in the works also for pip ( https://g/ > >> >> > ithub.com%2Fpypa%2Fpip%2Fpull%2F12388&data=05%7C02%7CJens.Scheff > >> >> > ler > >> >> > %40de.bosch.com%7Cda57d392cf6a4799ce6608dc36a01ff7%7C0ae51e1907c > >> >> > 84e > >> >> > 4bbb6d648ee58410f4%7C0%7C0%7C638445308555401666%7CUnknown%7CTWFp > >> >> > bGZ > >> >> > sb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > >> >> > 6Mn > >> >> > 0%3D%7C0%7C%7C%7C&sdata=Ycl1VKKK3Rb6iMVLq4kX3OToJXe119GlfUBE8DXK > >> >> > 9dc > >> >> > %3D&reserved=0) - though it's not clear how > >> >> much > >> >> > it will help as the Batch Dowloader in pip is involved only > >> >> > after resolution. We will see after it is implemented if it > changes things. > >> >> > > >> >> > I am also now switching PROD builds to use uv to see how much we > >> >> > can > >> >> save, > >> >> > but I leave `pip` as default for releases and users, the only > >> >> difference is > >> >> > CI - I've added separate step for `pip` PROD build to compare > >> >> > and to > >> >> make > >> >> > sure it's running fine in CI. > >> >> > > >> >> > The numbers: > >> >> > > >> >> > * for "upgrade to newer dependencies" scenario - uv is WAY > >> >> > faster - as I thought. In the "current" stage of the main it is: > >> >> > ~7m pip, 5 s > >> (!) uv. > >> >> > Here caching of uv makes a huge difference, and while there is > >> >> > some > >> >> work in > >> >> > `pip` and resolvelib (looking at PRs/issues) it's going to be > >> >> > quite some time to get similar results from pip and "upgrade" > >> >> > builds will go down eventually from 12m to 5 m - which is a > >> >> > major improvement - especially > >> >> for > >> >> > elapsed time of CI builds. > >> >> > > >> >> > * from what I see package installation is super-fast in uv. > >> >> > Installing > >> >> 614 > >> >> > packages takes (wait for it) 1s (!) where I saw it taking way > >> >> > over a > >> >> minute > >> >> > with `pip`. This will be hard to beat I think with Python vs. rust. > >> >> > > >> >> > Some notes about differences I saw: > >> >> > > >> >> > PIP and UV lead to slightly different resolutions when upgrading. > >> >> > This > >> >> is > >> >> > not a surprise because different heuristics are involved (the > >> >> > resolution algorithm is np-complete > >> >> > (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F > >> >> > %2F > >> >> > research.swtch.com%2Fversion-sat&data=05%7C02%7CJens.Scheffler%4 > >> >> > 0de > >> >> > .bosch.com%7Cda57d392cf6a4799ce6608dc36a01ff7%7C0ae51e1907c84e4b > >> >> > bb6 > >> >> > d648ee58410f4%7C0%7C0%7C638445308555405774%7CUnknown%7CTWFpbGZsb > >> >> > 3d8 > >> >> > eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% > >> >> > 3D% > >> >> > 7C0%7C%7C%7C&sdata=2yp6zlzYsFMa2Qfua6T62ADSn5q2A8hUNSVSVd3VC8Q%3 > >> >> > D&r > >> >> > eserved=0) and it's very inefficient to run the full > >> >> > resolution, so both pip and uv > >> >> take a > >> >> > little different approach for shortcuts and limiting the > >> >> > possible space > >> >> of > >> >> > solutions. I've done a few PRs limiting (lower-bound) some > >> >> > dependencies > >> >> to > >> >> > bring them closer) - but at the end what we get is "correct" in > >> >> > both > >> >> cases > >> >> > - I continue running `pip check` to make sure that whatever UV > >> >> > finds is also correct according to `pip`. Nothing really major > >> >> > there. There were literally few cases that required some manual > >> >> > adjustments. Nothing unmanageable also in the future, I was > >> >> > doing similar tweaks with `pip` > >> >> as > >> >> > well to help with the resolution. > >> >> > > >> >> > Example of differences (left. first is pip, right, second is uv) > >> >> > > >> >> > < importlib-resources==5.13.0 > >> >> > --- > >> >> > > importlib-resources==6.1.1 > >> >> > > >> >> > vs. > >> >> > > >> >> > < pycountry==23.12.11 > >> >> > --- > >> >> > > pycountry==22.3.5 > >> >> > > >> >> > It means that with `uv` we have a newer version of > >> >> > importlib_resources > >> >> but > >> >> > an older version of pycountry. > >> >> > > >> >> > This one I will handle by bumping pycountry in case of facebook > >> >> > provider and bump it to > 23.12 as the old version is 1.5 years > old. > >> >> > > >> >> > J. > >> >> > > >> >> > > >> >> > On Sun, Feb 25, 2024 at 12:52 AM Hussein Awala > >> >> > <huss...@awala.fr> > >> >> wrote: > >> >> > > >> >> >> That's impressive! I love this tool, not only for reducing CI > >> >> >> time but also for saving the environment. > >> >> >> Some of the previous improvements were to further parallelize > >> >> >> CI jobs > >> >> to > >> >> >> complete the CI faster, but this tool will help reduce the > >> >> >> overall > >> >> time. > >> >> >> > >> >> >> Big +1 > >> >> >> > >> >> >> On Sun, Feb 25, 2024 at 12:34 AM Jarek Potiuk > >> >> >> <ja...@potiuk.com> > >> >> wrote: > >> >> >> > >> >> >> > Hello here. > >> >> >> > > >> >> >> > I have a PR > >> >> >> > > >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%25 > >> >> >> > 2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F37683&data=05%7C02%7 > >> >> >> > CJe > >> >> >> > ns.Scheffler%40de.bosch.com%7Cda57d392cf6a4799ce6608dc36a01ff > >> >> >> > 7%7 > >> >> >> > C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C63844530855541012 > >> >> >> > 5%7 > >> >> >> > CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC > >> >> >> > JBT > >> >> >> > iI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=I8DF0ugyN53LKTOZ > >> >> >> > y8N > >> >> >> > dhKS%2FUKGdZuI9SoOVIwgx9MI%3D&reserved=0 that > >> >> >> implements: > >> >> >> > > >> >> >> > * ability to choose either uv or PIP when building our images > >> >> >> > * CI images are built with uv by default (but you can use > >> >> `--no-use-uv` > >> >> >> as > >> >> >> > a flag and switch back to `pip` > >> >> >> > * PROD images are built with pip by default (but you can us > >> >> `--use-uv` > >> >> >> as a > >> >> >> > flag an switch to uv > >> >> >> > > >> >> >> > The preliminary tests show indeed that uv not only has a much > >> >> >> > faster baseline, but also their use of caching fits > >> >> >> > extremely well into our strategy of building images and we > >> >> >> > will get huge improvements of our > >> >> CI > >> >> >> > build timing when using uv. > >> >> >> > > >> >> >> > Just for the context - our CI images when built are using a > >> >> >> > caching strategy to optimise for f > >> >> >> > > >> >> >> > 1) fast building when there are no changes (around 1 minute > >> >> >> > to build > >> >> >> with > >> >> >> > pip), > >> >> >> > 2) slower building when someone adds or modifies > >> >> >> > non-conflicting > >> >> >> dependency > >> >> >> > (around. 8 minutes to build, out of which ~ 6 m is pip > >> >> >> > resolution and > >> >> >> > installation) > >> >> >> > 3) much longer build time when there are conflicting > >> >> >> > dependencies or > >> >> >> when > >> >> >> > we change Dockerfile or scripts or when Python base image > >> >> >> > changes > >> >> >> (around > >> >> >> > 27 minutes build out of which pip resolving is ~ 20m). > >> >> >> > > >> >> >> > Those are all `pip` numbers. Currently `pip` does not use > >> >> >> > resolution caching between the steps. Comparison of some > >> >> >> > basic installation > >> >> steps > >> >> >> from > >> >> >> > initial tests show that UV is way faster: > >> >> >> > > >> >> >> > * Resolving and Installing airflow with [devel-ci] (610 > >> >> dependencies): > >> >> >> pip > >> >> >> > ~ 6m, uv ~ 1m 30 s > >> >> >> > * Re-resolving and reinstalling [devel-ci] using local > >> >> pyproject.toml; > >> >> >> pip > >> >> >> > ~ 4m (cache is not used), uv ~ 4s (!!!!) - because cache is > >> >> >> > used in > >> >> this > >> >> >> > case. > >> >> >> > > >> >> >> > I have not yet tested well (but I will once they happen) > >> >> >> > --eager > >> >> >> upgrade of > >> >> >> > dependencies (pip - very much depends but it's often in the > >> >> >> > range of > >> >> 10 > >> >> >> > minutes) - I expect it not to take more than 2-3 minutes with > >> >> >> > uv > >> >> >> > > >> >> >> > So overall it looks like we are looking at those improvements: > >> >> >> > > >> >> >> > 1) Regular builds with no dependency changes: pip.~ 1m , uv ~ > >> >> >> > 1m (because we are using docker layer caching and pip > >> >> >> > resolution and installation is not used at all) > >> >> >> > 2) Updating dependencies: 8m with pip will probably go down > >> >> >> > with uv > >> >> to ~ > >> >> >> > 3.30s => 60% improvement and in many cases ~ 2.5 m when there > >> >> >> > are no > >> >> >> remote > >> >> >> > changes and cache is used (70% improvement) > >> >> >> > 3) Re-resolving and reinstalling everything 27 m will > >> >> >> > probably go > >> >> down > >> >> >> with > >> >> >> > uv to ~ 9m => 67% improvements. > >> >> >> > > >> >> >> > If those numbers hold and the resolution quality will be > >> >> >> > comparable > >> >> to > >> >> >> > `pip` - then well, it's definitely worth it - and the numbers > >> >> >> > are > >> >> very > >> >> >> > close to what the `uv` authors claimed. > >> >> >> > > >> >> >> > I am impressed :) > >> >> >> > > >> >> >> > J. > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > On Thu, Feb 22, 2024 at 5:25 AM Amogh Desai < > >> >> amoghdesai....@gmail.com> > >> >> >> > wrote: > >> >> >> > > >> >> >> > > I agree with Niko here. > >> >> >> > > > >> >> >> > > If someone is willing to give it a try, we should enable it > >> >> >> > experimentally > >> >> >> > > and give it a stint for a couple of weeks. If we see > >> >> >> > > significant > >> >> >> results, > >> >> >> > > we can adopt it. > >> >> >> > > > >> >> >> > > Thanks & Regards, > >> >> >> > > Amogh Desai > >> >> >> > > > >> >> >> > > On Thu, Feb 22, 2024 at 3:32 AM Oliveira, Niko > >> >> >> > <oniko...@amazon.com.invalid > >> >> >> > > > > >> >> >> > > wrote: > >> >> >> > > > >> >> >> > > > The Astral folks also seem very focused on it being a > >> >> >> drop-in/compliant > >> >> >> > > > replacement for pip. So I think it's definitely worth > >> >> >> > > > dropping > >> >> it in > >> >> >> > and > >> >> >> > > > seeing if we get the expected performance improvements. > >> >> >> > > > If tests > >> >> >> still > >> >> >> > > pass > >> >> >> > > > and user facing constraints and install instructions > >> >> >> > > > remain > >> >> >> unchanged I > >> >> >> > > > don't see why not, if someone is willing to spend the > >> >> >> > > > time on > >> it. > >> >> >> Never > >> >> >> > > > mind the extra features it would give us (I, like others, > >> >> >> > > > am also > >> >> >> very > >> >> >> > > > excited about --resolution=lowest, ability). > >> >> >> > > > > >> >> >> > > > ________________________________ > >> >> >> > > > From: Andrey Anshin <andrey.ans...@taragol.is> > >> >> >> > > > Sent: Tuesday, February 20, 2024 12:26:56 AM > >> >> >> > > > To: dev@airflow.apache.org > >> >> >> > > > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] > >> >> >> > > > Considering > >> >> >> trying > >> >> >> > > > out uv for our CI workflows > >> >> >> > > > > >> >> >> > > > CAUTION: This email originated from outside of the > >> organization. > >> >> Do > >> >> >> not > >> >> >> > > > click links or open attachments unless you can confirm > >> >> >> > > > the sender > >> >> >> and > >> >> >> > > know > >> >> >> > > > the content is safe. > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > AVERTISSEMENT: Ce courrier électronique provient d’un > >> >> >> > > > expéditeur > >> >> >> > externe. > >> >> >> > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > >> >> >> > > > si > >> >> vous ne > >> >> >> > > pouvez > >> >> >> > > > pas confirmer l’identité de l’expéditeur et si vous > >> >> >> > > > n’êtes pas > >> >> >> certain > >> >> >> > > que > >> >> >> > > > le contenu ne présente aucun risque. > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > I share Andrey's skepticism. It's just yet another tool > >> >> >> > > > > which > >> >> has > >> >> >> an > >> >> >> > > > unclear > >> >> >> > > > development strategy. > >> >> >> > > > > >> >> >> > > > My point was more about a matter of presentation. If > >> >> >> > > > someone told > >> >> >> you > >> >> >> > > "this > >> >> >> > > > is a new tool, like a killer of previous tools" then you > >> >> >> > > > might > >> >> think > >> >> >> > > > "Yeah...yeah...yeah.. yet another replacement to tool X... > >> >> >> > > > not > >> >> >> really > >> >> >> > > > interesting". On the other hand if someone told you what > >> >> >> > > > in cases > >> >> >> you > >> >> >> > > might > >> >> >> > > > solve, then this might be a mind changer. > >> >> >> > > > > >> >> >> > > > Especially the promising `--resolution=lowest` option. We > >> >> >> > > > always > >> >> >> want > >> >> >> > to > >> >> >> > > > test something with minimal dependencies because we are > >> >> >> > > > not sure > >> >> >> that > >> >> >> > it > >> >> >> > > > might work with pretty old dependencies, and recently > >> >> >> > > > I've > >> >> started > >> >> >> to > >> >> >> > > work > >> >> >> > > > on POC to collect minimal versions of the Airflow and > >> Providers. > >> >> >> And at > >> >> >> > > the > >> >> >> > > > moment when I almost finished it the uv was released. > >> >> >> > > > Well > >> >> >> sometimes it > >> >> >> > > is > >> >> >> > > > better to wait a bit and maybe someone would invent the > >> >> >> > > > same solution 😁 and you don't have to spend a personal > time. > >> >> >> > > > > >> >> >> > > > So as POC I'm on it, we still need a `pip` and validate > >> >> >> > > > some > >> >> stuff > >> >> >> by a > >> >> >> > > pip > >> >> >> > > > because it is only one officially supported way to > >> >> >> > > > install > >> >> Airflow > >> >> >> but > >> >> >> > if > >> >> >> > > > something could be improved in the CI then I'm on it, in > >> >> >> > > > most > >> >> cases > >> >> >> it > >> >> >> > > > would be behind of Breeze and many of the contributors > >> >> >> > > > might be > >> >> even > >> >> >> > not > >> >> >> > > > noticed that something changed. > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > >> >> >> > > > On Tue, 20 Feb 2024 at 09:56, Jarek Potiuk > >> >> >> > > > <ja...@potiuk.com> > >> >> >> wrote: > >> >> >> > > > > >> >> >> > > > > Actually - of you read that blog post, the strategy is > >> >> >> > > > > clear - > >> >> >> they > >> >> >> > aim > >> >> >> > > > to > >> >> >> > > > > create a comprehensive packaging tooling and improvnts > >> >> >> > > > > are > >> >> >> measured > >> >> >> > > > (80-100 > >> >> >> > > > > times they claim - I using caching - they (unlike pip) > >> >> >> > > > > use a > >> >> lot > >> >> >> of > >> >> >> > > local > >> >> >> > > > > caching including resolving dependencies). > >> >> >> > > > > > >> >> >> > > > > So I think both arguments are not valid if you ask me. > >> >> >> > > > > > >> >> >> > > > > wt., 20 lut 2024, 02:37 użytkownik Alexander Shorin < > >> >> >> > kxe...@apache.org > >> >> >> > > > > >> >> >> > > > > napisał: > >> >> >> > > > > > >> >> >> > > > > > I share Andrey's skepticism. It's just yet another > >> >> >> > > > > > tool which > >> >> >> has > >> >> >> > an > >> >> >> > > > > > unclear development strategy. Should you make it a > >> >> >> > > > > > free > >> >> testing > >> >> >> > > suite? > >> >> >> > > > > What > >> >> >> > > > > > project would receive in exchange? A lot of words > >> >> >> > > > > > about being > >> >> >> > faster, > >> >> >> > > > but > >> >> >> > > > > > how much? Are these milliseconds worth to change the > >> >> >> > > > > > stable > >> >> tool > >> >> >> > > with a > >> >> >> > > > > new > >> >> >> > > > > > one? And will it notably improve something? > >> >> >> > > > > > > >> >> >> > > > > > I think it's worth to try it just for fun and provide > >> >> feedback, > >> >> >> but > >> >> >> > > > it'll > >> >> >> > > > > > have to pass a long road to become such stable as pip. > >> >> >> > > > > > > >> >> >> > > > > > -- > >> >> >> > > > > > ,,,^..^,,, > >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > > On Tue, Feb 20, 2024 at 3:06 AM Jarek Potiuk < > >> >> ja...@potiuk.com> > >> >> >> > > wrote: > >> >> >> > > > > > > >> >> >> > > > > > > My opinion: > >> >> >> > > > > > > > >> >> >> > > > > > > I think there is a place for a number of such tools. > >> >> >> > > > > > > For a > >> >> >> long > >> >> >> > > time > >> >> >> > > > > the > >> >> >> > > > > > > packaging team and `pip` team have been working not > >> >> >> > > > > > > only on > >> >> >> `pip` > >> >> >> > > > > > > implementation but also (and most importantly) to > >> >> >> > > > > > > make sure > >> >> >> that > >> >> >> > > what > >> >> >> > > > > > `pip` > >> >> >> > > > > > > does is to be the beacon of standardisation of > >> >> >> > > > > > > packaging > >> >> APIs > >> >> >> and > >> >> >> > > > PEPs. > >> >> >> > > > > > It > >> >> >> > > > > > > will never IMHO have a lot of the fancy features > >> >> >> > > > > > > that other > >> >> >> tools > >> >> >> > > > might > >> >> >> > > > > > > provide (like the ones I mentioned). It will always > >> >> >> > > > > > > be > >> >> there > >> >> >> to > >> >> >> > > > provide > >> >> >> > > > > > the > >> >> >> > > > > > > robust and solid CLI to run all packaging things, > >> >> >> > > > > > > but there > >> >> >> are > >> >> >> > > > plenty > >> >> >> > > > > of > >> >> >> > > > > > > opportunities to provide improved or modified, or > >> >> >> > > > > > > more (or > >> >> >> less) > >> >> >> > > > > > > opinionated ways of doing things that are > >> >> >> > > > > > > addressing some > >> >> >> cases > >> >> >> > > that > >> >> >> > > > > > `pip` > >> >> >> > > > > > > team simply will not be able or willing to handle, > >> >> preferring > >> >> >> > > "pure" > >> >> >> > > > > > > standard approach vs. implement all the optional > things. > >> >> For > >> >> >> > > example > >> >> >> > > > > the > >> >> >> > > > > > > way how pre-releases are handled can be improved to > >> >> >> > > > > > > be more > >> >> >> > > > selective. > >> >> >> > > > > > The > >> >> >> > > > > > > PEP describing it gives the tools an option to add > >> >> >> > > > > > > more > >> >> fancy > >> >> >> > > > > behaviours > >> >> >> > > > > > > (some of which we could find useful in our CI > tooling). > >> >> Should > >> >> >> > > `pip` > >> >> >> > > > > > > implement those - I don't think so. It would > >> >> >> > > > > > > distract > >> >> >> maintainers > >> >> >> > > > from > >> >> >> > > > > > > other more important things. It is quite ok to use > >> >> >> > > > > > > other > >> >> >> tooling > >> >> >> > in > >> >> >> > > > > > places > >> >> >> > > > > > > like our CI, where they do some parts of the > >> >> >> > > > > > > installation > >> >> >> better. > >> >> >> > > > > > > > >> >> >> > > > > > > For me `pip` is going more into the direction of > >> >> >> > > > > > > `usable > >> >> >> > reference > >> >> >> > > > > > > implementation of package installed` - any > >> >> >> > > > > > > standard/ PEP > >> >> will > >> >> >> not > >> >> >> > > > > matter > >> >> >> > > > > > if > >> >> >> > > > > > > `pip` does not implement it. But others might go in > >> >> different > >> >> >> > > > > directions > >> >> >> > > > > > > and implement some less popular features and do it > >> >> >> > > > > > > better, > >> >> >> > faster, > >> >> >> > > > with > >> >> >> > > > > > > greater flexibility. IMHO it's a win-win. > >> >> >> > > > > > > > >> >> >> > > > > > > J. > >> >> >> > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > On Mon, Feb 19, 2024 at 11:40 PM Andrey Anshin < > >> >> >> > > > > andrey.ans...@taragol.is > >> >> >> > > > > > > > >> >> >> > > > > > > wrote: > >> >> >> > > > > > > > >> >> >> > > > > > > > Yesterday my friend shared with me that tool and > >> >> >> > > > > > > > I've > >> >> been > >> >> >> told > >> >> >> > > > that > >> >> >> > > > > > more > >> >> >> > > > > > > > presumably it would be a niche tool. I've been > >> >> >> > > > > > > > told "who > >> >> >> needs > >> >> >> > > yet > >> >> >> > > > > > > another > >> >> >> > > > > > > > installer which stands to resolve all your problems' > >> '. > >> >> >> > > > > > > > I guess I was wrong? > >> >> >> > > > > > > > > >> >> >> > > > > > > > On Tue, 20 Feb 2024 at 00:53, Jarek Potiuk < > >> >> >> ja...@potiuk.com> > >> >> >> > > > wrote: > >> >> >> > > > > > > > > >> >> >> > > > > > > > > Hey everyone, > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > Few days ago the ruff creators have released a > >> >> >> > > > > > > > > new tool > >> >> >> uv - > >> >> >> > > > which > >> >> >> > > > > is > >> >> >> > > > > > > an > >> >> >> > > > > > > > > extremely fast (written in rust) and fully > >> >> >> > > > > > > > > featured > >> >> tool > >> >> >> > > > generally > >> >> >> > > > > > > fully > >> >> >> > > > > > > > > compatible with `pip`. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > Blog post here: > >> >> >> > > > > > > > > https://eur03.safelinks.protection.outlook.com/ > >> >> >> > > > > > > > > ?ur > >> >> >> > > > > > > > > l=https%3A%2F%2Fastral.sh%2Fblog%2Fuv&data=05%7 > >> >> >> > > > > > > > > C02 > >> >> >> > > > > > > > > %7CJens.Scheffler%40de.bosch.com%7Cda57d392cf6a > >> >> >> > > > > > > > > 479 > >> >> >> > > > > > > > > 9ce6608dc36a01ff7%7C0ae51e1907c84e4bbb6d648ee58 > >> >> >> > > > > > > > > 410 > >> >> >> > > > > > > > > f4%7C0%7C0%7C638445308555414247%7CUnknown%7CTWF > >> >> >> > > > > > > > > pbG > >> >> >> > > > > > > > > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT > >> >> >> > > > > > > > > iI6 > >> >> >> > > > > > > > > Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=KTqeT > >> >> >> > > > > > > > > xus > >> >> >> > > > > > > > > gSBxgBClVc8LhjvPCJAhcmlkXM%2FK%2B53EzYM%3D&rese > >> >> >> > > > > > > > > rve > >> >> >> > > > > > > > > d=0 > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > It looks like It has a number of things that > >> >> >> > > > > > > > > would make > >> >> >> our > >> >> >> > CI > >> >> >> > > > > cases > >> >> >> > > > > > > and > >> >> >> > > > > > > > > tooling quite a bit faster and better including > >> >> >> > > > > > > > > a few > >> >> >> things > >> >> >> > > > that I > >> >> >> > > > > > > have > >> >> >> > > > > > > > > implemented some workarounds for and some that > >> >> >> > > > > > > > > I have > >> >> not > >> >> >> > > > > > > > > implemented because `pip` had no good solution. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > I looked at the docs and it solves some > >> >> >> > > > > > > > > problems that > >> >> are > >> >> >> > > > currently > >> >> >> > > > > > > > > difficult or impossible to handle with `pip`: > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > * ability to use overrides (which are > >> >> >> > > > > > > > > constraints on > >> >> >> > steroids - > >> >> >> > > > > > > allowing > >> >> >> > > > > > > > to > >> >> >> > > > > > > > > override limits specified by the packages - > >> >> >> > > > > > > > > this will > >> >> be > >> >> >> very > >> >> >> > > > > useful > >> >> >> > > > > > to > >> >> >> > > > > > > > > better handle our cases with "chicken-egg" > >> >> >> > > > > > > > > providers > >> >> (for > >> >> >> > > example > >> >> >> > > > > > like > >> >> >> > > > > > > we > >> >> >> > > > > > > > > had in FAB) where we have pre-release packages > >> >> depending > >> >> >> on > >> >> >> > > each > >> >> >> > > > > > other > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > * different resolution strategies including > >> >> >> > --resolution=lowest > >> >> >> > > > > which > >> >> >> > > > > > > > will > >> >> >> > > > > > > > > finally allow us to see whether airflow's lower > >> >> >> > > > > > > > > bounds > >> >> are > >> >> >> > > still > >> >> >> > > > > > > holding > >> >> >> > > > > > > > > (i.e. - will our test still pass if we use the > >> >> >> > > > > > > > > lowest > >> >> >> > supported > >> >> >> > > > > > version > >> >> >> > > > > > > > of > >> >> >> > > > > > > > > our dependencies? this is something i wanted > >> >> >> > > > > > > > > to do for > >> >> >> quite > >> >> >> > > > some > >> >> >> > > > > > time > >> >> >> > > > > > > > and > >> >> >> > > > > > > > > recorded an issue for that - > >> >> >> > > > > > > > > https://eur03.safelinks.protection.outlook.com/ > >> >> >> > > > > > > > > ?ur > >> >> >> > > > > > > > > l=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2 > >> >> >> > > > > > > > > Fis > >> >> >> > > > > > > > > sues%2F35549&data=05%7C02%7CJens.Scheffler%40de > >> >> >> > > > > > > > > .bo > >> >> >> > > > > > > > > sch.com%7Cda57d392cf6a4799ce6608dc36a01ff7%7C0a > >> >> >> > > > > > > > > e51 > >> >> >> > > > > > > > > e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638445308 > >> >> >> > > > > > > > > 555 > >> >> >> > > > > > > > > 418852%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw > >> >> >> > > > > > > > > MDA > >> >> >> > > > > > > > > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D > >> >> >> > > > > > > > > %7C > >> >> >> > > > > > > > > 0%7C%7C%7C&sdata=Nz7du2MmavpWhHcFFfd8Qj2SbKWZcm > >> >> >> > > > > > > > > Xxs > >> >> >> > > > > > > > > OlfMGgftwQ%3D&reserved=0 but lack of tooling > >> >> >> > > > > > > > > support made it a wish, with > >> >> >> > > > > > `--resolution=lowest` > >> >> >> > > > > > > it > >> >> >> > > > > > > > > seems like super-easy thing to do. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > * It is said to be many, many times faster - > >> >> >> > > > > > > > > with > >> >> better > >> >> >> > > caching > >> >> >> > > > > and > >> >> >> > > > > > > > > resolution speeds (similarly like with ruff > >> >> >> > > > > > > > > they claim > >> >> >> orders > >> >> >> > > of > >> >> >> > > > > > > > magnitude > >> >> >> > > > > > > > > speedups in a number of cases). We can likely > >> >> >> > > > > > > > > make very > >> >> >> good > >> >> >> > > use > >> >> >> > > > of > >> >> >> > > > > > it > >> >> >> > > > > > > > and > >> >> >> > > > > > > > > speed up some parts of our CI workflow > >> significantly. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > I might likely do some experimenting with uv in > >> >> >> > > > > > > > > our > >> >> >> > toolchain, > >> >> >> > > > but > >> >> >> > > > > > > wanted > >> >> >> > > > > > > > > to make sure we are all aware of it - and ask > >> >> >> > > > > > > > > if > >> >> someone > >> >> >> has > >> >> >> > > > > > something > >> >> >> > > > > > > > > against it (and maybe someone would like to do > >> >> >> > > > > > > > > some > >> >> work > >> >> >> > there > >> >> >> > > > > trying > >> >> >> > > > > > > it > >> >> >> > > > > > > > > out - I will be happy to guide others with the > >> >> dev/tooling > >> >> >> > > > mindset > >> >> >> > > > > > and > >> >> >> > > > > > > > > incline to do some changes there/review PRs and > >> >> cooperate > >> >> >> on > >> >> >> > > > > testing > >> >> >> > > > > > > > those > >> >> >> > > > > > > > > things. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > It's not a user-facing change, and I do not > >> >> >> > > > > > > > > think we > >> >> want > >> >> >> to > >> >> >> > > get > >> >> >> > > > > rid > >> >> >> > > > > > of > >> >> >> > > > > > > > > `pip` as an installation tool in general (in > >> >> >> > > > > > > > > our images > >> >> >> and > >> >> >> > > user > >> >> >> > > > > > facing > >> >> >> > > > > > > > > side) - it's mostly an internal CI tooling > >> >> >> > > > > > > > > improvement > >> >> I > >> >> >> am > >> >> >> > > > > thinking > >> >> >> > > > > > > of. > >> >> >> > > > > > > > > Maybe at some point in time we can recommend it > >> >> >> > > > > > > > > also > >> >> for > >> >> >> > > > > development > >> >> >> > > > > > > > > workflows, and maybe someday it will gain > >> >> >> > > > > > > > > enough > >> >> >> popularity > >> >> >> > to > >> >> >> > > > > think > >> >> >> > > > > > > > about > >> >> >> > > > > > > > > recommending it to our users, but definitely > >> >> >> > > > > > > > > not now > >> >> nor > >> >> >> in > >> >> >> > > even > >> >> >> > > > > > > mid-term > >> >> >> > > > > > > > > future. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > Let me know what you think. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > Repo here: > >> >> >> > > > > > > > > https://eur03.safelinks.protection.outlook.com/ > >> >> >> > > > > > > > > ?ur > >> >> >> > > > > > > > > l=https%3A%2F%2Fgithub.com%2Fastral-sh%2Fuv&dat > >> >> >> > > > > > > > > a=0 > >> >> >> > > > > > > > > 5%7C02%7CJens.Scheffler%40de.bosch.com%7Cda57d3 > >> >> >> > > > > > > > > 92c > >> >> >> > > > > > > > > f6a4799ce6608dc36a01ff7%7C0ae51e1907c84e4bbb6d6 > >> >> >> > > > > > > > > 48e > >> >> >> > > > > > > > > e58410f4%7C0%7C0%7C638445308555424433%7CUnknown > >> >> >> > > > > > > > > %7C > >> >> >> > > > > > > > > TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI > >> >> >> > > > > > > > > iLC > >> >> >> > > > > > > > > JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata > >> >> >> > > > > > > > > =Ln > >> >> >> > > > > > > > > > >> XRuNo6aJwsLPWwbSJrls47%2BfqH2JSMpyt61h%2F0e1g%3D&reserved=0 > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > J. > >> >> >> > > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > > >> >> >> > > >> >> >> > >> >> > > >> >> > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > > > ________________________________ > Strike Technologies, LLC (“Strike”) is part of the GTS family of > companies. Strike is a technology solutions provider, and is not a broker > or dealer and does not transact any securities related business directly > whatsoever. This communication is the property of Strike and its > affiliates, and does not constitute an offer to sell or the solicitation of > an offer to buy any security in any jurisdiction. It is intended only for > the person to whom it is addressed and may contain information that is > privileged, confidential, or otherwise protected from disclosure. > Distribution or copying of this communication, or the information contained > herein, by anyone other than the intended recipient is prohibited. If you > have received this communication in error, please immediately notify Strike > at i...@striketechnologies.com, and delete and destroy any copies hereof. > ________________________________ > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments > are intended solely for the addressee. This transmission is covered by the > Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > information contained in this transmission is confidential in nature and > protected from further use or disclosure under U.S. Pub. L. 106-102, 113 > U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > legal privilege. Your use or disclosure of this information for any purpose > other than that intended by its transmittal is strictly prohibited, and may > subject you to fines and/or penalties under federal and state law. If you > are not the intended recipient of this transmission, please DESTROY ALL > COPIES RECEIVED and confirm destruction to the sender via return > transmittal. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org >
