Filed https://issues.apache.org/jira/browse/BEAM-8900 to address the inefficiency discussed here. Thanks everyone.
On Thu, Dec 5, 2019 at 2:53 PM Valentyn Tymofieiev <valen...@google.com> wrote: > Note that so far we have not been staging wheels, since SDK does not have > a knowledge of a target platform, but there is > https://issues.apache.org/jira/browse/BEAM-4032 to add this support. > > On Thu, Dec 5, 2019 at 2:35 PM Chad Dombrova <chad...@gmail.com> wrote: > >> On Thu, Dec 5, 2019 at 12:36 PM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >> Ah nice, so then the workflow would be: download [missing] deps from pypi >>> into a long-lived cache directory, then download copy the same deps >>> into a short-lived temporary directory, using long-lived cache directory >>> as SoT, then stage files from a short-lived temporary directory and clean >>> it up. Is that what you are suggesting, Chad? >>> >> Yes, I just did a quick test to confirm: >> >> # download or build wheels of anything that's missing from the cache >> # note: we're including gcp extras: >> pip wheel apache_beam[gcp]==2.16 --wheel-dir /tmp/wheel-cache >> # copy some of those wheels somewhere else >> # note: we're excluding gcp extras >> pip download apache_beam==2.16 --no-binary --find-links=/tmp/wheel-cache >> --dest /tmp/wheel-dest/ >> # rerun to confirm that cached wheels are being re-used instead of >> downloaded from pypi >> pip wheel apache_beam[gcp]==2.16 --wheel-dir /tmp/wheel-cache >> >> /tmp/wheel-dest/ will now have a subset of the deps from >> /tmp/wheel-cache, excluding the gcp extras. >> >> Note that for some reason the equal sign after —find-links is required, >> at least for me on pip 19.1.1. Using a space resulted in an error. >> >> -chad >> >> >>
