On Thu, Dec 5, 2019 at 12:36 PM Valentyn Tymofieiev <valen...@google.com> wrote:
Ah nice, so then the workflow would be: download [missing] deps from pypi > into a long-lived cache directory, then download copy the same deps into > a short-lived temporary directory, using long-lived cache directory as > SoT, then stage files from a short-lived temporary directory and clean it > up. Is that what you are suggesting, Chad? > Yes, I just did a quick test to confirm: # download or build wheels of anything that's missing from the cache # note: we're including gcp extras: pip wheel apache_beam[gcp]==2.16 --wheel-dir /tmp/wheel-cache # copy some of those wheels somewhere else # note: we're excluding gcp extras pip download apache_beam==2.16 --no-binary --find-links=/tmp/wheel-cache --dest /tmp/wheel-dest/ # rerun to confirm that cached wheels are being re-used instead of downloaded from pypi pip wheel apache_beam[gcp]==2.16 --wheel-dir /tmp/wheel-cache /tmp/wheel-dest/ will now have a subset of the deps from /tmp/wheel-cache, excluding the gcp extras. Note that for some reason the equal sign after —find-links is required, at least for me on pip 19.1.1. Using a space resulted in an error. -chad
