Hmm... this is a difficult question. I think adding support for a pip.conf probably makes the most sense, despite it being yet another option.
Another alternative is to simply pre-install the dependencies you want (or even just override /etc/pip.conf) in a custom container. On Wed, Sep 9, 2020 at 5:27 PM Chad Dombrova <chad...@gmail.com> wrote: > Hi all, > We are running into problems trying to use our own pypi mirror with Beam. > For those who are not well versed in the esotera of python package > management, pip provides a few ways to specify urls for the pypi index > server: > > - command line > > <https://pip.pypa.io/en/stable/reference/pip_install/#install-index-url>[1]: > via --index-url > - environment variables > <https://pip.pypa.io/en/stable/user_guide/#environment-variables>[2]: > via PIP_INDEX_URL. In Beam, we don’t have any way to influence the > environment of the boot process that runs pip install. > - pip.conf <https://pip.pypa.io/en/stable/user_guide/#config-file>[3]: > we could provide this as an artifact, but we don’t have any way of placing > it in the correct location (e.g. /etc/pip.conf) on the instance that > runs pip install. > - requirements.txt files can specify certain pip install flags > > <https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format>[4], > such as --index-url. As such, passing a requirements file via > --requirements_file would theoretically work, but we also want to be > able to provide dev packages as wheels via --extra_package, which > would be installed independently from the requirements file and thus use > the default pypi index. We may be able to upload our wheel as an artifact > and refer to it using a local path in the requirements file, but this > solution seems a bit brittle as the local artifacts path is different for > each job. > > Are there any known solutions to this problem? Here are some ideas: > > - add support for providing a pip.conf as a known artifact type (akin > to --requirements_file). this is by far the most powerful and > straightforward solution, but do we have the stomach for yet another cli > option? > - add support for providing a destination path for artifacts, which > would let us install it into /etc/pip.conf. I can see strong > safety/security concerns around this. > - provide a guarantee that the working directory for the boot process > is inside the artifact directory: then we could refer to wheels inside our > requirements file using relative paths. > > We're happy to make a pull request to add support for this feature, but > it'd be great to have some input on the ideal solution before we begin. > > thanks! > -chad > > [1] https://pip.pypa.io/en/stable/reference/pip_install/#install-index-url > [2] https://pip.pypa.io/en/stable/user_guide/#environment-variables > [3] https://pip.pypa.io/en/stable/user_guide/#config-file > [4] > https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format > > -chad > >
