Hi all, We are running into problems trying to use our own pypi mirror with Beam. For those who are not well versed in the esotera of python package management, pip provides a few ways to specify urls for the pypi index server:
- command line <https://pip.pypa.io/en/stable/reference/pip_install/#install-index-url>[1]: via --index-url - environment variables <https://pip.pypa.io/en/stable/user_guide/#environment-variables>[2]: via PIP_INDEX_URL. In Beam, we don’t have any way to influence the environment of the boot process that runs pip install. - pip.conf <https://pip.pypa.io/en/stable/user_guide/#config-file>[3]: we could provide this as an artifact, but we don’t have any way of placing it in the correct location (e.g. /etc/pip.conf) on the instance that runs pip install. - requirements.txt files can specify certain pip install flags <https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format>[4], such as --index-url. As such, passing a requirements file via --requirements_file would theoretically work, but we also want to be able to provide dev packages as wheels via --extra_package, which would be installed independently from the requirements file and thus use the default pypi index. We may be able to upload our wheel as an artifact and refer to it using a local path in the requirements file, but this solution seems a bit brittle as the local artifacts path is different for each job. Are there any known solutions to this problem? Here are some ideas: - add support for providing a pip.conf as a known artifact type (akin to --requirements_file). this is by far the most powerful and straightforward solution, but do we have the stomach for yet another cli option? - add support for providing a destination path for artifacts, which would let us install it into /etc/pip.conf. I can see strong safety/security concerns around this. - provide a guarantee that the working directory for the boot process is inside the artifact directory: then we could refer to wheels inside our requirements file using relative paths. We're happy to make a pull request to add support for this feature, but it'd be great to have some input on the ideal solution before we begin. thanks! -chad [1] https://pip.pypa.io/en/stable/reference/pip_install/#install-index-url [2] https://pip.pypa.io/en/stable/user_guide/#environment-variables [3] https://pip.pypa.io/en/stable/user_guide/#config-file [4] https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format -chad
