Re: can pip be made using local Debian packages for any dependencies
Hey. On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft wrote: > > I'm pretty sure that most if not all debian packages already ship the > required information for pip to see them as installed, and if they are > installed and they satisfy the dependency constraints that pip has for those > projects, then they'll be used. Are you sure? I have a test project with a pyproject.toml that contains: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" i.e. no version requirement on setuptools. Also: $ dpkg -l python3-setuptools ... ii python3-setuptools 66.1.1-1 all Python3 Distutils Enhancements Yet when I do e.g.: $ pip install --editable . Defaulting to user installation because normal site-packages is not writeable Obtaining file:///home/test/example Installing build dependencies ... error error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> [2 lines of output] ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none) ERROR: No matching distribution found for setuptools [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. Now the particular error probably comes from my pip.conf, because I set --no-index, at least when I try to install some package that is not installed as Debian package, e.g.: $ pip install acme Defaulting to user installation because normal site-packages is not writeable ERROR: Could not find a version that satisfies the requirement acme (from versions: none) ERROR: No matching distribution found for acme But I need something like that to actually disable any remote downloads. Perhaps you're right however, cause when I run with an empty pip.conf, it does e.g. download files to: ~/.local/lib/python3.11/site-packages/natsort* (which I set as a project dependency), but it doesn't create something like that for setuptools. On the other hand: I've just installed Debian's python3-natsort, rm -rf ~/.local/ and did pip install --editable . again (without pip.conf), and it still seems to download and install natsort to ~/.local . So that doesn't look, as if it would recognize the one from Debian. But even if that would actually work, why does it need the index at https://pypi.org/simple to use the local packages from Debian? > The question of having pip automatically install a debian package instead of > using PyPI basically comes down to a few problems: That might be nice for some users, but I personally wouldn't even need or want that. It should perhaps rather just tell which Debian packages need to be installed. Thanks, Philippe.
Re: can pip be made using local Debian packages for any dependencies
On Sun, Feb 12, 2023 at 9:05 AM Ian Norton wrote: > I requested this kind of thing from the pip folks as > https://github.com/pypa/pip/issues/11644 and others have requested > similar, such as https://github.com/pypa/pip/issues/11607 While they wrote it would be discussed, it doesn't really seem as if they would care too much or as if anything would have happened since then? :-( I cannot really comment too much about the rest you wrote, as I'm not really a Python or Debian packaging expert. What I'd like to have is "simply" ;-) to prevent installation of any remote code (as pip and similar tools, also for other languages, do) and still use them as far as I have the packages from Debian installed. So for example, even if the pyproject.toml would specify some version of a dependency that is not in Debian, it should rather fail than downloading anything from somewhere. Regards, Philippe
Re: can pip be made using local Debian packages for any dependencies
I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used. The question of having pip automatically install a debian package instead of using PyPI basically comes down to a few problems: - pip doesn't know how to invoke apt (and yum, and Conda, and everything else). - how do we translate "foo" on PyPI to the equivalent package in Debian (and Fedora, and Conda, etc) - how do we handle virtual environments? Nobody has ever come up with a particularly good answer to those problems, so pip upstream hasn't been able to even consider doing something like that. Of course Debian could patch it's copy of pip to do that, that would make some of those questions easier, but not all of them, and would drastically diverge it's behavior from the "expected" behavior, which makes it a rough idea in general for Debian to diverge that drastically. On 2/12/2023 3:18:55 AM, Ian Norton wrote: https://packaging.python.org/en/latest/specifications/recording-installed-packages/ defines the python spec where a package such as pyparsing would create a tree of files under: site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is essentially a sha256-based manifest of files and some others. On Sun, 12 Feb 2023 at 08:12, Ian Norton wrote: > > You've made me wonder if it would be feasible to have a debian-centric > tool that populates .dist-info from debs? > > On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote: > > > > I requested this kind of thing from the pip folks as > > https://github.com/pypa/pip/issues/11644 and others have requested > > similar, such as https://github.com/pypa/pip/issues/11607 > > > > On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote: > > > > > > Hey. > > > > > > I hope this is not too off topic. > > > > > > As far as I understand, dh-python, when building packages somehow > > > automatically uses the Debian package names and even prevents e.g. > > > setuptools from downloading any dependencies by setting a (hopefully > > > not running) proxy. > > > > > > > > > I wondered whether it's possible to make tools like pip and setuptools > > > directly use the Debian python packages when resolving dependencies. > > > > > > The main motivation are security constraints, so I had to configure > > > pip so that it cannot just download packages from PyPI (which is > > > rather easy, simply setting no-index in pip.conf). > > > > > > But then of course it also fails to e.g. do an editable install of a > > > locally developed package, when it tries to resolve the dependencies. > > > > > > So I wondered whether it's possible to prevent pip from downloading > > > any remote stuff, while still resolving dependencies (respectively > > > consider them as being resolved) *if* the package is locally installed > > > from the Debian archive? > > > (If a dependency isn't installed from a package it may of course fail.) > > > > > > > > > Thanks, > > > Philippe. > > > > > > PS: Please keep me CCed. > > > [88f0dcef-4969-48af-98f1-0c69549b4875]
Re: can pip be made using local Debian packages for any dependencies
An example using alpine (it seems some packages sporadically include the dist-info folders): There is some more recent info here https://peps.python.org/pep-0627/ / # apk add py3-pip (1/6) Installing py3-six (1.16.0-r3) (2/6) Installing py3-retrying (1.3.3-r3) (3/6) Installing py3-parsing (3.0.9-r0) (4/6) Installing py3-packaging (21.3-r2) (5/6) Installing py3-setuptools (65.6.0-r0) (6/6) Installing py3-pip (22.3.1-r1) Executing busybox-1.35.0-r29.trigger OK: 82 MiB in 34 packages / # pip install pyparsing==3.0.9 Requirement already satisfied: pyparsing==3.0.9 in /usr/lib/python3.10/site-packages (3.0.9) In the above example, pip refuses to double-install pyparsing only because of the dist-info file, however lots of directly pip installed packages produce egg-info folders instead as a I guess those are installed from source by pip. Debian could produce dist-info folders for those instead. Just thinking out-loud Ian On Sun, 12 Feb 2023 at 08:18, Ian Norton wrote: > > https://packaging.python.org/en/latest/specifications/recording-installed-packages/ > defines the python spec where a package such as pyparsing would create > a tree of files under: > site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is > essentially a sha256-based manifest of files and some others. > > On Sun, 12 Feb 2023 at 08:12, Ian Norton wrote: > > > > You've made me wonder if it would be feasible to have a debian-centric > > tool that populates .dist-info from debs? > > > > On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote: > > > > > > I requested this kind of thing from the pip folks as > > > https://github.com/pypa/pip/issues/11644 and others have requested > > > similar, such as https://github.com/pypa/pip/issues/11607 > > > > > > On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote: > > > > > > > > Hey. > > > > > > > > I hope this is not too off topic. > > > > > > > > As far as I understand, dh-python, when building packages somehow > > > > automatically uses the Debian package names and even prevents e.g. > > > > setuptools from downloading any dependencies by setting a (hopefully > > > > not running) proxy. > > > > > > > > > > > > I wondered whether it's possible to make tools like pip and setuptools > > > > directly use the Debian python packages when resolving dependencies. > > > > > > > > The main motivation are security constraints, so I had to configure > > > > pip so that it cannot just download packages from PyPI (which is > > > > rather easy, simply setting no-index in pip.conf). > > > > > > > > But then of course it also fails to e.g. do an editable install of a > > > > locally developed package, when it tries to resolve the dependencies. > > > > > > > > So I wondered whether it's possible to prevent pip from downloading > > > > any remote stuff, while still resolving dependencies (respectively > > > > consider them as being resolved) *if* the package is locally installed > > > > from the Debian archive? > > > > (If a dependency isn't installed from a package it may of course fail.) > > > > > > > > > > > > Thanks, > > > > Philippe. > > > > > > > > PS: Please keep me CCed. > > > >
Re: can pip be made using local Debian packages for any dependencies
https://packaging.python.org/en/latest/specifications/recording-installed-packages/ defines the python spec where a package such as pyparsing would create a tree of files under: site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is essentially a sha256-based manifest of files and some others. On Sun, 12 Feb 2023 at 08:12, Ian Norton wrote: > > You've made me wonder if it would be feasible to have a debian-centric > tool that populates .dist-info from debs? > > On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote: > > > > I requested this kind of thing from the pip folks as > > https://github.com/pypa/pip/issues/11644 and others have requested > > similar, such as https://github.com/pypa/pip/issues/11607 > > > > On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote: > > > > > > Hey. > > > > > > I hope this is not too off topic. > > > > > > As far as I understand, dh-python, when building packages somehow > > > automatically uses the Debian package names and even prevents e.g. > > > setuptools from downloading any dependencies by setting a (hopefully > > > not running) proxy. > > > > > > > > > I wondered whether it's possible to make tools like pip and setuptools > > > directly use the Debian python packages when resolving dependencies. > > > > > > The main motivation are security constraints, so I had to configure > > > pip so that it cannot just download packages from PyPI (which is > > > rather easy, simply setting no-index in pip.conf). > > > > > > But then of course it also fails to e.g. do an editable install of a > > > locally developed package, when it tries to resolve the dependencies. > > > > > > So I wondered whether it's possible to prevent pip from downloading > > > any remote stuff, while still resolving dependencies (respectively > > > consider them as being resolved) *if* the package is locally installed > > > from the Debian archive? > > > (If a dependency isn't installed from a package it may of course fail.) > > > > > > > > > Thanks, > > > Philippe. > > > > > > PS: Please keep me CCed. > > >
Re: can pip be made using local Debian packages for any dependencies
You've made me wonder if it would be feasible to have a debian-centric tool that populates .dist-info from debs? On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote: > > I requested this kind of thing from the pip folks as > https://github.com/pypa/pip/issues/11644 and others have requested > similar, such as https://github.com/pypa/pip/issues/11607 > > On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote: > > > > Hey. > > > > I hope this is not too off topic. > > > > As far as I understand, dh-python, when building packages somehow > > automatically uses the Debian package names and even prevents e.g. > > setuptools from downloading any dependencies by setting a (hopefully > > not running) proxy. > > > > > > I wondered whether it's possible to make tools like pip and setuptools > > directly use the Debian python packages when resolving dependencies. > > > > The main motivation are security constraints, so I had to configure > > pip so that it cannot just download packages from PyPI (which is > > rather easy, simply setting no-index in pip.conf). > > > > But then of course it also fails to e.g. do an editable install of a > > locally developed package, when it tries to resolve the dependencies. > > > > So I wondered whether it's possible to prevent pip from downloading > > any remote stuff, while still resolving dependencies (respectively > > consider them as being resolved) *if* the package is locally installed > > from the Debian archive? > > (If a dependency isn't installed from a package it may of course fail.) > > > > > > Thanks, > > Philippe. > > > > PS: Please keep me CCed. > >
Re: can pip be made using local Debian packages for any dependencies
I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested similar, such as https://github.com/pypa/pip/issues/11607 On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote: > > Hey. > > I hope this is not too off topic. > > As far as I understand, dh-python, when building packages somehow > automatically uses the Debian package names and even prevents e.g. > setuptools from downloading any dependencies by setting a (hopefully > not running) proxy. > > > I wondered whether it's possible to make tools like pip and setuptools > directly use the Debian python packages when resolving dependencies. > > The main motivation are security constraints, so I had to configure > pip so that it cannot just download packages from PyPI (which is > rather easy, simply setting no-index in pip.conf). > > But then of course it also fails to e.g. do an editable install of a > locally developed package, when it tries to resolve the dependencies. > > So I wondered whether it's possible to prevent pip from downloading > any remote stuff, while still resolving dependencies (respectively > consider them as being resolved) *if* the package is locally installed > from the Debian archive? > (If a dependency isn't installed from a package it may of course fail.) > > > Thanks, > Philippe. > > PS: Please keep me CCed. >
