Re: dh_python for single python scripts?
Hey again On Fri, Aug 16, 2024 at 6:26 PM Philippe Cerfon wrote: > Still I fail to understand, where that auto-completed my-script-file > comes from in ptpython. > Any ideas what I'm doing wrong? Maybe it's not me doing something wrong. I have: ls -al /usr/bin/*.py lrwxrwxrwx 1 root root 45 Jun 25 00:44 /usr/bin/clang-tidy-diff-16.py -> ../lib/llvm-16/share/clang/clang-tidy-diff.py -rwxr-xr-x 1 root root 15k Jan 4 2024 /usr/bin/lsusb.py -rwxr-xr-x 1 root root 5,8k Aug 14 19:31 /usr/bin/mesa-overlay-control.py -rwxr-xr-x 1 root root 3,5k Aug 12 2021 /usr/bin/pfs_split_exposures.py lrwxrwxrwx 1 root root 33 Jun 25 00:44 /usr/bin/run-clang-tidy-16.py -> ../lib/llvm-16/bin/run-clang-tidy And all these names are also auto-completed in ptpython when trying to import. Maybe what's left, though, are these py3compile calls, which - I guess dh_python - makes, but which are probably not working in my case? Thanks, Philippe.
Re: dh_python for single python scripts?
Hey Stefano. (Thanks, also Andrey). On Fri, Aug 16, 2024 at 9:47 AM Stefano Rivera wrote: > > Is that even intended to work with dh_python? > > dh_python doesn't care what form your python package is in. It just > looks for the metadata in .dist-info / .egg-info. Then there must be something I do wrong. What I have is: - src/my-script-file.py (yes, hyphens here, which would of course be bad if it were a package - maybe this is the reason for my troubles?) - pyproject.toml [project] name = "my-script" requires-python = ">=3.11" dynamic = ["version"] dependencies = [ "numpy" ] [build-system] requires = ["setuptools>=61.0.0", "setuptools_scm[toml]>=6.2.0"] build-backend = "setuptools.build_meta" [tool.setuptools] script-files = ["src/my-script-file.py"] [tool.setuptools_scm] At first I tried: [project.scripts] my-script = "my-script:main" instead of the script-files = ["src/my-script-file.py"] But that didn't really work (which I guess is no surprise) as project.scripts expects a packge/module to load from, but my-script is none. With script-files = ["src/my-script-file.py"], the result from: python3 -m build --no-isolation *does* include my script at, and the Debian package also places it in /usr/bin - but I somehow get that my-script-file "into" Python. "into Python" is what I wrote before, namely when I run ptpython and use the completion of import my... it shows me my-script-file as if it were a module. So I think something is still wrong, but I cannot find out what. With the above pyproject.toml, the Debian package would contain: drwxr-xr-x root/root 0 2024-08-15 23:56 ./ drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/ drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/bin/ -rwxr-xr-x root/root109240 2024-08-15 23:56 ./usr/bin/my-script-file.py drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/lib/ drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/lib/python3/ drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/lib/python3/dist-packages/ drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/lib/python3/dist-packages/my_script_file-1.0.0.dist-info/ -rw-r--r-- root/root 7 2024-08-15 23:56 ./usr/lib/python3/dist-packages/my_script_file-1.0.0.dist-info/INSTALLER -rw-r--r-- root/root 1824 2024-08-15 23:56 ./usr/lib/python3/dist-packages/my_script_file-1.0.0.dist-info/METADATA -rw-r--r-- root/root91 2024-08-15 23:56 ./usr/lib/python3/dist-packages/my_script_file-1.0.0.dist-info/WHEEL -rw-r--r-- root/root 1 2024-08-15 23:56 ./usr/lib/python3/dist-packages/my_script_file-1.0.0.dist-info/top_level.txt drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/share/ drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/share/doc/ drwxr-xr-x root/root 0 2024-08-15 23:56 ./usr/share/doc/my-script-file/ -rw-r--r-- root/root 169 2024-08-15 23:56 ./usr/share/doc/my-script-file/changelog.Debian.gz -rw-r--r-- root/root 1373 2024-08-15 23:56 ./usr/share/doc/my-script-file/copyright So it retains the hypens, execpt for the dir in dist-packages, where it replaces them with _. It also generates a postinst: #!/bin/sh set -e # Automatically added by dh_python3 if command -v py3compile >/dev/null 2>&1; then py3compile -p my-script-file fi if command -v pypy3compile >/dev/null 2>&1; then pypy3compile -p my-script-file || true fi # End automatically added section Which I guess makes no sense (as there is no such package), or does it? And accordingly a prerm. btw: When I rename the src/my-script-file.py to src/my-script-file (no extension), then the contents of the .deb look the same, but the postinst/prerm are gone. Still I fail to understand, where that auto-completed my-script-file comes from in ptpython. Any ideas what I'm doing wrong? > So, if your package declares dependencies in in the packaging, and > installs correctly with python3-build, it'll work. I had some stupid mistake in my pyproject.toml, after noticing that, it correctly "auto-detected" the dependencies from that and translated them into Debian package names :-) Thanks, Philippe
dh_python for single python scripts?
Hey. I'm already using dh_python for (Python) packages where I have a pyproject.toml with some [project.scripts] section and use python3-setuptools for building, which works quite nicely. Now I do have some standalone python scripts for which it makes not much sense to make them (Python) packages (i.e. there's no __init__.py, etc.), yet I'd still like to have stuff like automatically getting the right Debian package dependencies in a Debian package. Is that even intended to work with dh_python? If so, how does it work, cause so far I didn't manage to get it working. AFAIU using [project.scripts] section doesn't work, as I have no (Python) package. I tried with: [tool.setuptools] script-files = ["src/my-script.py"] While that builds me a .deb: a) script-files is discouraged per setuptools documentation b) the Debian packge has none of the python packages/modules that I import auto-detected as dependencies c) After installing the package, tools like ptpyhon think there'd be a my-script (notice the hyphen) package (i.e. when I type import my it shows me my-script as auto-completion suggestion). Not sure where that comes from, cause there's only: └── usr ├── bin │ └── my-script.py ├── lib │ └── python3 │ └── dist-packages │ └── my_script-1.0.0.dist-info │ ├── INSTALLER │ ├── METADATA │ ├── WHEEL │ └── top_level.txt perhaps from the py3compile in the postinst? Thanks for any help :-) Philippe
Re: can pip be made using local Debian packages for any dependencies
Hey. On Fri, Feb 17, 2023 at 3:35 PM Stefano Rivera wrote: > There is probably some value there. You're safer from a variety of > attacks that *could* theoretically happen on PyPI. Well such language specific package repos (like pypi, npmjs, etc.) have already been (numerous) times been victims from such attacks. So it's not just a theoretical issue, I think. > But, let me deflate Debian's reputation a bit here. > Debian security support doesn't mean you're completely protected. There > is probably a human behind a Debian upload that has vetted the upload > and thinks it is safe. They thought this thing was useful to package for > Debian (so probably not malware), and did some review to see that it > installed itself correctly. They may have reviewed the upstream code, > they may not have. They may review new upstream version diffs, they may > not. (Generally, small things are easy to review, big complex things are > impossible to.) Sure, there are no code audits by Debian maintainers, there's no guarantee that the maintainer retrieved the code in a secure way (whatever that would be). Still, even the download method of PyPI (and friends) - https - has all kinds of issues. Typically one has something of around 150 root CAs trusted, plus several thousands(?) of intermediate CAs from them. Many of those in the hands quite questionable countries or organizations. Again, the DM might just as well download the code via https (and not verify some upstream gpg, if present at all). But at least this would cause *all* Debian users (of that package) to be compromised, which in turn makes it much more likely that any compromise would get noticed. Attacks specific to a single person are no longer easily possible. > But, on balance, for many problems the gains here aren't worth the pain > of restricting yourself to Python modules published in Debian stable > releases. Well I guess it's clear there's no 100% protection. In the end it's simply just nice if one can easily choose whichever one wants. If someone wants to use PyPI code, fine, if someone wants to restrict himself to Debian-only, fine too. Anyway, thanks :-) Regards, Philippe.
Re: can pip be made using local Debian packages for any dependencies
On Wed, Feb 15, 2023 at 9:09 PM Stefano Rivera wrote: > And our selection of Python modules is far from complete. It's not > Debian's intent to provide a mirror of PyPI within Debian. Generally > speaking, we package the modules that we find useful for supporting > building and shipping other python modules and applications. > > We'll only have a single version of each package. And they're usually > not the versions developers want, because in any stable release they're > probably out of date. Well in my case the main motivation was security (i.e. only using code) that has security support by Debian. But shouldn't that use case also be interesting for Debian Maintainers? Whenever their pip would need to download something from PyPI, it would mean that some dependency is likely not fulfilled in Debian (unless of course that Debian package is simply not installed). Cheers, Philippe
Re: can pip be made using local Debian packages for any dependencies
On Wed, Feb 15, 2023 at 5:35 PM Stefano Rivera wrote: > You sure it isn't doing an isolated build? Try --no-build-isolation. I absolutely am not :D (I'm really not a Python expert... or about it's various build systems, pip, etc.). And that seems to have done the trick. And it further seems that there's a counter part with which I can do python3 -m build namely: --no-isolation Am I right that these options simply mean that it doesn't use a venv for the build, thereby "seeing" Debian's already installed packages (or better said the eggs/wheels/whatsoever from them)? Just for my confirmation: If my goal is to simply never ever have pip/build/etc. download anything from some location, but only use local sources (e.g. stuff from Debian package) is the following safe? I set /etc/pip.conf to: [global] no-index = true index-url = https://localhost/ cert = /usr/local/share/ca-certificates/dummy.crt (guess in principle the no-index should already be enough) It seems to work (in the sense of blocking downloads) for both, pip and python -m build ... but I don't quite understand why it also works for build (thought that was independent of pip). Awesome! Thanks guys :-) Philippe
Re: can one change the path of generated entry point console_scripts
Hey Stefano On Wed, Feb 15, 2023 at 5:37 PM Stefano Rivera wrote: > Just move it somewhere else later in the build? e.g. after dh_install. I had tried that before, with a debian/mypackage.install file but got an error that it doesn't find the file. Then I realized that I cannot use e.g.: usr/bin/scriptusr/sbin/ but have to use: debian/mypackage/usr/bin/scriptusr/sbin/ in the .install file. Is that the right way? Or is there a more proper "way"? Thanks, Philippe.
can one change the path of generated entry point console_scripts
Hey. When I use dh-python to build a package that contains a pyproject.toml and uses python3-setuptools for building like e.g. described here https://setuptools.pypa.io/en/latest/userguide/entry_point.html and I use that entry point feature to have a script auto-generated that calls my main():[project.scripts] somescript = "package:main" is it possible to change the path were that script is finally put it the package (i.e. not ./usr/bin/)? I tried to look it up in other Debian source packages, but all that I checked and which contain Python scripts in non-/usr/bin locations simply put a real file there, and not such a auto-generated one. Thanks, Philippe.
Re: can pip be made using local Debian packages for any dependencies
Hey. On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft wrote: > > I'm pretty sure that most if not all debian packages already ship the > required information for pip to see them as installed, and if they are > installed and they satisfy the dependency constraints that pip has for those > projects, then they'll be used. Are you sure? I have a test project with a pyproject.toml that contains: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" i.e. no version requirement on setuptools. Also: $ dpkg -l python3-setuptools ... ii python3-setuptools 66.1.1-1 all Python3 Distutils Enhancements Yet when I do e.g.: $ pip install --editable . Defaulting to user installation because normal site-packages is not writeable Obtaining file:///home/test/example Installing build dependencies ... error error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> [2 lines of output] ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none) ERROR: No matching distribution found for setuptools [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. Now the particular error probably comes from my pip.conf, because I set --no-index, at least when I try to install some package that is not installed as Debian package, e.g.: $ pip install acme Defaulting to user installation because normal site-packages is not writeable ERROR: Could not find a version that satisfies the requirement acme (from versions: none) ERROR: No matching distribution found for acme But I need something like that to actually disable any remote downloads. Perhaps you're right however, cause when I run with an empty pip.conf, it does e.g. download files to: ~/.local/lib/python3.11/site-packages/natsort* (which I set as a project dependency), but it doesn't create something like that for setuptools. On the other hand: I've just installed Debian's python3-natsort, rm -rf ~/.local/ and did pip install --editable . again (without pip.conf), and it still seems to download and install natsort to ~/.local . So that doesn't look, as if it would recognize the one from Debian. But even if that would actually work, why does it need the index at https://pypi.org/simple to use the local packages from Debian? > The question of having pip automatically install a debian package instead of > using PyPI basically comes down to a few problems: That might be nice for some users, but I personally wouldn't even need or want that. It should perhaps rather just tell which Debian packages need to be installed. Thanks, Philippe.
Re: can pip be made using local Debian packages for any dependencies
On Sun, Feb 12, 2023 at 9:05 AM Ian Norton wrote: > I requested this kind of thing from the pip folks as > https://github.com/pypa/pip/issues/11644 and others have requested > similar, such as https://github.com/pypa/pip/issues/11607 While they wrote it would be discussed, it doesn't really seem as if they would care too much or as if anything would have happened since then? :-( I cannot really comment too much about the rest you wrote, as I'm not really a Python or Debian packaging expert. What I'd like to have is "simply" ;-) to prevent installation of any remote code (as pip and similar tools, also for other languages, do) and still use them as far as I have the packages from Debian installed. So for example, even if the pyproject.toml would specify some version of a dependency that is not in Debian, it should rather fail than downloading anything from somewhere. Regards, Philippe
can pip be made using local Debian packages for any dependencies
Hey. I hope this is not too off topic. As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully not running) proxy. I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies. The main motivation are security constraints, so I had to configure pip so that it cannot just download packages from PyPI (which is rather easy, simply setting no-index in pip.conf). But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies. So I wondered whether it's possible to prevent pip from downloading any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive? (If a dependency isn't installed from a package it may of course fail.) Thanks, Philippe. PS: Please keep me CCed.
