[Distutils]Approaching wheel 1.0
Today I pushed 29 new commits to the pypa/wheel repository, closing around 15 issues. This is a major overhaul/refactoring of the wheel project. Here's the short list of major changes: * The documentation has been rewritten to conform to PyPA guidelines (h ttps://gist.github.com/qwcode/8431828) * Wheel signing and verifying features were stripped out, as they were scarcely used and would have potentially presented a significant maintenance burden * The wheel install feature was also stripped out, largely from the same reason as above * Added a new command, "wheel pack" that will create a new wheel file from a previously unpacked wheel directory, regenerating the RECORD file in the process * Added a new metadata key, "license_files" which can be used to include multiple license files in wheels (this deprecates the "license_file" key) There are still a few issues to work out before I'm comfortable releasing 1.0.0 final. For that, I would like to enlist the help of the Python packaging community, as I may not have the necessary expertise: * Py_UNICODE_SIZE / WITH_PYMALLOC / Py_DEBUG on Windows: https://github .com/pypa/wheel/issues/171 * Supporting tools like milksnake by allowing platform specific wheel tags with ABI set to "none": * https://github.com/pypa/wheel/issues/125 * https://github.com/pypa/wheel/issues/153 * https://github.com/pypa/wheel/issues/175 * https://github.com/pypa/wheel/issues/185 * Automatically detecting C extensions that use the limited ABI and tagging wheels without an explicit --py-limited-abi: https://github.com /pypa/wheel/issues/198 * bdist_wheel not cleaning build directory (really a distutils problem, but can wheel reasonably do something here?) https://github.com/pypa/wh eel/issues/117 I mostly need help defining the concrete steps to take to resolve these issues.-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/X2NNYA25EBAQYTPKRSJASPIBY6YT5VKB/
[Distutils] Dependency resolution in pip
I recently stumbled into a worrying problem with pip. I found out that doing "pip install pusher requests" installs urllib3 v1.23 as a dependency even though requests specifically restricts the version to lower than 1.23. Then if instead I do "pip install requests pusher" it installs urllib3 v1.22 as expected. As I recall, pip has long had a problem with combining version specifiers and extras when the same target has been required from multiple sources. What I wanted to ask was, is this a simple bug, or a larger unresolved design problem? Should pip also take into consideration the requirements from existing installed packages so pip won't end up installing upgrades they're incompatible with?-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/S4P6WTZL2GTLYM2E2BFPRRWE7DVUE5FD/
[Distutils] Re: Excluding specific Python modules from wheels
I am currently working on implementing a "wheel repack" command which lets you unpack a wheel, modify it and then repack it again while keeping the wheel RECORD consistent. I think this is something that would solve your use case. ke, 2018-05-30 kello 10:07 +0200, Michal Petrucha kirjoitti: > Hello everyone, > > I've been looking for a way to ensure that certain modules don't end > up in a wheel, while the rest of the package they reside in does. If I > only cared about sdist, I could add a MANIFEST.in, in which I'd > exclude those specific files, however, unfortunately, MANIFEST.in has > no effect on bdists (at least of the wheel kind). > > The use case is that our application auto-generates a parser and lexer > with ply, and that parser might not work with different versions of > ply. Since we don't have a whole lot of control over what version > users have installed in their environments, we'd like to generate > those modules in the target environment. > > I took a deep dive into distutils and setuptools, and as far as I can > see, any Python modules residing inside a package listed in the > packages argument to setup() are included in the distribution > unconditionally. Searching this mailing list only reveals a short > thread from nine years ago [1] without any solution... > > For now, the easiest hacky solution for me is to add a couple of > os.remove calls to setup.py, but I'm not a big fan of setup.py messing > with the source tree. > > And as a follow-up question, is there any post-installation hook that > we could use to trigger regeneration of those files? > > Any ideas? > > Michal > > > [1]: > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/NB7NBE2IE7N4KZQZW7IWZ2XE4RQ7DTEG/ > > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/V2LCHK3672DLQ7BZLPH5S46O4A6RYFRC/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/NVNXHTHTX62MPEBB5UO46CSLXYLB2X2G/
Re: [Distutils] Removing wheel signing features from the wheel library
to, 2018-03-22 kello 21:56 +, Thomas Kluyver kirjoitti: > On Thu, Mar 22, 2018, at 9:25 PM, alex.gronh...@nextday.fi wrote: > > > I've been wondering about something – zip files already contain CRC > > based checksums for each the stored file. What benefit is there in > > storing a RECORD file which basically duplicates this > > functionality? > > > > In terms of providing a foundation for security checks, I think CRC > checksums are insufficient - they are meant to detect random data > corruption, not a deliberate effort to make a malicious file. If someone wanted to make a malicious file, what's preventing them from modifying the RECORD to match the modified file when there is no cryptographic signing involved? > > You could simply use a cryptographic hash of the entire wheel zip > file. I guess the advantage of storing file hashes in RECORD is that > they can be checked against the installed code, not just the wheel > package. > > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Removing wheel signing features from the wheel library
to, 2018-03-22 kello 16:40 -0400, Wes Turner kirjoitti: > On Thursday, March 22, 2018, Daniel Holth wrote: > > The feature was a building block that was intended to be used in > > much the same way that SHA package hashes are used, providing > > similar security to the ssh-style TOFU model, but less security > > than other imaginable public key systems. The idea was that it > > could provide more security in practice because the signatures > > could exist and be present with the archive, unlike gpg which > > provides loads of security in theory only. Unfortunately wheel > > signatures were never built up. I don't think anyone was tricked > > into believing the primitive provided security on its own. > > The hashes serve as file integrity check but provide no assurance > that they are what the author intended to distribute because there is > no cryptographic signature. > > File hashes help detect bit flips -- due to solar flares -- in > storage or transit, but do not mitigate against malicious package > modification to packages in storage or transit. I've been wondering about something – zip files already contain CRC based checksums for each the stored file. What benefit is there in storing a RECORD file which basically duplicates this functionality? > AFAIU, TUF (The Update Framework) has a mechanism for limiting which > signing keys are valid for which package? Are pre-shared keys then > still necessary, or do we then rely on a PKI where one compromised > CA cert can then forge any other cert? > https://theupdateframework.github.io/ > > On Thu, Mar 22, 2018 at 2:21 PM Nathaniel Smith ared > > wrote: > > > Even if no maintenance were required, it's still a feature that > > > promises to provide security but doesn't. This kind of feature > > > has negative value. > > > I'd also suggest adding a small note to the PEP documenting that > > > the signing feature didn't work out, and maybe linking to > > > Donald's package signing blog post. I know updating PEPs isn't > > > the most common thing, but it's the main documentation of the > > > wheel format and it'll save confusion later. > > > > > > On Mar 22, 2018 10:57 AM, "Wes Turner" > > > wrote: > > > > What maintenance is required? > > > > Here's a link to the previous discussion of this issue: > > > > > > > > "Remove or deprecate wheel-signing features" > > > > https://github.com/pypa/wheel/issues/196 > > > > > > > > What has changed? There is still no method for specifying a > > > > keyring; whereas with GPG, all keys in the ring are trusted. > > > > > > > > On Thursday, March 22, 2018, Nick Coghlan > > > > wrote: > > > > > On 22 March 2018 at 22:35, > > > > > wrote: > > > > > > I am not changing the format of RECORD, I'm simply removing > > > > > > the > > > > > > > > > > > > cryptographic signing and verifying functionality, just the > > > > > > way you > > > > > > > > > > > > described. Hash checking will stay. As we agreed earlier, > > > > > > those > > > > > > > > > > > > features could be deprecated or removed from the PEP > > > > > > entirely. > > > > > > > > > > Cool, that's what I thought you meant, but I figured I should > > > > > double check since our discussion was a while ago now :) > > > > > > > > > > Cheers, > > > > > Nick. > > > > > > > > > > ___ > > > > > Distutils-SIG maillist - Distutils-SIG@python.org > > > > > https://mail.python.org/mailman/listinfo/distutils-sig___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Removing wheel signing features from the wheel library
to, 2018-03-22 kello 13:57 -0400, Wes Turner kirjoitti: > What maintenance is required? It's hard to say for sure, since that depends on what requirements come up in the future. One thing this certainly affects is the test suite which I plan to rewrite because I feel it is woefully inadequate. Removing mostly unused features helps reduce the effort required for that. Ditto for the general code refactoring which I'm also planning on doing. > Here's a link to the previous discussion of this issue: > > "Remove or deprecate wheel-signing features" > https://github.com/pypa/wheel/issues/196 > > What has changed? There is still no method for specifying a keyring; > whereas with GPG, all keys in the ring are trusted. > > On Thursday, March 22, 2018, Nick Coghlan wrote: > > On 22 March 2018 at 22:35, wrote: > > > I am not changing the format of RECORD, I'm simply removing the > > > > > > cryptographic signing and verifying functionality, just the way > > > you > > > > > > described. Hash checking will stay. As we agreed earlier, those > > > > > > features could be deprecated or removed from the PEP entirely. > > > > Cool, that's what I thought you meant, but I figured I should > > double check since our discussion was a while ago now :) > > > > Cheers, > > Nick. > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Removing wheel signing features from the wheel library
to, 2018-03-22 kello 21:44 +1000, Nick Coghlan kirjoitti: > On 22 March 2018 at 05:03, wrote: > > After spending quite some time thinking about this, I've decided to > > cut > > out the wheel signature related features from the wheel codebase, > > unless there is significant resistance among the readers of this > > mailing list. For those not involved in the previous discussion, > > the > > reasoning is that the codebase can be significantly simplified by > > removing this rarely used feature whose practical value is > > questionable > > at best, given the lack of infrastructure for public key > > distribution. > > Clarifying the scope here: is this about removing the hashes from the > RECORD file, or just about dropping the native support for injecting > the RECORD.jws and/or RECORD.p7s file? I ask as both of those > features are covered in the same section of PEP 427: https://www.pyth > on.org/dev/peps/pep-0427/#signed-wheel-files > > If it's just the latter, then I don't see any problem with that at > all - the generated wheels will still be completely compliant with > PEP 427, it's just that anyone that does want to sign RECORD will > need to extract from the archive, sign it, then add the signature > file back in. > > Changing the format of RECORD would be a problem though, since it's a > documented requirement that installers are expected to check those at > installation time. I am not changing the format of RECORD, I'm simply removing the cryptographic signing and verifying functionality, just the way you described. Hash checking will stay. As we agreed earlier, those features could be deprecated or removed from the PEP entirely. > > Cheers, > Nick. > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Removing wheel signing features from the wheel library
After spending quite some time thinking about this, I've decided to cut out the wheel signature related features from the wheel codebase, unless there is significant resistance among the readers of this mailing list. For those not involved in the previous discussion, the reasoning is that the codebase can be significantly simplified by removing this rarely used feature whose practical value is questionable at best, given the lack of infrastructure for public key distribution. If, as a result, there is a public outcry (which I sincerely doubt), I can always add it back to the next release. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
