Re: [Distutils] Immutable Files on PyPI
On Sun, Sep 28, 2014 at 10:28 PM, Marius Gedminas mar...@pov.lt wrote: On Sun, Sep 28, 2014 at 02:21:11PM -0700, Chris Jerdonek wrote: Would this also affect the ability to update the readme information for a version on PyPI (i.e. the information displayed on the default home page generated by PyPI for the release) once the version has already been uploaded to PyPI? There are sometimes issues encountered with the display of that information that can't easily be checked without doing an actual version upload. restview has a --pypi-strict mode for this use-case #shamelessplug https://pypi.python.org/pypi/restview I haven't recently tried reuploading the metadata for a version, mainly because of uncertainty around how PyPI would handle it. In the past when I needed to fix stupid mistakes in my long_description, I'd edit it using the web interface. Thanks for the plug, Marius. But I should clarify that the metadata issues aren't restricted to the long_description. For example, there was a case where a co-maintainer of a project uploaded a new version of the project; and for reasons still unknown to me, the Trove classifiers did not render on the PyPI home page for the project, even though they rendered as expected for previous releases (and there was no change to the setup.py aside from the version). --Chris arius Gedminas -- if (DefRel.empty() == false) -- apt-pkg/policy.cc (apt 0.5.23) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 28.09.2014 23:59, Donald Stufft wrote: On Sep 28, 2014, at 5:36 PM, M.-A. Lemburg m...@egenix.com mailto:m...@egenix.com wrote: On 28.09.2014 21:31, Donald Stufft wrote: Hello All! I'd like to discuss the idea of moving PyPI to having immutable files. This would mean that once you publish a particular file you can never reupload that file again with different contents. This would still allow deleting the file or reuploading it if the checksums match what was there prior. This would be good for a few reasons: * It represents best practices for version numbers. Ideally if two people have version 2.1 of a project, they'll have the same code, however as it stands two people installing at two different times could have two very different versions. * This will make improving the PyPI infrastructure easier, in particular it will make it simpler to move away from using a glusterfs storage array and switch to a redudant set of cloud object stores. In the past this was brought up and a few points were brought against it, those were: 1. That authors could simply change files that were hosted on not PyPI anyways so it didn't really do much. 2. That it was too hard to test a release prior to uploading it due to the nature of distutils requiring you to build the release in the same command as the upload. With the fact that pip no longer hits external URLs by default, I believe that the first item is no longer that large of a factor. People can do whatever they want on external URLs of course, however if something is coming from PyPI as end users should now be aware of, they can know it is immutable. Now that there is twine, which allows uploading already created packages, I also believe that the second item is no longer a concern. People can easily create a distribution using ``setup.py sdist``, test it, and then upload that exact thing they tested using ``twine upload path to sdist``. -1. It does happen that files need to be reuploaded because of a bug in the release process and how people manage their code is really *their* business, not that of PyPI. Can you describe a reasonable hypothetical situation where this would occur often enough as to be something that is likely to happen on a consistent basis? Originally the problem was there was little ability to easily upload pre-created files so there was a reasonable chance that there may be a packaging bug that didn’t get exposed until you actually packaged + released. With the advent of twine though it’s now possible to test the exact bits that get uploaded to PyPI making that particular issue no longer a problem. However, the fact that the files are not immutable *do* cause a number of problems that need to be worked around in the mirroring infrastructure, the CDN, and for scaling PyPI out and removing the glusterfs component. You are missing out on cases, where the release process causes files to be omitted, human errors where packagers forget to apply changes to e.g. documentation files, version files, change logs, etc., where packagers want to add information that doesn't affect the software itself, but meta information included in the distribution files. Such changes often do not affect the software itself, and so are not detected by software tests. If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM. This would mean that package authors could never correct mistakes, remove broken packages distribution files, ones which they may be forced to remove for legal reasons, ones which they find are infected with a virus or trojan, ones which they uploaded for fun or by mistake. This doesn't have anything to do with making the user experience a better one. It is ignorant to assume that package authors who sometimes delete distribution files, or at least want to have the possibility to do so, don't care for their users. We are in Python land, so most authors will know what they are doing and do care for their users. After all: Why do you think I'm arguing against this proposal ? Because I want users of our packages to get the best experience they can get, by downloading complete, correct and working distribution files. This whole idea also has another angle, namely a legal one: the PSF doesn't own the distribution files it hosts on PyPI. So far, the argument to not fix the much too broad license on PyPI was that authors were able to delete files on PyPI to work around the unneeded irrevocable part of that license. With the suggested change, authors would have to give up complete control over their distribution files to the PSF in order for their packages to be installable by pip using its default settings. This kind of lock-in and removal of author rights is not something I can support as PSF director. Those authors are the ones that have created a large part of
Re: [Distutils] Immutable Files on PyPI
On 29 September 2014 09:46, M.-A. Lemburg m...@egenix.com wrote: If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM. My understanding (I'm sure Donald will correct me if I'm wrong) is that it will still be possible to delete files. What will *not* be possible will be to later upload a replacement for a deleted file which contains different content. From a user perspective this means that it's possible for a release to be withdrawn, but replacements will always be new versions, and if I download a particular version, what I get won't depend on when I do the download. This seems like a good thing to me. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29.09.2014 00:51, Nick Coghlan wrote: On 29 Sep 2014 07:37, M.-A. Lemburg m...@egenix.com wrote: -1. It does happen that files need to be reuploaded because of a bug in the release process and how people manage their code is really *their* business, not that of PyPI. FWIW, I am getting increasingly annoyed how PyPI and pip try to dictate the way package authors are supposed to build, manage and host their Python packages and release process. Can we please stop this ? As others have noted, these changes represent the PyPA being opinionated on behalf of the user community, to provide the best possible user experience for the overall Python ecosystem. See my reply to Donald. I find this wrong on several different levels. PyPI is run by the PSF, it's a community resource we provide for package authors and downloaders. We (the PSF) don't take sides. Instead, we want to help everyone feel at home: the package authors who provide the Python eco system with fresh software, as well as the users who greatly benefit from this software. The PyPA takes care of the technical aspects of this, but not the ethical and community building aspects. We'll accommodate the existing publisher community as far as is feasible (that's why PEP 440 is as complicated as it is, for example), but there are going to be times where improving the end user experience means adding new constraints on publishers. External hosting (using PyPI as an index, without also using it for release file hosting) is the primary escape clause for software publishers that prefer to do things differently from the way PyPI does them. That's a user experience we'll also continue to work to improve, to ensure it is clear that it's a fully supported part of the distribution model. Right, so authors will move away from PyPI and put their stuff up elsewhere. Now, how does this help our community ? What if people find that they can only get packages using conda instead of pip, or only by cloning from github, because package authors don't want to bother cutting distribution files anymore ? Do you seriously want to force package authors to cut a new release just because a single uploaded distribution file is broken for some reason and then ask all users who have already installed one of the non-broken ones to upgrade again, even though they are not affected ? Please repeat with me: Package authors care for their users :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 29 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-09-30: Python Meeting Duesseldorf ... tomorrow : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Mon, Sep 29, 2014 at 10:46 +0200, M.-A. Lemburg wrote: On 28.09.2014 23:59, Donald Stufft wrote: On Sep 28, 2014, at 5:36 PM, M.-A. Lemburg m...@egenix.com mailto:m...@egenix.com wrote: On 28.09.2014 21:31, Donald Stufft wrote: Hello All! I'd like to discuss the idea of moving PyPI to having immutable files. This would mean that once you publish a particular file you can never reupload that file again with different contents. This would still allow deleting the file or reuploading it if the checksums match what was there prior. This would be good for a few reasons: * It represents best practices for version numbers. Ideally if two people have version 2.1 of a project, they'll have the same code, however as it stands two people installing at two different times could have two very different versions. * This will make improving the PyPI infrastructure easier, in particular it will make it simpler to move away from using a glusterfs storage array and switch to a redudant set of cloud object stores. In the past this was brought up and a few points were brought against it, those were: 1. That authors could simply change files that were hosted on not PyPI anyways so it didn't really do much. 2. That it was too hard to test a release prior to uploading it due to the nature of distutils requiring you to build the release in the same command as the upload. With the fact that pip no longer hits external URLs by default, I believe that the first item is no longer that large of a factor. People can do whatever they want on external URLs of course, however if something is coming from PyPI as end users should now be aware of, they can know it is immutable. Now that there is twine, which allows uploading already created packages, I also believe that the second item is no longer a concern. People can easily create a distribution using ``setup.py sdist``, test it, and then upload that exact thing they tested using ``twine upload path to sdist``. -1. It does happen that files need to be reuploaded because of a bug in the release process and how people manage their code is really *their* business, not that of PyPI. Can you describe a reasonable hypothetical situation where this would occur often enough as to be something that is likely to happen on a consistent basis? Originally the problem was there was little ability to easily upload pre-created files so there was a reasonable chance that there may be a packaging bug that didn’t get exposed until you actually packaged + released. With the advent of twine though it’s now possible to test the exact bits that get uploaded to PyPI making that particular issue no longer a problem. However, the fact that the files are not immutable *do* cause a number of problems that need to be worked around in the mirroring infrastructure, the CDN, and for scaling PyPI out and removing the glusterfs component. You are missing out on cases, where the release process causes files to be omitted, human errors where packagers forget to apply changes to e.g. documentation files, version files, change logs, etc., where packagers want to add information that doesn't affect the software itself, but meta information included in the distribution files. I've had such cases myself. That's the only real cave-eat i see with the proposal. Then again, wheels don't allow uploading docs/changelogs today. And pypi would continue to allow to change metadata [*]. I also see the advantage of immutability of the (filename-content) relation so I am +0 on the proposal currently. Such changes often do not affect the software itself, and so are not detected by software tests. If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM. No, Donald said deleting would be fine. But you couldn't then re-upload to the same filename with a different checksum because pypi would memorize those properties. This would mean that package authors could never correct mistakes, remove broken packages distribution files, ones which they may be forced to remove for legal reasons, ones which they find are infected with a virus or trojan, ones which they uploaded for fun or by mistake. In this case you would just delete the release under Donald's proposal. best, holger [*] In some way, retro-actively changing the license in release metadata is also questionable. Maybe it should just be made clear that the license pypi metadata is not reliable and one needs to check with the release file itself. I've had a number of companies contact me over related licensing issues of my pypi published software. This doesn't have anything to do with making the user experience a better one. It is ignorant to
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014 18:49, M.-A. Lemburg m...@egenix.com wrote: You are missing out on cases, where the release process causes files to be omitted, human errors where packagers forget to apply changes to e.g. documentation files, version files, change logs, etc., where packagers want to add information that doesn't affect the software itself, but meta information included in the distribution files. Such changes often do not affect the software itself, and so are not detected by software tests. Fixing such packaging errors is the primary intended use case of the post field in PEP 440. Alternatively, many projects will just spin a new release that addresses those issues, just as they would for any other bug. Both of those approaches have the advantage of letting users (especially those operating mirrors, or downloading tarballs and feeding them into a separate redistribution system) easily tell whether or not they have the fixed version. If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM. No, deletion remains supported. The only capability being removed is silent substitution of hosted files with different ones bearing the same name. So if, for example, release 3.1.4 had a packaging error, then deleting it would still be possible, but the replacement would need to be something like 3.1.4.post1 or 3.1.5, rather than being permitted to reuse the 3.1.4 name. This would mean that package authors could never correct mistakes, remove broken packages distribution files, ones which they may be forced to remove for legal reasons, ones which they find are infected with a virus or trojan, ones which they uploaded for fun or by mistake. Removal will still be supported, for these kinds of reasons. Only the silent file substitution loophole is being eliminated - packaging fixes are still fully supported, they're just going to be required to be done in a way that is visible to end users. This doesn't have anything to do with making the user experience a better one. It is ignorant to assume that package authors who sometimes delete distribution files, or at least want to have the possibility to do so, don't care for their users. We are in Python land, so most authors will know what they are doing and do care for their users. Again, only the silent file substitution loophole is being removed. Outright deletion remains a fully supported feature. After all: Why do you think I'm arguing against this proposal ? Because I want users of our packages to get the best experience they can get, by downloading complete, correct and working distribution files. You are entirely correct that removing the ability to delete hosted files would be a bad idea (even though we can't guarantee deletion from third party mirrors). However, there is nothing user friendly about retaining the ability for software publishers to silently replace the contents of a PyPI hosted file without bumping the version number. In particular, that practice actively risks breaking deployments for anyone using the peep installer to automatically cache the first seen hash for each released file (since peep can't tell the difference between the package author doing it and a malicious attacker doing it). This whole idea also has another angle, namely a legal one: the PSF doesn't own the distribution files it hosts on PyPI. So far, the argument to not fix the much too broad license on PyPI was that authors were able to delete files on PyPI to work around the unneeded irrevocable part of that license. With the suggested change, authors would have to give up complete control over their distribution files to the PSF in order for their packages to be installable by pip using its default settings. We are not removing the ability to delete files. This kind of lock-in and removal of author rights is not something I can support as PSF director. Those authors are the ones that have created a large part of our Python eco system and they are the ones that have put in work to get Python to where it is now: one of the best integrated programming languages you can find. We owe a lot to those authors and need to care for them. Yes, we do, but requiring them to bump their version numbers when changing the contents of published files seems like an entirely reasonable constraint to me. If the proposal was to remove the ability to delete files entirely, I would be on your side. Fortunately, that is not the proposal - the proposal is solely to prevent silently modifying their contents without renaming them. Finally, changes such as the above will result in more authors to switch to alternative hosting platforms such as conda/binstar.org or plain github clone + setup.py install (which is becoming increasingly popular). Do you really believe that this will make the user experience a better one in the long run ? Other release
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014 19:04, M.-A. Lemburg m...@egenix.com wrote: Do you seriously want to force package authors to cut a new release just because a single uploaded distribution file is broken for some reason and then ask all users who have already installed one of the non-broken ones to upgrade again, even though they are not affected ? Yes, I do. Silently changing released artefacts is actively user hostile. It breaks mirroring, it breaks redistribution, it breaks security audits, and it can even break installation for security conscious users that are using peep rather than pip. Please repeat with me: Package authors care for their users :-) If that's the case, then checking releases on devpi or the PyPI test instance shouldn't be a problem. I am personally quite open to suggestions for making such checks easier to automate in a consistent way. I am thoroughly *against* retaining a general capability to silently substitute the contents of previously released files with a different payload solely to handle the case of packaging errors that aren't otherwise severe enough to warrant bumping the version number - if they're that insignificant that users that installed the broken one don't need to update, then there doesn't seem to be any urgency in getting the fix published at all, so the package author may even decide to wait until their next release, rather than pushing out an immediate fix. Regards, Nick. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 29 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-09-30: Python Meeting Duesseldorf ... tomorrow : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014 19:50, Nick Coghlan ncogh...@gmail.com wrote: On 29 Sep 2014 19:04, M.-A. Lemburg m...@egenix.com wrote: Do you seriously want to force package authors to cut a new release just because a single uploaded distribution file is broken for some reason and then ask all users who have already installed one of the non-broken ones to upgrade again, even though they are not affected ? Yes, I do. Silently changing released artefacts is actively user hostile. It breaks mirroring, it breaks redistribution, it breaks security audits, and it can even break installation for security conscious users that are using peep rather than pip. One caveat on this: it would potentially be convenient to have a release field in the wheel naming scheme, and adopt a similar approach for other binary formats like Windows installers, specifically to allow those to be updated without needing to do a full source version update. It's the silent substitution of file contents I have a fundamental problem with, not the notion of being able to publish an updated platform specific build artefact without having to bump the source release version. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 29, 2014, at 6:01 AM, Nick Coghlan ncogh...@gmail.commailto:ncogh...@gmail.com wrote: On 29 Sep 2014 19:50, Nick Coghlan ncogh...@gmail.commailto:ncogh...@gmail.com wrote: On 29 Sep 2014 19:04, M.-A. Lemburg m...@egenix.commailto:m...@egenix.com wrote: Do you seriously want to force package authors to cut a new release just because a single uploaded distribution file is broken for some reason and then ask all users who have already installed one of the non-broken ones to upgrade again, even though they are not affected ? Yes, I do. Silently changing released artefacts is actively user hostile. It breaks mirroring, it breaks redistribution, it breaks security audits, and it can even break installation for security conscious users that are using peep rather than pip. One caveat on this: it would potentially be convenient to have a release field in the wheel naming scheme, and adopt a similar approach for other binary formats like Windows installers, specifically to allow those to be updated without needing to do a full source version update. It's the silent substitution of file contents I have a fundamental problem with, not the notion of being able to publish an updated platform specific build artefact without having to bump the source release version. Wheel files already include the idea of a build number baked into the filename. That would be a different filename and thus would be allowed to be uploaded even if you deleted the original Wheel. Is there something about that which wouldn’t work or did it just slip your mind? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
(Fixed quoting indent + some own comments) On Mon, Sep 29, 2014 at 11:04 +, Donald Stufft wrote: On Sep 29, 2014, at 6:01 AM, Nick Coghlan ncogh...@gmail.commailto:ncogh...@gmail.com wrote: On 29 Sep 2014 19:50, Nick Coghlan ncogh...@gmail.commailto:ncogh...@gmail.com wrote: On 29 Sep 2014 19:04, M.-A. Lemburg m...@egenix.commailto:m...@egenix.com wrote: Do you seriously want to force package authors to cut a new release just because a single uploaded distribution file is broken for some reason and then ask all users who have already installed one of the non-broken ones to upgrade again, even though they are not affected ? Yes, I do. Silently changing released artefacts is actively user hostile. It breaks mirroring, it breaks redistribution, it breaks security audits, and it can even break installation for security conscious users that are using peep rather than pip. One caveat on this: it would potentially be convenient to have a release field in the wheel naming scheme, and adopt a similar approach for other binary formats like Windows installers, specifically to allow those to be updated without needing to do a full source version update. It's the silent substitution of file contents I have a fundamental problem with, not the notion of being able to publish an updated platform specific build artefact without having to bump the source release version. Wheel files already include the idea of a build number baked into the filename. That would be a different filename and thus would be allowed to be uploaded even if you deleted the original Wheel. Is there something about that which wouldn’t work or did it just slip your mind? FWIW I'd prefer to go with the each filename maps to one binary content or was deleted guarantee irrespective if it's a wheel, tar, egg or zip file. Besides, the cited mirroring/distribution simplifications wouldn't otherwise materialize i guess. holger --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 29, 2014, at 4:46 AM, M.-A. Lemburg m...@egenix.commailto:m...@egenix.com wrote: You are missing out on cases, where the release process causes files to be omitted, human errors where packagers forget to apply changes to e.g. documentation files, version files, change logs, etc., where packagers want to add information that doesn't affect the software itself, but meta information included in the distribution files. Such changes often do not affect the software itself, and so are not detected by software tests. If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM. This would mean that package authors could never correct mistakes, remove broken packages distribution files, ones which they may be forced to remove for legal reasons, ones which they find are infected with a virus or trojan, ones which they uploaded for fun or by mistake. This doesn't have anything to do with making the user experience a better one. It is ignorant to assume that package authors who sometimes delete distribution files, or at least want to have the possibility to do so, don't care for their users. We are in Python land, so most authors will know what they are doing and do care for their users. After all: Why do you think I'm arguing against this proposal ? Because I want users of our packages to get the best experience they can get, by downloading complete, correct and working distribution files. This whole idea also has another angle, namely a legal one: the PSF doesn't own the distribution files it hosts on PyPI. So far, the argument to not fix the much too broad license on PyPI was that authors were able to delete files on PyPI to work around the unneeded irrevocable part of that license. With the suggested change, authors would have to give up complete control over their distribution files to the PSF in order for their packages to be installable by pip using its default settings. Others already said it, but let me be clear about it, this proposal does not in any way seek to prevent authors from being able to delete files from PyPI. It still allows them to delete anything at anytime and it still publishes that information for mirrors (although mirrors are certainly under no obligation to respect it if they desire not to). I completely agree with you that disallowing authors to *delete* files would be incredibly short sighted and wrong and I would be one of the people against such a change. This proposal is strictly limited to the ability to delete a particular file, let's say foobar-1.0.tar.gz and then reupload a different foobar-1.0.tar.gz in it's place. If a mistake is made in the release, that's *ok* it can be deleted, the only constraint is that with this change you'd need to increment the version in some way, likely with a .postN or just bumping the last digit, to signal to users that the bits in this has changed in some way. This kind of lock-in and removal of author rights is not something I can support as PSF director. Those authors are the ones that have created a large part of our Python eco system and they are the ones that have put in work to get Python to where it is now: one of the best integrated programming languages you can find. We owe a lot to those authors and need to care for them. I *do* deeply care for the experience as an author as well as someone installing from PyPI. After all I use PyPI in both capacities on a regular basis. Finally, changes such as the above will result in more authors to switch to alternative hosting platforms such as conda/binstar.orghttp://binstar.org or plain github clone + setup.py install (which is becoming increasingly popular). Do you really believe that this will make the user experience a better one in the long run ? If we want to make it attractive for package authors to host their packages on PyPI, we have to give them flexibility, respect their rights and be welcoming. I don't believe it's accurate to say people are switching away from PyPI in any sort of relevant numbers. PyPI's usage is increasing, both in the number of people releasing packages and in the number of people consuming packages. Particularly the number of people consuming packages has risen massively. Do you have any numbers or proof to backup the claim that people are switching away? To be completly honest the feedback that I get and see is overwhelmingly positive for every change that has been made so far. That's not to say there haven't been those who have been against some or all of the changes but those people are generally in an extreme minority. This isn't to say that the changes have been globally liked, but that it would be very surprising to me that people are moving away from PyPI and if you have numbers/proof of that I would love to see it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014 21:04, Donald Stufft donald.stu...@rackspace.com wrote: On Sep 29, 2014, at 6:01 AM, Nick Coghlan ncogh...@gmail.com wrote: One caveat on this: it would potentially be convenient to have a release field in the wheel naming scheme, and adopt a similar approach for other binary formats like Windows installers, specifically to allow those to be updated without needing to do a full source version update. It's the silent substitution of file contents I have a fundamental problem with, not the notion of being able to publish an updated platform specific build artefact without having to bump the source release version. Wheel files already include the idea of a build number baked into the filename. That would be a different filename and thus would be allowed to be uploaded even if you deleted the original Wheel. Is there something about that which wouldn’t work or did it just slip your mind? Slipped my mind - we generally leave it out of examples, so I managed to forget the capability was already part of the spec. That means this case should already be fully covered then, even if it requires a pre-upload file renaming. It does suggest that a short section on Dealing with release errors might need to find a home somewhere in PyPUG, though. Cheers, Nick. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014 21:20, holger krekel hol...@merlinux.eu wrote: (Fixed quoting indent + some own comments) On Mon, Sep 29, 2014 at 11:04 +, Donald Stufft wrote: On Sep 29, 2014, at 6:01 AM, Nick Coghlan ncogh...@gmail.commailto: ncogh...@gmail.com wrote: It's the silent substitution of file contents I have a fundamental problem with, not the notion of being able to publish an updated platform specific build artefact without having to bump the source release version. Wheel files already include the idea of a build number baked into the filename. That would be a different filename and thus would be allowed to be uploaded even if you deleted the original Wheel. Is there something about that which wouldn’t work or did it just slip your mind? FWIW I'd prefer to go with the each filename maps to one binary content or was deleted guarantee irrespective if it's a wheel, tar, egg or zip file. Besides, the cited mirroring/distribution simplifications wouldn't otherwise materialize i guess. Right, this is my perspective as well. The point that the wheel format already includes a build ordering field was significant because that file naming scheme has an official specification. Other commands like bdist_egg, bdist_dumb and bdist_wininst aren't as strict about the expected file names, although it would be good to define a suggested optional build numbering convention at least for bdist_egg, such that easy_install will do the right thing, even if the full source level version number isn't bumped. Cheers, Nick. holger --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014, at 13:58, Nick Coghlan ncogh...@gmail.com wrote: Right, this is my perspective as well. The point that the wheel format already includes a build ordering field was significant because that file naming scheme has an official specification. Other commands like bdist_egg, bdist_dumb and bdist_wininst aren't as strict about the expected file names, although it would be good to define a suggested optional build numbering convention at least for bdist_egg, such that easy_install will do the right thing, even if the full source level version number isn't bumped. This is just as relevant for sdists as well. It is quite common to see a broken release due to a missing or wrong MANIFEST.in. Wichert.___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On September 29, 2014 at 8:54:26 AM, Wichert Akkerman (wich...@wiggy.net) wrote: On 29 Sep 2014, at 13:58, Nick Coghlan ncogh...@gmail.com wrote: Right, this is my perspective as well. The point that the wheel format already includes a build ordering field was significant because that file naming scheme has an official specification. Other commands like bdist_egg, bdist_dumb and bdist_wininst aren't as strict about the expected file names, although it would be good to define a suggested optional build numbering convention at least for bdist_egg, such that easy_install will do the right thing, even if the full source level version number isn't bumped. This is just as relevant for sdists as well. It is quite common to see a broken release due to a missing or wrong MANIFEST.in. Test them prior to uploading them. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014, at 15:21, Donald Stufft don...@stufft.io wrote: On September 29, 2014 at 8:54:26 AM, Wichert Akkerman (wich...@wiggy.net) wrote: On 29 Sep 2014, at 13:58, Nick Coghlan ncogh...@gmail.com wrote: Right, this is my perspective as well. The point that the wheel format already includes a build ordering field was significant because that file naming scheme has an official specification. Other commands like bdist_egg, bdist_dumb and bdist_wininst aren't as strict about the expected file names, although it would be good to define a suggested optional build numbering convention at least for bdist_egg, such that easy_install will do the right thing, even if the full source level version number isn't bumped. This is just as relevant for sdists as well. It is quite common to see a broken release due to a missing or wrong MANIFEST.in. Test them prior to uploading them. You can make the exact same argument about binary distributions, so I don’t understand what you’re trying to say here? Mistakes are made everywhere - I’m just trying to point out that a packaging error is not unique to binary distributions. Wichert. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On September 29, 2014 at 9:25:37 AM, Wichert Akkerman (wich...@wiggy.net) wrote: On 29 Sep 2014, at 15:21, Donald Stufft don...@stufft.io wrote: On September 29, 2014 at 8:54:26 AM, Wichert Akkerman (wich...@wiggy.net) wrote: On 29 Sep 2014, at 13:58, Nick Coghlan ncogh...@gmail.com wrote: Right, this is my perspective as well. The point that the wheel format already includes a build ordering field was significant because that file naming scheme has an official specification. Other commands like bdist_egg, bdist_dumb and bdist_wininst aren't as strict about the expected file names, although it would be good to define a suggested optional build numbering convention at least for bdist_egg, such that easy_install will do the right thing, even if the full source level version number isn't bumped. This is just as relevant for sdists as well. It is quite common to see a broken release due to a missing or wrong MANIFEST.in. Test them prior to uploading them. You can make the exact same argument about binary distributions, so I don’t understand what you’re trying to say here? Mistakes are made everywhere - I’m just trying to point out that a packaging error is not unique to binary distributions. Wichert. The difference is that a binary distribution is a produced *from* a source distribution. This means that if a source distribution is incorrect then any binary distributions built from that source distribution needs to be regenerated. This affects the Wheels which are placed on PyPI, Wheels created “downstream” (e.g. a Wheel cache on a user’s machine, a shared wheel builder via devpi + devpi-builder), and it affects other packaging systems such as the OS package tools like apt-get, FreeBSD ports, Homebrew etc. However if a Wheel is incorrect that really only effects direct consumers of that Wheel. It’s not a generally supported to take a binary distribution and repack it into another form, however even if someone was doing so, the “build number” metadata in the Wheel tag signifies that a new version *of that wheel* has been created. Another way of looking at it, is that the version number identifies the source distribution, while the version number, build number, and platform tags identifies the wheel files. This is reflected in their respective filenames and since this proposal is simply boiled down to “there should exist, at all points in time only one set of bytes per uniquely identified file”. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sun, Sep 28, 2014 at 3:31 PM, Donald Stufft donald.stu...@rackspace.com wrote: Hello All! I'd like to discuss the idea of moving PyPI to having immutable files. ... +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 28, 2014, at 07:31 PM, Donald Stufft wrote: I'd like to discuss the idea of moving PyPI to having immutable files. This would mean that once you publish a particular file you can never reupload that file again with different contents. This would still allow deleting the file or reuploading it if the checksums match what was there prior. Although I have abused this in the past, as others have pointed out, because once uploaded I realize there is a bug in the package. There's a certain class of such bugs that prompt a quick re-upload rather than a version rev, such as some display problem on PyPI (because of package metadata), or some follow on packaging bug, such as a missing MANIFEST.in causing Debian package build to fail. Yes, the latter is more easily checked before upload, but sometimes you feel optimistic. ;) This won't make your lives easier, but I'd like to propose some support for embargoed uploads. These would be normal uploads except that they wouldn't be publicly available until a 'publish' button were pushed. Such embargoed uploads wouldn't be subject to the checksum limitation, and we'd have to figure out exactly how such packages would be available (certainly to a logged in owner of the project via the web, but perhaps through an authenticated scriptable interface). Even if you decide against supporting something like this, I'd still be okay with the checksum restriction. You never run out of version numbers. -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Mon, Sep 29, 2014 at 9:36 AM, Barry Warsaw ba...@python.org wrote: On Sep 28, 2014, at 07:31 PM, Donald Stufft wrote: I'd like to discuss the idea of moving PyPI to having immutable files. This would mean that once you publish a particular file you can never reupload that file again with different contents. This would still allow deleting the file or reuploading it if the checksums match what was there prior. Although I have abused this in the past, as others have pointed out, because once uploaded I realize there is a bug in the package. There's a certain class of such bugs that prompt a quick re-upload rather than a version rev, such as some display problem on PyPI (because of package metadata), or some follow on packaging bug, such as a missing MANIFEST.in causing Debian package build to fail. Yes, the latter is more easily checked before upload, but sometimes you feel optimistic. ;) This won't make your lives easier, but I'd like to propose some support for embargoed uploads. These would be normal uploads except that they wouldn't be publicly available until a 'publish' button were pushed. Such embargoed uploads wouldn't be subject to the checksum limitation, and we'd have to figure out exactly how such packages would be available (certainly to a logged in owner of the project via the web, but perhaps through an authenticated scriptable interface). Even if you decide against supporting something like this, I'd still be okay with the checksum restriction. You never run out of version numbers. -Barry That's essentially what I see as the chief use-case for testpypi.python.org. I don't think pypi.python.org needs to support this as well. Simple is better than complex after all :) Cheers, Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On September 29, 2014 at 10:41:07 AM, Ian Cordasco (graffatcolmin...@gmail.com) wrote: On Mon, Sep 29, 2014 at 9:36 AM, Barry Warsaw ba...@python.org wrote: On Sep 28, 2014, at 07:31 PM, Donald Stufft wrote: I'd like to discuss the idea of moving PyPI to having immutable files. This would mean that once you publish a particular file you can never reupload that file again with different contents. This would still allow deleting the file or reuploading it if the checksums match what was there prior. Although I have abused this in the past, as others have pointed out, because once uploaded I realize there is a bug in the package. There's a certain class of such bugs that prompt a quick re-upload rather than a version rev, such as some display problem on PyPI (because of package metadata), or some follow on packaging bug, such as a missing MANIFEST.in causing Debian package build to fail. Yes, the latter is more easily checked before upload, but sometimes you feel optimistic. ;) This won't make your lives easier, but I'd like to propose some support for embargoed uploads. These would be normal uploads except that they wouldn't be publicly available until a 'publish' button were pushed. Such embargoed uploads wouldn't be subject to the checksum limitation, and we'd have to figure out exactly how such packages would be available (certainly to a logged in owner of the project via the web, but perhaps through an authenticated scriptable interface). Even if you decide against supporting something like this, I'd still be okay with the checksum restriction. You never run out of version numbers. -Barry That's essentially what I see as the chief use-case for testpypi.python.org. I don't think pypi.python.org needs to support this as well. Simple is better than complex after all :) Cheers, Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig Yea I don’t think PyPI needs anything for this, if someone wants to do it they can use testpypi.python.org, or they can stand up a devpi instance which offers a similar thing plus a lot more for a release process. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
+1 On Sep 29, 2014 7:29 AM, Jim Fulton j...@zope.com wrote: On Sun, Sep 28, 2014 at 3:31 PM, Donald Stufft donald.stu...@rackspace.com wrote: Hello All! I'd like to discuss the idea of moving PyPI to having immutable files. ... +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 29, 2014, at 09:40 AM, Ian Cordasco wrote: That's essentially what I see as the chief use-case for testpypi.python.org. I don't think pypi.python.org needs to support this as well. Simple is better than complex after all :) Can we then make it easy to upload to testpypi via the cli? Maybe it already is or I'm using the wrong tools. (I'm just a dumb `setup.py upload` monkey.) Cheers, -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Mon, Sep 29, 2014 at 10:03 AM, Barry Warsaw ba...@python.org wrote: On Sep 29, 2014, at 09:40 AM, Ian Cordasco wrote: That's essentially what I see as the chief use-case for testpypi.python.org. I don't think pypi.python.org needs to support this as well. Simple is better than complex after all :) Can we then make it easy to upload to testpypi via the cli? Maybe it already is or I'm using the wrong tools. (I'm just a dumb `setup.py upload` monkey.) Cheers, -Barry There are some easy to follow instructions linked from testpypi's homepage: https://wiki.python.org/moin/TestPyPI Did you want more than that? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On September 29, 2014 at 11:04:42 AM, Barry Warsaw (ba...@python.org) wrote: On Sep 29, 2014, at 09:40 AM, Ian Cordasco wrote: That's essentially what I see as the chief use-case for testpypi.python.org. I don't think pypi.python.org needs to support this as well. Simple is better than complex after all :) Can we then make it easy to upload to testpypi via the cli? Maybe it already is or I'm using the wrong tools. (I'm just a dumb `setup.py upload` monkey.) Cheers, -Barry ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig I think you can do it with setup.py upload -r https://testpypi.python.org/, you might have to add it to your ~/.pypirc first though like: https://bpaste.net/show/d3a0edff41b4 and then do setup.py upload -r testpypi. Though you should go change your password and never type setup.py upload again. It is unsafe, use twine instead (twine upload -r testpypi). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 30 Sep 2014 00:43, Donald Stufft don...@stufft.io wrote: Yea I don’t think PyPI needs anything for this, if someone wants to do it they can use testpypi.python.org, or they can stand up a devpi instance which offers a similar thing plus a lot more for a release process. It occurs to me that a devpi quickstart for OpenShift (or another PaaS's) free tier could be useful - if a devpi instance is just for pre-release testing of packages, then the free tier should accommodate it comfortably. Cheers, Nick. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29 Sep 2014 22:09, Wichert Akkerman wich...@wiggy.net wrote: On 29 Sep 2014, at 13:58, Nick Coghlan ncogh...@gmail.com wrote: Right, this is my perspective as well. The point that the wheel format already includes a build ordering field was significant because that file naming scheme has an official specification. Other commands like bdist_egg, bdist_dumb and bdist_wininst aren't as strict about the expected file names, although it would be good to define a suggested optional build numbering convention at least for bdist_egg, such that easy_install will do the right thing, even if the full source level version number isn't bumped. This is just as relevant for sdists as well. It is quite common to see a broken release due to a missing or wrong MANIFEST.in. As Donald noted, if the source package contents change, the public version should really change, with all binary artefacts being updated accordingly. I didn't make that clear though - it was an unstated assumption. Cheers, Nick. Wichert. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On September 29, 2014 at 5:33:38 PM, Nick Coghlan (ncogh...@gmail.com) wrote: On 30 Sep 2014 00:43, Donald Stufft wrote: Yea I don’t think PyPI needs anything for this, if someone wants to do it they can use testpypi.python.org, or they can stand up a devpi instance which offers a similar thing plus a lot more for a release process. It occurs to me that a devpi quickstart for OpenShift (or another PaaS's) free tier could be useful - if a devpi instance is just for pre-release testing of packages, then the free tier should accommodate it comfortably. I'm not familiar with with what is available on OpenShift, however this sounds like something that would be great for Heroku's Deploy button[1]. [1] https://devcenter.heroku.com/articles/heroku-button --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29 September 2014 23:16, Donald Stufft don...@stufft.io wrote: It occurs to me that a devpi quickstart for OpenShift (or another PaaS's) free tier could be useful - if a devpi instance is just for pre-release testing of packages, then the free tier should accommodate it comfortably. I'm not familiar with with what is available on OpenShift, however this sounds like something that would be great for Heroku's Deploy button[1]. [1] https://devcenter.heroku.com/articles/heroku-button I've no experience with either of those services, but something like that sounds awesome! (I've toyed with setting up my own local devpi instance, but never got round to it... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
