Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 17.02.2013 11:11, Nick Coghlan wrote: > FYI > > > -- Forwarded message -- > From: Nick Coghlan > Date: Sun, Feb 17, 2013 at 8:10 PM > Subject: PEP 426 is now the draft spec for distribution metadata 2.0 > To: "DistUtils mailing list\"\"" > > > The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ > > Major changes since the last draft: > > 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the > same major.minor semantics as are defined for wheel versions in PEP > 427 (i.e. if a tool sees a major version number it doesn't recognise, > it should give up rather than trying to guess what to do with it, > while it's OK to process a higher minor version) > > 2. The "Private-Version" field is added, and several examples are > given showing how to use it in conjunction with translated public > versions when a project wants to use a version numbering scheme that > the standard installation tools won't understand. > > 3. The environment markers section more clearly covers the need to > handle parentheses (this was mentioned in the text, but not the > pseudo-grammar), and the fields which support those markers have an > explicit cross-reference to that section of the spec. > > 4. Leading/trailing whitespace and date based versioning are > explicitly excluded from valid public versions > > 5. Version compatibility statistics are provided for this PEP relative > to PEP 386 (the analysis script has been added to the PEP repo if > anyone wants to check my numbers) > > 6. I have reclaimed BDFL-Delegate status (with Guido's approval) > > Since getting wheel support into pip no longer depends on this version > of the metadata spec, I plan to leave it open for comment for another > week, and then accept it next weekend if I don't hear any significant > objections. Overall, I think the meta data for Python packages is getting too complicated. Without a support module in the stdlib implementing the required parsing, evaluation and formatting mechanisms needed to analyze and write the format, I'm -1 on adding yet another format version on top of the stack. At the moment, discussing yet another version update is mostly academic, since not even version 1.2 has been picked up by the tools yet: distutils still writes version 1.1 meta data and doesn't even understand 1.2 meta data. The only tool in wide spread use that understands part of the 1.2 data is setuptools/distribute, but it can only understand the Requires-Dist field of that version of the spec (only because the 1.1 Requires field was deprecated) and interprets a Provides-Extra field which isn't even standard. All other 1.2 fields are ignored. setuptools/distribute still writes 1.1 meta-data. I've never seen environment markers being used or supported in the wild. I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0
On Tuesday, 19 February 2013, M.-A. Lemburg wrote: > > The only tool in wide spread use that understands part of the 1.2 data > is setuptools/distribute, but it can only understand the Requires-Dist > field of that version of the spec (only because the 1.1 Requires field was > deprecated) and interprets a Provides-Extra field which isn't > even standard. All other 1.2 fields are ignored. > setuptools/distribute still writes 1.1 meta-data. > > I've never seen environment markers being used or supported > in the wild. > > I'm not against modernizing the format, but given that version 1.2 > has been out for around 8 years now, without much following, > I think we need to make the implementation bit a requirement > before accepting the PEP. > The wheel project uses metadata 2.0 and environment markers - indeed, the PEP was written to formalise what wheel was implementing (specifically so that pip was happy to incorporate support). Standard library support was hampered by the difficulty of changing Distutils - an issue which may have gone away now. I agree that standard library support would be good, either via Distutils or by incorporating distlib, but I don't think it is essential for acceptance. Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: > On 17.02.2013 11:11, Nick Coghlan wrote: > I'm not against modernizing the format, but given that version 1.2 > has been out for around 8 years now, without much following, > I think we need to make the implementation bit a requirement > before accepting the PEP. It is being implemented in distlib, and the (short!) appendix to the PEP itself shows how to read the format using the standard library's email module. v2.0 is designed to fix many of the issues that prevented the adoption of v1.2, including tweaks to the standardised version scheme and the addition of a formal extension mechanism to avoid the ad hoc extensions that occurred with earlier metadata versions. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0
Le Tue, 19 Feb 2013 10:24:15 +, Paul Moore a écrit : > On Tuesday, 19 February 2013, M.-A. Lemburg wrote: > > > > > The only tool in wide spread use that understands part of the 1.2 > > data is setuptools/distribute, but it can only understand the > > Requires-Dist field of that version of the spec (only because the > > 1.1 Requires field was deprecated) and interprets a Provides-Extra > > field which isn't even standard. All other 1.2 fields are ignored. > > setuptools/distribute still writes 1.1 meta-data. > > > > I've never seen environment markers being used or supported > > in the wild. > > > > I'm not against modernizing the format, but given that version 1.2 > > has been out for around 8 years now, without much following, > > I think we need to make the implementation bit a requirement > > before accepting the PEP. > > > > The wheel project uses metadata 2.0 and environment markers - indeed, > the PEP was written to formalise what wheel was implementing > (specifically so that pip was happy to incorporate support). I'm unsure what this means. Does the "wheel project" (I thought wheel was a format? is it also a project?) reimplement its own wheel of metadata parsing and generating? > I agree that standard library support would be good, either via > Distutils or by incorporating distlib, but I don't think it is > essential for acceptance. I think Marc-André is right that the acceptability of the standard should be judged on the availability of (preferably standard) implementations. If the standard isn't implemented, then perhaps it means it's too ambitious / too complex / victim of the second-system effect. So I kind of agree with him the PEP shouldn't be accepted until there's a decent patch pending for the stdlib. That's not a judgement on the PEP's quality. Just an advice to remain cautious, especially if the previous metadata version already enjoys a very poor adoption rate. By the way, pip may be nice, but it's still a third-party tool, and perhaps not even as widespread as distribute / setuptools. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0
Le Tue, 19 Feb 2013 11:36:09 +0100, Antoine Pitrou a écrit : > > I think Marc-André is right that the acceptability of the standard > should be judged on the availability of (preferably standard) > implementations. If the standard isn't implemented, then perhaps it > means it's too ambitious / too complex / victim of the second-system > effect. > > So I kind of agree with him the PEP shouldn't be accepted until > there's a decent patch pending for the stdlib. That's not a judgement > on the PEP's quality. Just an advice to remain cautious, especially > if the previous metadata version already enjoys a very poor adoption > rate. > > By the way, pip may be nice, but it's still a third-party tool, and > perhaps not even as widespread as distribute / setuptools. Well, Nick kind of proved my wrong in his reply to Marc-André :-) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 8:36 PM, Antoine Pitrou wrote: > Le Tue, 19 Feb 2013 10:24:15 +, > Paul Moore a écrit : >> On Tuesday, 19 February 2013, M.-A. Lemburg wrote: >> >> > >> > The only tool in wide spread use that understands part of the 1.2 >> > data is setuptools/distribute, but it can only understand the >> > Requires-Dist field of that version of the spec (only because the >> > 1.1 Requires field was deprecated) and interprets a Provides-Extra >> > field which isn't even standard. All other 1.2 fields are ignored. >> > setuptools/distribute still writes 1.1 meta-data. >> > >> > I've never seen environment markers being used or supported >> > in the wild. >> > >> > I'm not against modernizing the format, but given that version 1.2 >> > has been out for around 8 years now, without much following, >> > I think we need to make the implementation bit a requirement >> > before accepting the PEP. >> > >> >> The wheel project uses metadata 2.0 and environment markers - indeed, >> the PEP was written to formalise what wheel was implementing >> (specifically so that pip was happy to incorporate support). > > I'm unsure what this means. Does the "wheel project" (I thought wheel > was a format? is it also a project?) reimplement its own wheel of > metadata parsing and generating? > >> I agree that standard library support would be good, either via >> Distutils or by incorporating distlib, but I don't think it is >> essential for acceptance. > > I think Marc-André is right that the acceptability of the standard > should be judged on the availability of (preferably standard) > implementations. If the standard isn't implemented, then perhaps it > means it's too ambitious / too complex / victim of the second-system > effect. The most complex part of the PEP is still the version ordering, and that's implemented in the analysis script (initially by Vinay, with one small tweak by me to fix the regular expression for the new validity rules). Everything else was either already implemented for distutils2/distlib, or is a trivial check (like scanning for "a", "b", "c" or "dev" to identify pre-releases and filter them out if none of those characters are present in the corresponding version specifier). Nothing in the PEP is particularly original - almost all of it is either stolen from other build and packaging systems, or is designed to provide a *discoverable* alternative to existing setuptools/distribute/pip practices (specifically, the new extension mechanism means things like entry points can be defined in the standard metadata files without python-dev needing to get involved). Probably the most novel concepts in the PEP are "Obsoleted-By" (so a project can issue one last release to inform users directly that it is dead, and there is an alternative available) and the "Private-Version" field to handle the case where a projects' preferred version scheme requires translation to match the more restricted format understood by the installation tools. Yes, the spec in aggregate is complex, but it's complex because software distribution and dependency analysis are complex tasks. If people want to simplify it, they can try to rebut the rationale for the individual changes that is included at the end of the PEP, but "it's complex" alone is not an argument. > So I kind of agree with him the PEP shouldn't be accepted until there's > a decent patch pending for the stdlib. That's not a judgement on the > PEP's quality. Just an advice to remain cautious, especially if the > previous metadata version already enjoys a very poor adoption rate. There were at least a couple of very good reasons for not adopting metadata 1.2: - it didn't eliminate the need for ad hoc metadata extension files like entry_points.txt, so it didn't solve one of the problems it set out to solve - it's proposed version ordering scheme was incompatible with the prevailing usage on PyPI in a couple of key ways (hence the systematic statistical comparisons in the new PEP - the numbers that are already there shows that the current version of the standard is highly compatible even with *hidden* revisions included in the statistics. I'm currently redoing the analysis to also show public versions only, which should make the proportions look even better, as many of the remaining glitches only affected old pre-releases). The adoption of metadata 1.2 also got caught up in the "distutils must die" rhetoric that meant the distutils2 project got bogged down in trying to eliminate setup.py completely, when the only thing that *really* needs to die is "./setup.py install" (and even that can be acceptable on a development system, we just want to eliminate the need to ever run it on a deployment target). That's the key change in philosophy between this PEP and the previous one: I'm working backwards from what *installers* need, and creating a specification that allows build tools, including distutils, to give it to them. I *don't really care* how those build tools g
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
> I've never seen environment markers being used or supported > in the wild. > > I'm not against modernizing the format, but given that version 1.2 > has been out for around 8 years now, without much following, > I think we need to make the implementation bit a requirement > before accepting the PEP. Elephant in the room? Somethin's been around for so long and hasn't got any sort of mainstream support is *not* a good candidate for a requirement but maybe it's me being naive. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 11:28, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >> On 17.02.2013 11:11, Nick Coghlan wrote: >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. > > It is being implemented in distlib, and the (short!) appendix to the > PEP itself shows how to read the format using the standard library's > email module. Hmm, what is distlib and where does it live ? The PEP only shows how to parse the RFC822-style format used by the metadata. That's not what I was referring to. If a tools wants to support metadata 2.0, it has to support all the complicated stuff as well, i.e. handle the requires fields, the environment markers and version comparisons/sorting. > v2.0 is designed to fix many of the issues that prevented the adoption > of v1.2, including tweaks to the standardised version scheme and the > addition of a formal extension mechanism to avoid the ad hoc > extensions that occurred with earlier metadata versions. Some more notes: * I find it confusing that we now have two version schemes, one defined in PEP 426 (hidden in the middle of the document) and one in PEP 386. It would be better to amend or replace PEP 386, since that's where you look for Python version strings. * PEP 426 doesn't include any mention of the egg distribution format, even though it's the most popular distribution format at the moment. It should at least include the location of the metadata file in eggs (EGG-INFO/PKG-INFO) and egg installations (/EGG-INFO/PKG-INFO). Not sure whether related or not, I also think it would be a good idea to make the metadata file available on PyPI for download (could be sent there when registering the package release). The register command only posts the data as 1.0 metadata, but includes fields from metadata 1.1. PyPI itself only displays part of the data. It would be useful to have the metadata readily available for inspection on PyPI without having to download one of the distribution files first. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
On Feb 19, 2013 6:57 AM, wrote: > > I've never seen environment markers being used or supported > > in the wild. > > > > I'm not against modernizing the format, but given that version 1.2 > > has been out for around 8 years now, without much following, > > I think we need to make the implementation bit a requirement > > before accepting the PEP. > > Elephant in the room? Somethin's been around for so long and hasn't got > any sort of > mainstream support is *not* a good candidate for a requirement but maybe > it's me > being naive. > I think 1.2 was started and then amended over a long period of time to become what it is today. I wanted to edit it again just to add Provides-Extra. It turned out to be more practical to make a new PEP. Nick explains that for complicated reasons the implementation of Metadata 1.2 was not successful. For Metadata 2.0 we had a different strategy: put it into distribute first. You will find that this aids adoption. distlib will provide another implementation that is in many ways interchangeable. We also have three implementations of an installer for the related wheel format, three implementations of the wheel builder, and two surprisingly short implementations of environment markers. Yes, the wheel project has the same name as the wheel format. Metadata 2.0's Provides-Extra field is there to represent the setuptools "extras" feature which is necessary for a very large number of distributions on pypi. For a while the most common environment markers will look like just "extra == 'foo'". So it should not be surprising that distribute is the first tool to recognize Provides-Extra. Previously distribute did not use METADATA / PKG-INFO for dependencies, instead reading from requires.txt If you can get around the fact that we've incorporated most of another PEP 386 into v2.0, rewritten and explained everything to death, I would argue that 2.0 is actually simpler than 1.2. Only four headers are actually required in a valid Metadata 2.0 file and a number of the fields are informational and do not have to be understood by the installer. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > On 19.02.2013 11:28, Nick Coghlan wrote: >> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >>> On 17.02.2013 11:11, Nick Coghlan wrote: >>> I'm not against modernizing the format, but given that version 1.2 >>> has been out for around 8 years now, without much following, >>> I think we need to make the implementation bit a requirement >>> before accepting the PEP. >> >> It is being implemented in distlib, and the (short!) appendix to the >> PEP itself shows how to read the format using the standard library's >> email module. > > Hmm, what is distlib and where does it live ? As part of the post-mortem of packaging's removal from Python 3.3, several subcomponents were identified as stable and useful. distlib is those subcomponents extracted into a separate repository by Vinay Sajip. It will be proposed as the standard library infrastructure for building packaging related tools, while distutils will become purely a build system and have nothing to do with installing software directly (except perhaps on developer machines). > The PEP only shows how to parse the RFC822-style format used by the > metadata. That's not what I was referring to. > > If a tools wants to support metadata 2.0, it has to support all > the complicated stuff as well, i.e. handle the requires fields, > the environment markers and version comparisons/sorting. Which is what distutils2 can be used for now, and what distlib will provide without the unwanted build system infrastructure in distutils2. >> v2.0 is designed to fix many of the issues that prevented the adoption >> of v1.2, including tweaks to the standardised version scheme and the >> addition of a formal extension mechanism to avoid the ad hoc >> extensions that occurred with earlier metadata versions. > > Some more notes: > > * I find it confusing that we now have two version schemes, > one defined in PEP 426 (hidden in the middle of the document) > and one in PEP 386. It would be better to amend or replace > PEP 386, since that's where you look for Python version strings. You can't understand version specifiers without understanding the sort order defined for the version scheme, so documenting them separately is just a recipe for confusion. I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the sorting scheme it defines is broken, and the distutils changes proposed in that PEP are never going to happen. > * PEP 426 doesn't include any mention of the egg distribution format, > even though it's the most popular distribution format at the moment. > It should at least include the location of the metadata file > in eggs (EGG-INFO/PKG-INFO) and egg installations > (/EGG-INFO/PKG-INFO). "Other tools involved in Python distribution may also use this format." The egg format has never been, and never will be, officially endorsed by python-dev. The wheel format is the standard format for binary distribution, and PEP 376 defines the standard location for metadata on installed distributions. > > Not sure whether related or not, I also think it would be a good idea > to make the metadata file available on PyPI for download (could be sent > there when registering the package release). The register command > only posts the data as 1.0 metadata, but includes fields from > metadata 1.1. PyPI itself only displays part of the data. It's not related, but I plan to propose the adoption of TUF (with GPG based signatures) for PyPI's end-to-end security solution, and the conversion of the metadata files to JSON for distribution through TUF's metadata support. (Donald Stufft already wrote PEP 426 <-> JSON bidirectional converter as part of an unrelated experiment) > It would be useful to have the metadata readily available for > inspection on PyPI without having to download one of the > distribution files first. Indeed, which is a large part of why TUF (aka The Update Framework: https://www.updateframework.com/) is such an interesting security solution. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
On Tue, Feb 19, 2013 at 9:57 PM, wrote: >> I've never seen environment markers being used or supported >> in the wild. >> >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. PEP 345 has NOT been an approved specification for 8 years. It was first drafted in 2005, but not Accepted until 2010: http://hg.python.org/peps/rev/7d78653a0e79 (as Daniel noted, what became PEP 426 actually started as *another* round of edits to PEP 345) PEP 345 was a first attempt at defining a formal standard that adequately described an *existing* implementation defined software distribution ecosystem built around the behaviour of setuptools. It failed, because it did *not* achieve that goal, and because there were associated declarations along the lines of "to migrate to this standard, you must completely change the way you build and distribute your Python code" (by conflating the setup.py -> setup.cfg migration of distutils2 with the use of the new metadata standard). The response was an overwhelming meh, as people willing migrated to distribute and pip (based on the de facto setuptools standards), while the distutils2 project never made it to production readiness because it was trying to solve too many problems at once. By contrast, the PEP 376 metadata format created at the same time has been adopted quite well. PEPs 345 and 386 were also accepted at a time when python-dev had neither interest nor extensive experience in packaging systems (or if anyone did have such experience, they weren't volunteering that information). So what's changed since 2010 that makes me think the time is right for a new metadata version? 1. We've recognised that without setuptools and/or distribute on board to generate it, and pip to consume it, any new metadata version is dead in the water. That groundwork has been done (mostly by Daniel as part of his work on the wheel format) 2. We've systematically gone through and resolved the significant discrepancies between the formal versioning scheme and the implementation defined behaviour of setuptools and distribute. This means that standards compliant versions will sort the same way in those tools, even if they are not updated. 3. We've addressed other objections PJ Eby (the author of setuptools) had to the previous version of the format (notably the handling of project obsolescence). 4. Various other missing features from setuptools are now supported directly (such as extras) 5. The defined interaction of version specifiers and pre- and post- releases was not useful. The new version fixes that to be something developers are more likely to want (i.e. they won't get pre-releases unless they explicitly reference one). 6. We've enhanced the format so it can handle extensions in a systematic fashion with clearly defined lines of authority based on PyPI distribution names, rather than adding arbitrarily named files to a distribution's info directory without any clear indication of where to find that specification for the file's meaning. 7. I've updated the metadata specification itself, including the addition of the Private-Version field, to more clearly explain to developers that the public Version in their metadata is primarily a tool for communicating the relative ordering of versions to installation tools, but they're still free to use their own preferred version labels internally, and certainly are not obliged to actually use the full complexity of the standard versioning scheme that the installers understand. It's OK if people don't want to read the detailed rationale provided for each of the major changes as part of the PEP, or if they want to dispute a particular piece of that rationale. But merely going "it's too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" is not a reasonable response. Software distribution is complicated - trying to oversimplify it is one of the reasons setuptools became necessary. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] XML DoS vulnerabilities and exploits in Python
Hello, in August 2012 I found a DoS vulnerability in expat and XML libraries in Python's standard library. Since then I have found several more issues. I have been working on fixes ever since. The README of https://pypi.python.org/pypi/defusedxml contains detailed explanations of my research and all issues Blog post: http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html Hotfixes: https://pypi.python.org/pypi/defusedxml https://pypi.python.org/pypi/defusedexpat Repositories: https://bitbucket.org/tiran/defusedxml https://bitbucket.org/tiran/defusedexpat https://bitbucket.org/tiran/expat CVE (work in progress) CVE-2013-1664 Unrestricted entity expansion induces DoS vulnerabilities in Python XML libraries (XML bomb) CVE-2013-1665 External entity expansion in Python XML libraries inflicts potential security flaws and DoS vulnerabilities Regards, Christian Extract from the documentation: Synopsis The results of an attack on a vulnerable XML library can be fairly dramatic. With just a few hundred Bytes of XML data an attacker can occupy several Gigabytes of memory within seconds. An attacker can also keep CPUs busy for a long time with a small to medium size request. Under some circumstances it is even possible to access local files on your server, to circumvent a firewall, or to abuse services to rebound attacks to third parties. The attacks use and abuse less common features of XML and its parsers. The majority of developers are unacquainted with features such as processing instructions and entity expansions that XML inherited from SGML. At best they know about from experience with HTML but they are not aware that a document type definition (DTD) can generate an HTTP request or load a file from the file system. None of the issues is new. They have been known for a long time. Billion laughs was first reported in 2003. Nevertheless some XML libraries and applications are still vulnerable and even heavy users of XML are surprised by these features. It's hard to say whom to blame for the situation. It's too short sighted to shift all blame on XML parsers and XML libraries for using insecure default settings. After all they properly implement XML specifications. Application developers must not rely that a library is always configured for security and potential harmful data by default. Attack vectors == billion laughs / exponential entity expansion - The Billion Laughs attack -- also known as exponential entity expansion -- uses multiple levels of nested entities. The original example uses 9 levels of 10 expansions in each level to expand the string lol to a string of 3 * 10 9 bytes, hence the name "billion laughs". The resulting string occupies 3 GB (2.79 GiB) of memory; intermediate strings require additional memory. Because most parsers don't cache the intermediate step for every expansion it is repeated over and over again. It increases the CPU load even more. An XML document of just a few hundred bytes can disrupt all services on a machine within seconds. Example XML: ]> &d; quadratic blowup entity expansion - A quadratic blowup attack is similar to a Billion Laughs attack; it abuses entity expansion, too. Instead of nested entities it repeats one large entity with a couple of ten thousand chars over and over again. The attack isn't as efficient as the exponential case but it avoids triggering countermeasures of parsers against heavily nested entities. Some parsers limit the depth and breadth of a single entity but not the total amount of expanded text throughout an entire XML document. A medium-sized XML document with a couple of hundred kilobytes can require a couple of hundred MB to several GB of memory. When the attack is combined with some level of nested expansion an attacker is able to achieve a higher ratio of success. ]> &a;&a;&a;... repeat external entity expansion (remote) -- Entity declarations can contain more than just text for replacement. They can also point to external resources by public identifiers or system identifiers. System identifiers are standard URIs. When the URI is a URL (e.g. a http:// locator) some parsers download the resource from the remote location and embed them into the XML document verbatim. Simple example of a parsed external entity: http://www.python.org/some.xml";> ]> ⅇ The case of parsed external entities works only for valid XML content. The XML standard also supports unparsed external entities with a NData declaration. External entity expansion opens the door to plenty of exploits. An attacker can abuse a vulnerable XML library and application to rebound and forward network requests with the IP address of the server. It highly depends on the parser and the application what kind of exploit is possible. For example: * An attacker can circumvent firewalls and gain acces
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
On 19 February 2013 13:59, Nick Coghlan wrote: > It's OK if people don't want to read the detailed rationale provided > for each of the major changes as part of the PEP, or if they want to > dispute a particular piece of that rationale. But merely going "it's > too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" > is not a reasonable response. Software distribution is complicated - > trying to oversimplify it is one of the reasons setuptools became > necessary. Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426, 390, 376 and 386 are all relevant to one extent or another, and only PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils - setup.cfg) are still in Draft format. The others are all Accepted or Final. And yet PEPs 345, 390 and 386 are unused and likely to remain so. I believe that the only ones that are *actually* of use are 241, 314 and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of Installed Python Distributions) although 376 still has a number of flaws. I'd suggest that these should be marked as Final, and the others as Rejected, so that we have a clear statement of what is actually supported. Making sure that users have the means to write code that *uses* these standards using functionality available in the stdlib is then the next step as you say. It is critical that this is done, because packaging tools are unique in that the barrier to using external dependencies is particularly high for them - for example, pip's reliance on distribute/setuptools is necessary, but has been problematic at times. Paul PS Apologies for using a load of PEP numbers without explanation. Here's a glossary: Metadata 1.0 - PEP 241 Metadata 1.1 - PEP 314 Metadata 1.2 - PEP 345 Metadata 1.3 - PEP 426 Static Metadata for Distutils - PEP 390 Database of Installed Python Distributions - PEP 376 Changing the version comparison module in Distutils - PEP 386 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 2.7 buffer and memoryview documentation (issue# 17145)
http://bugs.python.org/issue17145 I'm curious as to whether or not anyone has reviewed the documentation update I made here. Context: Having both memory views and buffers in 2.7 (as well as the C-level API for each) is confusing. The initial bug report was to implement consistent behavior for objects supporting the new C-level buffer API or to update the docs. As new features are not being added to 2.7 (as pointed out by Stefan Krah), I made an update to the docs that I think made the disparity between the two (and when one should favour the usage of either) a little clearer. But maybe I'm off my rocker ;) Also, are contributor agreements also required for documentation? Thanks, Demian Brecht http://demianbrecht.github.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Built with VS2012 Express for desktop
Hi. I downloaded Python 3.3 source, opened up the solution in VS2012 Express for Desktop and built the "python" subproject using "Release" and "x64" configurations. I now have a "python.exe" in the PCBuild/amd64 subfolder that appears to be working as far as i can see. I have a few questions: a) Is there a test suite that I can run to verify that the build is working fine? b) I now intend to build some extensions (such as NumPy). Not sure if this is the right list, but would I need to modify something in distutils to get it working with VS2012? thanks, Rahul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
On Tue, Feb 19, 2013 at 12:37 PM, rahul garg wrote: > Hi. > > I downloaded Python 3.3 source, opened up the solution in VS2012 Express for > Desktop and built the "python" subproject using "Release" and "x64" > configurations. I now have a "python.exe" in the PCBuild/amd64 subfolder > that appears to be working as far as i can see. > > I have a few questions: > a) Is there a test suite that I can run to verify that the build is working > fine? Last I checked there are some issues, but most of the tests should pass. You would run "PCBuild\python.exe -m test" from the top level of your checkout to see this. It's also covered at http://docs.python.org/devguide/ > b) I now intend to build some extensions (such as NumPy). Not sure if this > is the right list, but would I need to modify something in distutils to get > it working with VS2012? Yes. You'll probably need to point distutils to the correct batch file that sets up a VS2012 build environment. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
Hello. This mailing list is to work on developing Python (adding new features to Python itself and fixing bugs); if you're having problems learning, understanding or using Python, please find another forum. Probably python-list/comp.lang.python mailing list/news group is the best place; there are Python developers who participate in it; you may get a faster, and probably more complete, answer there. See http://www.python.org/community/ for other lists/news groups/fora. Thank you for understanding. On Tue, Feb 19, 2013 at 01:37:15PM -0500, rahul garg wrote: > a) Is there a test suite that I can run to verify that the build is working > fine? http://docs.python.org/3.3/library/test.html Oleg. -- Oleg Broytmanhttp://phdru.name/[email protected] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
On Tue, Feb 19, 2013 at 11:26 AM, Paul Moore wrote: > On 19 February 2013 13:59, Nick Coghlan wrote: > > It's OK if people don't want to read the detailed rationale provided > > for each of the major changes as part of the PEP, or if they want to > > dispute a particular piece of that rationale. But merely going "it's > > too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" > > is not a reasonable response. Software distribution is complicated - > > trying to oversimplify it is one of the reasons setuptools became > > necessary. > > Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426, > 390, 376 and 386 are all relevant to one extent or another, and only > PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils - > setup.cfg) are still in Draft format. The others are all Accepted or > Final. And yet PEPs 345, 390 and 386 are unused and likely to remain > so. > > I believe that the only ones that are *actually* of use are 241, 314 > and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of > Installed Python Distributions) although 376 still has a number of > flaws. I'd suggest that these should be marked as Final, and the > others as Rejected, so that we have a clear statement of what is > actually supported. > > Making sure that users have the means to write code that *uses* these > standards using functionality available in the stdlib is then the next > step as you say. It is critical that this is done, because packaging > tools are unique in that the barrier to using external dependencies is > particularly high for them - for example, pip's reliance on > distribute/setuptools is necessary, but has been problematic at times. > > Paul > > PS Apologies for using a load of PEP numbers without explanation. > Here's a glossary: > > Metadata 1.0 - PEP 241 > Metadata 1.1 - PEP 314 > Metadata 1.2 - PEP 345 > Metadata 1.3 - PEP 426 > Static Metadata for Distutils - PEP 390 > Database of Installed Python Distributions - PEP 376 > Changing the version comparison module in Distutils - PEP 386 > The documentation is bad. The standard library will almost certainly grow code that can interpret these PEPs. Packaging tool authors can handle these specs but no sane person reads PEPs to figure out how to submit a simple library to pypi. No end user cares about metadata versions either or even notices that sdists contain PKG-INFO at their root. You get what setup() produces and that's it. Wheel installers can work without being installed into the target environment at all. This is very hard to understand if you have been using the distutils model (the installed package imports the installer and packages are always rebuilt from source) for more than a decade. This feature does something to remedy the setuptools chicken/egg problem. We have eliminated the egg ;-) Once the installer doesn't need to be installed, repeatedly, in every virtualenv, hundreds of times a day, users could just have a single up-to-date standalone copy of pip invoked as "python pip.zip install x". Additional packages needed only for builds (even setuptools) may be installed automatically by a build system (like pip) when building rather than using packages. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
On 2/19/2013 11:06 AM, Daniel Holth wrote: This feature does something to remedy the setuptools chicken/egg problem. We have eliminated the egg ;-) This is the most artfully crafted comment I've seen on topic on this list for some months! Thanks! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19 February 2013 13:40, Nick Coghlan wrote: >> If a tools wants to support metadata 2.0, it has to support all >> the complicated stuff as well, i.e. handle the requires fields, >> the environment markers and version comparisons/sorting. > > Which is what distutils2 can be used for now, and what distlib will > provide without the unwanted build system infrastructure in > distutils2. One particular item in Metadata 2.0 which needs to be addressed is requirements specifiers, as used in (for example) Requires-Dist. These take forms like "DistName (>= ver)". There are a number of potential issues here: 1. This differs from the pkg_resources format (which omits the parentheses). Having 2 subtly different formats is not a good idea in the long term. At the moment, pkg_resources format is used in pip requirements (on the command line and in requirement files) as well as in setuptools/distribute and possibly elsewhere. 2. There is currently no code that I am aware of that implements this spec. I don't believe distlib does (yet - give Vinay 5 minutes and who knows? :-)), pkg_resources as I said implements a different format, and distutils2, apart from being a big dependency to cover just this point, only matches the version (not the project name) and presumably does so using the distutils2 version ordering (which is incompatible with Metadata 2.0). 3. The format is fiddly to parse and process - nobody is likely to implement it for themselves without a library that does it (especially not when using pkg_resources is so easy). The PEP needs a rationale as to why the pkg_resources format wasn't used, and should suggest a migration path for projects which currently use the pkg_resources style (such as pip's requirements files). Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tuesday, February 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan (mailto:[email protected])> wrote: > > > If a tools wants to support metadata 2.0, it has to support all > > > the complicated stuff as well, i.e. handle the requires fields, > > > the environment markers and version comparisons/sorting. > > > > > > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > > > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > > It should be alongside the version stuff, atleast it was in distutils2. > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul > ___ > Python-Dev mailing list > [email protected] (mailto:[email protected]) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19 February 2013 20:36, Donald Stufft wrote: > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > > It should be alongside the version stuff, atleast it was in distutils2. Yes, sorry - distlib does include code for this: https://distlib.readthedocs.org/en/latest/tutorial.html#matching-versions-against-constraints I was looking for the term "requirement" which is not used, and missed it. (Either that or Vinay's been borrowing the time machine keys again!) Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan wrote: > >> If a tools wants to support metadata 2.0, it has to support all > >> the complicated stuff as well, i.e. handle the requires fields, > >> the environment markers and version comparisons/sorting. > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul We did add full support for Metadata 2.0 Requires-Dist in pkg_resources. The fiddling happens here: https://bitbucket.org/tarek/distribute/src/tip/pkg_resources.py?at=default#cl-2469 The code removes the parenthesis and adds == when no operator was specified to convert us back to the pkg_resources format. The rest of pkg_resources carries on as before. Wheel installs would not pull in dependencies otherwise. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 2:28 AM, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >> On 17.02.2013 11:11, Nick Coghlan wrote: >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. > > It is being implemented in distlib, and the (short!) appendix to the > PEP itself shows how to read the format using the standard library's > email module. Maybe this is already stated somewhere, but is there a plan for when distlib will be brought into the repository? Is there a reason not to do it now? It seems it would have more visibility that way (e.g. people could see it as part of the development version of the online docs and it would be in check-ins as are PEP edits), and its status relative to Python would be clearer. --Chris > > v2.0 is designed to fix many of the issues that prevented the adoption > of v1.2, including tweaks to the standardised version scheme and the > addition of a formal extension mechanism to avoid the ad hoc > extensions that occurred with earlier metadata versions. > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 14:40, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> * PEP 426 doesn't include any mention of the egg distribution format, >> even though it's the most popular distribution format at the moment. >> It should at least include the location of the metadata file >> in eggs (EGG-INFO/PKG-INFO) and egg installations >> (/EGG-INFO/PKG-INFO). > > "Other tools involved in Python distribution may also use this format." > > The egg format has never been, and never will be, officially endorsed > by python-dev. The wheel format is the standard format for binary > distribution, and PEP 376 defines the standard location for metadata > on installed distributions. Oh, come on, Nick, that's just silly. setuptools was included in stdlib for a short while, so the above is simply wrong. Eggs are the most widely used binary distribution format for Python package on PyPI: # wc *files.csv 25585 25598 1431013 2013-02-19-egg-files.csv 46194640 236694 2013-02-19-exe-files.csv 254 255 13402 2013-02-19-msi-files.csv 104691 104853 5251962 2013-02-19-tar-gz-files.csv 24 241221 2013-02-19-whl-files.csv 17937 18022 905913 2013-02-19-zip-files.csv 153110 153392 7840205 total (based on todays PyPI stats) It doesn't really help ignoring realities... and I'm saying that as one of the core devs who got setuptools kicked out of the stdlib again. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > On 19.02.2013 14:40, Nick Coghlan wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > >> * PEP 426 doesn't include any mention of the egg distribution format, > >> even though it's the most popular distribution format at the moment. > >> It should at least include the location of the metadata file > >> in eggs (EGG-INFO/PKG-INFO) and egg installations > >> (/EGG-INFO/PKG-INFO). > > > > "Other tools involved in Python distribution may also use this format." > > > > The egg format has never been, and never will be, officially endorsed > > by python-dev. The wheel format is the standard format for binary > > distribution, and PEP 376 defines the standard location for metadata > > on installed distributions. > > Oh, come on, Nick, that's just silly. setuptools was included in stdlib > for a short while, so the above is simply wrong. Eggs are the most > widely used binary distribution format for Python package on PyPI: > > # wc *files.csv > 25585 25598 1431013 2013-02-19-egg-files.csv >46194640 236694 2013-02-19-exe-files.csv > 254 255 13402 2013-02-19-msi-files.csv > 104691 104853 5251962 2013-02-19-tar-gz-files.csv > 24 241221 2013-02-19-whl-files.csv > 17937 18022 905913 2013-02-19-zip-files.csv > 153110 153392 7840205 total > > (based on todays PyPI stats) > > It doesn't really help ignoring realities... and I'm saying > that as one of the core devs who got setuptools kicked out of > the stdlib again. > > -- > Marc-Andre Lemburg > eGenix.com > The wheel philosophy is that it should be supported by both python-dev and setuptools and that you should feel happy about using setuptools if you like it whether or not python-dev (currently) endorses that. If you are using setuptools (distribute's pkg_resources) then you can use both at the same time. Distribute, distutils and setuptools' problems have not been well understood which I think is why there has been a need to discredit setuptools by calling it non-standard. It is the defacto standard. If your packages have dependencies there is no other choice. Wheel tries to solve the real problem by allowing you to build a package with setuptools while giving the end-user the choice of installing setuptools or not. Of course eggs are the most popular right now. The wheel format is very egg-like while avoiding some of egg's problems. See the comparison in the PEP or read the story on wheel's rtfd. The wheel project includes tools to losslessly convert eggs or bdist_wininst to wheel. I am confident distlib can thrive outside of the standard library! Why the rush to kill it before its prime? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 14:40, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> On 19.02.2013 11:28, Nick Coghlan wrote: >>> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: On 17.02.2013 11:11, Nick Coghlan wrote: I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP. >>> >>> It is being implemented in distlib, and the (short!) appendix to the >>> PEP itself shows how to read the format using the standard library's >>> email module. >> >> Hmm, what is distlib and where does it live ? > > As part of the post-mortem of packaging's removal from Python 3.3, > several subcomponents were identified as stable and useful. distlib is > those subcomponents extracted into a separate repository by Vinay > Sajip. > > It will be proposed as the standard library infrastructure for > building packaging related tools, while distutils will become purely a > build system and have nothing to do with installing software directly > (except perhaps on developer machines). Shouldn't those details be mentioned in the PEP ? >> The PEP only shows how to parse the RFC822-style format used by the >> metadata. That's not what I was referring to. >> >> If a tools wants to support metadata 2.0, it has to support all >> the complicated stuff as well, i.e. handle the requires fields, >> the environment markers and version comparisons/sorting. > > Which is what distutils2 can be used for now, and what distlib will > provide without the unwanted build system infrastructure in > distutils2. > >>> v2.0 is designed to fix many of the issues that prevented the adoption >>> of v1.2, including tweaks to the standardised version scheme and the >>> addition of a formal extension mechanism to avoid the ad hoc >>> extensions that occurred with earlier metadata versions. >> >> Some more notes: >> >> * I find it confusing that we now have two version schemes, >> one defined in PEP 426 (hidden in the middle of the document) >> and one in PEP 386. It would be better to amend or replace >> PEP 386, since that's where you look for Python version strings. > > You can't understand version specifiers without understanding the sort > order defined for the version scheme, so documenting them separately > is just a recipe for confusion. PEP 386 defines both. The point here is that the version scheme goes far beyond the metadata format and is complicated enough to warrant a separate PEP. > I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the > sorting scheme it defines is broken, and the distutils changes > proposed in that PEP are never going to happen. Hmm, Tarek is the author, so only he can withdraw the PEP, AFAIK. >> * PEP 426 doesn't include any mention of the egg distribution format, >> even though it's the most popular distribution format at the moment. >> It should at least include the location of the metadata file >> in eggs (EGG-INFO/PKG-INFO) and egg installations >> (/EGG-INFO/PKG-INFO). > > "Other tools involved in Python distribution may also use this format." > > The egg format has never been, and never will be, officially endorsed > by python-dev. The wheel format is the standard format for binary > distribution, and PEP 376 defines the standard location for metadata > on installed distributions. See my other reply. >> >> Not sure whether related or not, I also think it would be a good idea >> to make the metadata file available on PyPI for download (could be sent >> there when registering the package release). The register command >> only posts the data as 1.0 metadata, but includes fields from >> metadata 1.1. PyPI itself only displays part of the data. > > It's not related, but I plan to propose the adoption of TUF (with GPG > based signatures) for PyPI's end-to-end security solution, and the > conversion of the metadata files to JSON for distribution through > TUF's metadata support. (Donald Stufft already wrote PEP 426 <-> JSON > bidirectional converter as part of an unrelated experiment) Why convert the metadata format you are defining in PEP 426 to yet another format when it can be uploaded as file straight to PyPI ? TUF doesn't have anything to do with that, agreed ;-) >> It would be useful to have the metadata readily available for >> inspection on PyPI without having to download one of the >> distribution files first. > > Indeed, which is a large part of why TUF (aka The Update Framework: > https://www.updateframework.com/) is such an interesting security > solution. The suggestion to have the metadata available on PyPI doesn't have anything to do with security. It's about being able to determine compatibility and select the right distribution file for download. The metadata also helps in creating dependency graphs, which are useful for a lot of things. -- Marc-Andre Lemburg e
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 23:01, Daniel Holth wrote: > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > >> On 19.02.2013 14:40, Nick Coghlan wrote: >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: * PEP 426 doesn't include any mention of the egg distribution format, even though it's the most popular distribution format at the moment. It should at least include the location of the metadata file in eggs (EGG-INFO/PKG-INFO) and egg installations (/EGG-INFO/PKG-INFO). >>> >>> "Other tools involved in Python distribution may also use this format." >>> >>> The egg format has never been, and never will be, officially endorsed >>> by python-dev. The wheel format is the standard format for binary >>> distribution, and PEP 376 defines the standard location for metadata >>> on installed distributions. >> >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib >> for a short while, so the above is simply wrong. Eggs are the most >> widely used binary distribution format for Python package on PyPI: >> >> # wc *files.csv >> 25585 25598 1431013 2013-02-19-egg-files.csv >>46194640 236694 2013-02-19-exe-files.csv >> 254 255 13402 2013-02-19-msi-files.csv >> 104691 104853 5251962 2013-02-19-tar-gz-files.csv >> 24 241221 2013-02-19-whl-files.csv >> 17937 18022 905913 2013-02-19-zip-files.csv >> 153110 153392 7840205 total >> >> (based on todays PyPI stats) >> >> It doesn't really help ignoring realities... and I'm saying >> that as one of the core devs who got setuptools kicked out of >> the stdlib again. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> > > The wheel philosophy is that it should be supported by both python-dev and > setuptools and that you should feel happy about using setuptools if you > like it whether or not python-dev (currently) endorses that. If you are > using setuptools (distribute's pkg_resources) then you can use both at the > same time. > > Distribute, distutils and setuptools' problems have not been well > understood which I think is why there has been a need to discredit > setuptools by calling it non-standard. It is the defacto standard. If your > packages have dependencies there is no other choice. Wheel tries to solve > the real problem by allowing you to build a package with setuptools while > giving the end-user the choice of installing setuptools or not. > > Of course eggs are the most popular right now. The wheel format is very > egg-like while avoiding some of egg's problems. See the comparison in the > PEP or read the story on wheel's rtfd. The wheel project includes tools to > losslessly convert eggs or bdist_wininst to wheel. That's all fine, but it doesn't explain the refusal to add the documentation of the location of the PKG-INFO file in eggs ? > I am confident distlib can thrive outside of the standard library! Why the > rush to kill it before its prime? Who's trying to kill distlib ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg wrote: > On 19.02.2013 23:01, Daniel Holth wrote: > > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > > > >> On 19.02.2013 14:40, Nick Coghlan wrote: > >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg > wrote: > * PEP 426 doesn't include any mention of the egg distribution format, > even though it's the most popular distribution format at the moment. > It should at least include the location of the metadata file > in eggs (EGG-INFO/PKG-INFO) and egg installations > (/EGG-INFO/PKG-INFO). > >>> > >>> "Other tools involved in Python distribution may also use this format." > >>> > >>> The egg format has never been, and never will be, officially endorsed > >>> by python-dev. The wheel format is the standard format for binary > >>> distribution, and PEP 376 defines the standard location for metadata > >>> on installed distributions. > >> > >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib > >> for a short while, so the above is simply wrong. Eggs are the most > >> widely used binary distribution format for Python package on PyPI: > >> > >> # wc *files.csv > >> 25585 25598 1431013 2013-02-19-egg-files.csv > >>46194640 236694 2013-02-19-exe-files.csv > >> 254 255 13402 2013-02-19-msi-files.csv > >> 104691 104853 5251962 2013-02-19-tar-gz-files.csv > >> 24 241221 2013-02-19-whl-files.csv > >> 17937 18022 905913 2013-02-19-zip-files.csv > >> 153110 153392 7840205 total > >> > >> (based on todays PyPI stats) > >> > >> It doesn't really help ignoring realities... and I'm saying > >> that as one of the core devs who got setuptools kicked out of > >> the stdlib again. > >> > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > > > > The wheel philosophy is that it should be supported by both python-dev > and > > setuptools and that you should feel happy about using setuptools if you > > like it whether or not python-dev (currently) endorses that. If you are > > using setuptools (distribute's pkg_resources) then you can use both at > the > > same time. > > > > Distribute, distutils and setuptools' problems have not been well > > understood which I think is why there has been a need to discredit > > setuptools by calling it non-standard. It is the defacto standard. If > your > > packages have dependencies there is no other choice. Wheel tries to solve > > the real problem by allowing you to build a package with setuptools while > > giving the end-user the choice of installing setuptools or not. > > > > Of course eggs are the most popular right now. The wheel format is very > > egg-like while avoiding some of egg's problems. See the comparison in the > > PEP or read the story on wheel's rtfd. The wheel project includes tools > to > > losslessly convert eggs or bdist_wininst to wheel. > > That's all fine, but it doesn't explain the refusal to add the > documentation of the location of the PKG-INFO file in eggs ? It would just be a sentence, I wouldn't have a problem with it but I also don't see why it would be necessary. Even setuptools doesn't touch the file usually. Right now distribute's pkg_resources currently only understands Requires-Dist if it is inside a .dist-info directory. Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling to imagine a world where that is more visible than something on bitbucket. Half the comments have been about putting something in stdlib right away, something that I do not understand at all as a voracious PyPI or https://crate.io/ user. You should secure both the metadata and the packages but it doesn't have anything to do with the serialization format. Wheel is designed so that the metadata is at the end of the .zip and can be retrieved with an HTTP partial request. The feature may or may not be useful. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tuesday, February 19, 2013 at 6:16 PM, Daniel Holth wrote: > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling > to imagine a world where that is more visible than something on bitbucket. > Half the comments have been about putting something in stdlib right away, > something that I do not understand at all as a voracious PyPI or > https://crate.io/ user. Let's not add anything to the stdlib till it has real world usage. Doing otherwise is putting the cart before the horse. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft wrote: > Let's not add anything to the stdlib till it has real world usage. Doing > otherwise is putting the cart before the horse. I'd posit that anything successful will no longer need to be added to the standard library, to boot. Packaging hasn't done well there. I'd rather see a successful packaging story develop than bundle it into the standard library. The later just isn't that interesting any more. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20 February 2013 00:54, Fred Drake wrote: > I'd posit that anything successful will no longer need to be added to > the standard library, to boot. Packaging hasn't done well there. distlib may be the exception, though. Packaging tools are somewhat unique because of the chicken and egg issue involved in having a packaging tool with external dependencies - who installs your dependencies for you? So enabling technology (library code to perform packaging-related tasks, particularly in support of standardised formats) could be better available from the stdlib. > I'd rather see a successful packaging story develop than bundle it into the > standard library. The later just isn't that interesting any more. Bundling too early is a bad idea though. distlib is developing fast and to do so it needs (1) a development cycle independent of python's and (2) compatibility and ease of use with earlier versions of Python (the latter is also critical for adoption in place of custom code in packaging tools). Aiming for an accelerated level of development targeting inclusion in Python 3.4 is plausible, though. MAL pointed out that agreeing standards but not offering tools to support them in the stdlib is risky, as people have no incentive to adopt those standards. We've got 6 months or more until 3.4 feature freeze, let's not make any decision too soon, though. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20/02/13 11:54, Fred Drake wrote: On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft wrote: Let's not add anything to the stdlib till it has real world usage. Doing otherwise is putting the cart before the horse. I'd posit that anything successful will no longer need to be added to the standard library, to boot. Packaging hasn't done well there. I'd rather see a successful packaging story develop than bundle it into the standard library. The later just isn't that interesting any more. I keep hearing people say that the stdlib is not important, but I don't think that is true. There are lots of people who have problems with anything not in the standard library. - Beginners often have difficulty (due to inexperience, lack of confidence or knowledge) in *finding*, let alone installing and using, packages that aren't in the standard library. - To people in the Linux world, adding anything outside of your distro's packaging system is a nuisance. No matter how easy your packaging library makes it, you now have two sorts of packages: first-class packages that your distro will automatically update for you, and second-class ones that aren't. - People working in restrictive corporate systems often have to jump through flaming hoops before installing software. Packages in the stdlib are a no-brainer. Anything outside the stdlib has additional barriers to use, even if installing them is as simple as "some-package-manager install spam.py". For the avoidance of doubt, this is *not* a veiled request for "everything" to be in the stdlib, since that is impractical and stupid, just a reminder that the stdlib is still important and that no matter how easy packaging becomes, it will never be as easy as having something already there. -- Steven ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/19/2013 09:37 PM, Paul Moore wrote: > On 20 February 2013 00:54, Fred Drake wrote: >> I'd posit that anything successful will no longer need to be added >> to the standard library, to boot. Packaging hasn't done well >> there. > > distlib may be the exception, though. Packaging tools are somewhat > unique because of the chicken and egg issue involved in having a > packaging tool with external dependencies - who installs your > dependencies for you? So enabling technology (library code to perform > packaging-related tasks, particularly in support of standardised > formats) could be better available from the stdlib. The big blocker there is that users can't rely on having it in the stdlib until after they drop Python < 3.4 (assuming accelearated absorption) or even 3.5. Tres. - -- === Tres Seaver +1 540-429-0999 [email protected] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEkTAsACgkQ+gerLs4ltQ6VBgCePncI4cX7a8NEN6lj98CVBdAE UTUAnA+6zo8Zjmp6T4n0uL804PnHHvZ8 =OT8w -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman wrote: > Hello. > >This mailing list is to work on developing Python (adding new > features to Python itself and fixing bugs); if you're having problems > learning, understanding or using Python, please find another forum. > Probably python-list/comp.lang.python mailing list/news group is the > best place; there are Python developers who participate in it; you may > get a faster, and probably more complete, answer there. See > http://www.python.org/community/ for other lists/news groups/fora. Thank > you for understanding. > > Oleg, lately I have the feeling you're getting too automatic with this template response. The OP didn't ask how to reverse a list in Python, he was trying to build Python itself (on Windows, no less), which suggests that he may be trying to work *on* Python. Or maybe he isn't, but let's grant him the benefit of the doubt. If python-dev was overwhelmed with such posts, then perhaps we'd have to think about solving the problem; but there is no problem. Being kind to newbies is an important part of meaningful participation in an open source project. Eli ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7 buffer and memoryview documentation (issue# 17145)
On 2/19/2013 12:33 PM, Demian Brecht wrote: Comment on patch on issue. Also, are contributor agreements also required for documentation? I believe so, especially for substantial patches like yours. Just submit one and be done with it. You probably should choose the Academic License. http://www.python.org/psf/contrib/ -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Built with VS2012 Express for desktop
On Tue, 19 Feb 2013 20:37:36 -0800 Eli Bendersky wrote: > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman wrote: > > > Hello. > > > >This mailing list is to work on developing Python (adding new > > features to Python itself and fixing bugs); if you're having problems > > learning, understanding or using Python, please find another forum. > > Probably python-list/comp.lang.python mailing list/news group is the > > best place; there are Python developers who participate in it; you may > > get a faster, and probably more complete, answer there. See > > http://www.python.org/community/ for other lists/news groups/fora. Thank > > you for understanding. > > > > > Oleg, lately I have the feeling you're getting too automatic with this > template response. +1. This isn't the python-robots mailing-list. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth wrote: > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was > struggling to imagine a world where that is more visible than something on > bitbucket. I meant that bringing distlib into http://hg.python.org/cpython/ would give it more visibility to core devs and others that already keep an eye on python-checkins (the mailing list). And I think seeing the Sphinx-processed docs integrated and cross-referenced with http://docs.python.org/dev/ will help people understand better what has been done and how it fits in with the rest of CPython -- which I think would be useful to the community. It may also encourage involvement (e.g. by being part of the main tracker). In asking about the "plan" for doing this, I was thinking of the following remark by Nick: On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> >> Hmm, what is distlib and where does it live ? > > As part of the post-mortem of packaging's removal from Python 3.3, > several subcomponents were identified as stable and useful. distlib is > those subcomponents extracted into a separate repository by Vinay > Sajip. > > It will be proposed as the standard library infrastructure for > building packaging related tools, while distutils will become purely a > build system and have nothing to do with installing software directly > (except perhaps on developer machines). My question was basically whether there was a tentative plan for when it (or completed parts of it) will be proposed (e.g. when a certain amount of functionality is completed, etc). It's better not to do this at the last minute if 3.4 is the plan (as I think was attempted with packaging but for 3.3). On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano wrote: > > I keep hearing people say that the stdlib is not important, but I don't > think > that is true. There are lots of people who have problems with anything not > in > the standard library. > > - Beginners often have difficulty (due to inexperience, lack of confidence > or > knowledge) in *finding*, let alone installing and using, packages that > aren't > in the standard library. > > - To people in the Linux world, adding anything outside of your distro's > packaging system is a nuisance. No matter how easy your packaging library > makes it, you now have two sorts of packages: first-class packages that > your distro will automatically update for you, and second-class ones that > aren't. > > - People working in restrictive corporate systems often have to jump through > flaming hoops before installing software. I would also add that for people new to writing Python modules and that want to distribute them, it's hard to evaluate what they are "supposed" to use (distutils, setuptools, distribute, bento, etc). Just a day or two ago, this exact question was asked on the Distutils mailing list with subject "Confusion of a hobby programmer." Code not being in the standard library creates an extra mental hurdle to overcome. --Chris ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote: > On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth (mailto:[email protected])> wrote: > > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was > > struggling to imagine a world where that is more visible than something on > > bitbucket. > > > > > I meant that bringing distlib into http://hg.python.org/cpython/ would > give it more visibility to core devs and others that already keep an > eye on python-checkins (the mailing list). And I think seeing the > Sphinx-processed docs integrated and cross-referenced with > http://docs.python.org/dev/ will help people understand better what > has been done and how it fits in with the rest of CPython -- which I > think would be useful to the community. It may also encourage > involvement (e.g. by being part of the main tracker). > > On the other hand it makes contributing to it more annoying since it does not have pull requests, unless it was just a mirror. > In asking about the "plan" for doing this, I was thinking of the > following remark by Nick: > > On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan (mailto:[email protected])> wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg > (mailto:[email protected])> wrote: > > > > > > Hmm, what is distlib and where does it live ? > > > > As part of the post-mortem of packaging's removal from Python 3.3, > > several subcomponents were identified as stable and useful. distlib is > > those subcomponents extracted into a separate repository by Vinay > > Sajip. > > > > It will be proposed as the standard library infrastructure for > > building packaging related tools, while distutils will become purely a > > build system and have nothing to do with installing software directly > > (except perhaps on developer machines). > > > > > My question was basically whether there was a tentative plan for when > it (or completed parts of it) will be proposed (e.g. when a certain > amount of functionality is completed, etc). It's better not to do > this at the last minute if 3.4 is the plan (as I think was attempted > with packaging but for 3.3). > > On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano (mailto:[email protected])> wrote: > > > > I keep hearing people say that the stdlib is not important, but I don't > > think > > that is true. There are lots of people who have problems with anything not > > in > > the standard library. > > > > - Beginners often have difficulty (due to inexperience, lack of confidence > > or > > knowledge) in *finding*, let alone installing and using, packages that > > aren't > > in the standard library. > > > > - To people in the Linux world, adding anything outside of your distro's > > packaging system is a nuisance. No matter how easy your packaging library > > makes it, you now have two sorts of packages: first-class packages that > > your distro will automatically update for you, and second-class ones that > > aren't. > > > > - People working in restrictive corporate systems often have to jump through > > flaming hoops before installing software. > > > > > I would also add that for people new to writing Python modules and > that want to distribute them, it's hard to evaluate what they are > "supposed" to use (distutils, setuptools, distribute, bento, etc). > Just a day or two ago, this exact question was asked on the Distutils > mailing list with subject "Confusion of a hobby programmer." Code not > being in the standard library creates an extra mental hurdle to > overcome. > > I agree that eventually the stdlib needs standard tooling to work with the future (™) but until that future is in use adding it to the stdlib feels premature to me. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
