Re: [Distutils] Specific packaging goals and a tentative timeline
On 19 July 2013 14:06, Nick Coghlan wrote: > Already done or very close to done (Yay!): > > * improved PyPI SSL support > * setuptools/distribute merger > * easy_install SSL verification > * setuptools support for additional hashes beyond md5 > * pip 1.4 release with SSL verification and initial wheel support (soon!) Marcus pointed out that should be: * pip 1.3 release with SSL verification support * pip 1.4 release with initial wheel support (soon!) I also missed out: * PyPI relocation to OSU/OSL * PyPI CDN * Massive reduction in external link scraping (PEP 438) Good things have already been done, more good things are coming, we just need to pick a sensible order and time frame to avoid burning anyone out :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 19, 2013, at 12:39 AM, Donald Stufft wrote: > > On Jul 19, 2013, at 12:23 AM, Nick Coghlan wrote: > >> I like the idea of switching to zc.buildout style entry points - it >> makes it easier to get pip to a point where "no setuptools" means "can >> only install from wheel files" rather than "can't install anything" >> (that way pip can install setuptools from a wheel if it needs to build >> something else from source). > > I plan on making pip bundle setuptools regardless. > > To underline how important that is, it's been discovered (though we are still > working out _why_) that pip 1.3.1 on python 3.x+ is broken with setuptools > 0.7+. Historically we haven't tested old versions of pip against new versions > of setuptools (and with how quickly setuptools is releasing now a days that > matrix is going to become very big very fast). > > Bundling setuptools makes things way more stable and alleviates a lot of long > term support headaches. > > Also just to be specific entry points don't require setuptools, they require > pkg_resources which currently is installed as part of setuptools but can > likely be split out. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Just to expand a bit here. I think the only reason this worked at all historically is because setuptools hadn't changed much in the last few years so there wasn't much chance for regression. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 19, 2013, at 12:23 AM, Nick Coghlan wrote: > I like the idea of switching to zc.buildout style entry points - it > makes it easier to get pip to a point where "no setuptools" means "can > only install from wheel files" rather than "can't install anything" > (that way pip can install setuptools from a wheel if it needs to build > something else from source). I plan on making pip bundle setuptools regardless. To underline how important that is, it's been discovered (though we are still working out _why_) that pip 1.3.1 on python 3.x+ is broken with setuptools 0.7+. Historically we haven't tested old versions of pip against new versions of setuptools (and with how quickly setuptools is releasing now a days that matrix is going to become very big very fast). Bundling setuptools makes things way more stable and alleviates a lot of long term support headaches. Also just to be specific entry points don't require setuptools, they require pkg_resources which currently is installed as part of setuptools but can likely be split out. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] "ImportError: No module named setuptools" (when using "pip install --upgrade")
If you're getting "ImportError: No module named setuptools" when using "pip install --upgrade", see here for an explanation and solution: https://github.com/pypa/pip/issues/1064 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Worry about lack of focus
On 19 July 2013 10:39, Noah Kantrowitz wrote: > So we've recently seen a big resurgence in activity on improving Python > packaging. First off, thats good, hopefully thats why we are all here. That > said, I'm becoming worried about a possible lack of focus, and I know I'm not > the only one. Indeed, I realised I had a timeline sketched in my head, but had never actually shared it (and many people seem to see metadata 2.0 becoming relevant to end users *far* earlier than I had in mind - I don't see it as becoming relevant until some time in the middle of next year). Posted now, though (I started working on it this morning and just hit send before seeing this thread). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 19 July 2013 09:37, Vinay Sajip wrote: >> I think the point is that people might be dependent on this functionality and > >> changing it out from underneath them could break their world. > > > I got the point that Daniel made, and my question was about *how* their world > would break, and whether we really need to support multiple versions of > something installed side-by-side, with on-the-fly sys.path manipulation. If > that is a real requirement which should be supported, shouldn't there be a > PEP for it, if it's coming into Python? It's not supported by distutils, and > it has been a point of contention. It's a real requirement - Linux distros need it to work around parallel installation of backwards incompatible libraries in the system Python. Yes, it's an implementation defined feature of pkg_resources (not setuptools per se), but it's one that works well enough even if the error message can be opaque and the configuration can get a little arcane :) > A PEP would allow standardisation of the multiple-versions feature it it's > considered desirable, rather than definition by implementation (which I > understand you're not in favour of, in general). > > If it's not considered desirable and doesn't need support, then we only need > to consider if it's undeclared setuptools dependencies that we're concerned > with, or some other failure mode not yet identified - hence, my questions. I > like to get into specifics :-) I like the idea of switching to zc.buildout style entry points - it makes it easier to get pip to a point where "no setuptools" means "can only install from wheel files" rather than "can't install anything" (that way pip can install setuptools from a wheel if it needs to build something else from source). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Specific packaging goals and a tentative timeline
We have a lot of initiatives going every which way at the moment, so I figured it would be a good idea to get a common perception of what we consider to be the important near term goals and a realistic timeline for improving the packaging ecosystem (in particular, the timing relative to the CPython 3.4 release cycle). One of the big things I'd like us to do is ensure we separate out "urgent" things that are coupled to the 3.4 release cycle (like ensuring in-place upgrades for pip work on Windows) from the "important but not urgent" things where we can take more time (like metadata 2.0) This is kinda long, but if people aren't used to long emails from me by now, they haven't been paying attention ;) (PyPA devs - please forward this to pypa-dev for discussion with the folks that don't frequent distutils-sig) My current impression is that the goals below should be fairly realistic. Already done or very close to done (Yay!): * improved PyPI SSL support * setuptools/distribute merger * easy_install SSL verification * setuptools support for additional hashes beyond md5 * pip 1.4 release with SSL verification and initial wheel support (soon!) Before Python 3.4 feature freeze (currently November 23, 2013) * decide on a bundling or explicit bootstrapping scheme for pip (this still needs a PEP to help clarify the pros and cons of the various alternatives) * get RM & installer builder consensus on that scheme * make any necessary updates to CPython (e.g. possibly adding Lib/getpip.py) * (hopefully) add support for indirect imports (see http://mail.python.org/pipermail/import-sig/2013-July/000645.html for the draft PEP - thanks Eric for taking this from a rough idea in email to a concrete proposal!) Before Python 3.4 first release candidate (currently Jan 18, 2014) * pip 1.5 available (or at least release candidates) * setuptools releases as needed * improved handling of in-place pip upgrades on Windows * improved handling of pip/setuptools/pkg_resources division of responsibility * both pip and setuptools available as cross platform wheel files on PyPI * Key requirement: "pip uninstall setuptools" must be supported & must not fundamentally break pip (but may disable installation from anything other than wheel files) * Highly desirable: possible to install pkg_resources without installing setuptools * Highly desirable: possible to install setuptools without the easy_install script (just the script, having the implementation in the setuptools.commands subpackage is fine) Following Python 3.4 final release (currently Feb 22, 2014) * further proposals target pip 1.6 - decoupled from CPython release cycle * metadata 2.0 (PEP 426/440) * sdist 2.0 and wheel 1.1 * installation database format update * revisit distlib-based pip (assuming 1.5 isn't based on a vendored distlib) * revisit TUF-for-PyPI (that's more likely to be pip 1.7 timeframe, though...) Independent activities & miscellaneous suggestions * maybe suggest "pip install distlib" over pip gaining its own programmatic API? * PEP 8 cleanup (including clarification of what constitutes an internal API) * improved PyPI upload API (Donald's working on this) * getting Warehouse to a point where it can be brought online as "pypi-next.python.org" * TUF-for-PyPI exploration (the TUF folks seems to have this well in hand) * improved local PyPI hosting (especially devpi) Specifically on the "bundle or bootstrap pip" front, I'll note that due to the concerns regarding how bundling pip with the CPython MSI installer may interact with in-place upgrades, I'm leaning back towards explicit bootstrapping, with an option to run the bootstrap as part of the installation process for both CPython 3.4+ and the Python Launcher for Windows. Doing that also gives Linux distros something they can patch in the system Python to direct users towards the appropriate system package manager command. Regardless, we still need the various bundling-or-bootstrap alternatives that aren't covered in PEP 439 extracted from the list archives and turned into a PEP that compares them and suggests a preferred solution. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Worry about lack of focus
On 19 July 2013 13:26, Donald Stufft wrote: > > On Jul 18, 2013, at 8:39 PM, Noah Kantrowitz wrote: > >> So we've recently seen a big resurgence in activity on improving Python >> packaging. First off, thats good, hopefully thats why we are all here. That >> said, I'm becoming worried about a possible lack of focus, and I know I'm >> not the only one. There have been many ideas floated, and many PEPs either >> sketched out, reworked, or are stated to be in planning. I think perhaps we >> should work out some kind of shortlist of what we think can and should be >> accomplished in the short term and just keep a running list of topics that >> need energy but are lower priority. This would reduce the chances of hitting >> the "fix the whole world at once" situation that we have run in to before in >> this attempt, which often results in burnout and frustration all around. >> Just to kick things off here are the rough topics I can think of that I've >> seen discussed recently (ignoring that many of these are dependent on each >> other): >> >> * Including pip with Python 3.4 >> * Bundling setuptools with pip >> * Splitting setuptools and pkg_resources >> * Replacing the executable generation in pip with something new >> * Working out how to let pip upgrade itself on Windows >> * Entrypoints in distutils/the stdlib >> * Executable generation in distlib >> * Signing/vetting of releases >> * General improvements to the wheel format >> * General improvements to package metadata >> >> Apologies for anything I have mis-paraphrased or missed, but that is >> definitely a lot of things to have up in the air. Just want to make sure we >> can get everything done without anyone going crazy(er) and that we keep >> sight of whats going on. I'm not too worried about whats in progress... I am worried about disruption when we rush things - e.g. the current broken state of setuptools+pip. -Rob -- Robert Collins Distinguished Technologist HP Cloud Services ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Worry about lack of focus
On Jul 18, 2013, at 9:47 PM, Robert Collins wrote: > I'm not too worried about whats in progress... > > I am worried about disruption when we rush things - e.g. the current > broken state of setuptools+pip. > > -Rob I'd argue that doing too much at once will lead to rushing things and other brokenness. It spreads people out and provides less eyes on each component and less going back and forth because people just don't have the energy to keep track of everything. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Worry about lack of focus
On Jul 18, 2013, at 8:39 PM, Noah Kantrowitz wrote: > So we've recently seen a big resurgence in activity on improving Python > packaging. First off, thats good, hopefully thats why we are all here. That > said, I'm becoming worried about a possible lack of focus, and I know I'm not > the only one. There have been many ideas floated, and many PEPs either > sketched out, reworked, or are stated to be in planning. I think perhaps we > should work out some kind of shortlist of what we think can and should be > accomplished in the short term and just keep a running list of topics that > need energy but are lower priority. This would reduce the chances of hitting > the "fix the whole world at once" situation that we have run in to before in > this attempt, which often results in burnout and frustration all around. Just > to kick things off here are the rough topics I can think of that I've seen > discussed recently (ignoring that many of these are dependent on each other): > > * Including pip with Python 3.4 > * Bundling setuptools with pip > * Splitting setuptools and pkg_resources > * Replacing the executable generation in pip with something new > * Working out how to let pip upgrade itself on Windows > * Entrypoints in distutils/the stdlib > * Executable generation in distlib > * Signing/vetting of releases > * General improvements to the wheel format > * General improvements to package metadata > > Apologies for anything I have mis-paraphrased or missed, but that is > definitely a lot of things to have up in the air. Just want to make sure we > can get everything done without anyone going crazy(er) and that we keep sight > of whats going on. > > --Noah > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig As my last email said, I completely agree here. We don't just risk burn out but we also risk blessing solutions that haven't been thought through entirely. We also increase the "churn" of packaging making it more difficult for people who *aren't* following along to figure out what they are supposed to do. If we come in and try to advocate them changing huge swathes of their toolchain or how to do things they are going to get frustrated. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Worry about lack of focus
So we've recently seen a big resurgence in activity on improving Python packaging. First off, thats good, hopefully thats why we are all here. That said, I'm becoming worried about a possible lack of focus, and I know I'm not the only one. There have been many ideas floated, and many PEPs either sketched out, reworked, or are stated to be in planning. I think perhaps we should work out some kind of shortlist of what we think can and should be accomplished in the short term and just keep a running list of topics that need energy but are lower priority. This would reduce the chances of hitting the "fix the whole world at once" situation that we have run in to before in this attempt, which often results in burnout and frustration all around. Just to kick things off here are the rough topics I can think of that I've seen discussed recently (ignoring that many of these are dependent on each other): * Including pip with Python 3.4 * Bundling setuptools with pip * Splitting setuptools and pkg_resources * Replacing the executable generation in pip with something new * Working out how to let pip upgrade itself on Windows * Entrypoints in distutils/the stdlib * Executable generation in distlib * Signing/vetting of releases * General improvements to the wheel format * General improvements to package metadata Apologies for anything I have mis-paraphrased or missed, but that is definitely a lot of things to have up in the air. Just want to make sure we can get everything done without anyone going crazy(er) and that we keep sight of whats going on. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> version of them. So, I don't think it's accurate to consider > multi-version installation a totally new feature. (And AFAIK, the > point of contention isn't that setuptools *supports* multi-version > installation, it's that it's the *default* implementation.) That distutils features could be manipulated in some esoteric way doesn't mean that distutils supports multi-version installations - not by design, anyway. It's perfectly fine for setuptools, buildout and other third-party tools to support multi-version installations in whatever way they see fit - I only raised the question of a PEP because multi-version would be a significant new feature if in Python (leaving aside technicalities about whether something "bundled with Python" is "in Python"). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 8:15 PM, Donald Stufft wrote: > > On Jul 18, 2013, at 7:37 PM, Vinay Sajip wrote: > >>> I think the point is that people might be dependent on this functionality >>> and >> >>> changing it out from underneath them could break their world. >> >> >> I got the point that Daniel made, and my question was about *how* their >> world would break, and whether we really need to support multiple versions >> of something installed side-by-side, with on-the-fly sys.path manipulation. >> If that is a real requirement which should be supported, shouldn't there be >> a PEP for it, if it's coming into Python? It's not supported by distutils, >> and it has been a point of contention. >> >> A PEP would allow standardisation of the multiple-versions feature it it's >> considered desirable, rather than definition by implementation (which I >> understand you're not in favour of, in general). >> >> If it's not considered desirable and doesn't need support, then we only need >> to consider if it's undeclared setuptools dependencies that we're concerned >> with, or some other failure mode not yet identified - hence, my questions. I >> like to get into specifics :-) > > Yes I'm against implementation defined features. However this is already the > status quo for this particular implementation. Basically I'm worried we are > trying to fix too much at once. > > One of the major reasons for distutils/packaging failing was it tried to fix > the world in one fell swoop. I see this same pattern starting to happen > here. The problem is each solution has a bunch of corner cases and gotchas > and the more things we try to fix at once the less eyes we'll have on each > individual one and the more rushed the entire toolchain is going to be. > > I think it's *really* important we limit the scope of what we fix at any one > time. Right now we have PEP426, PEP440, PEP439, PEP427, Nick is talking about > an Sdist 2.0 PEP, Daniel just posted another PEP I haven't looked at yet, > this is going to be another PEP. On top of that we have a number of issues > related to those PEPs but not specifically part of those PEPs. > > A lot of things is being done right now and I personally have having trouble > keeping up and keeping things straight. I know i'm not the only one because > I've had a number of participants of these discussions privately tell me that > they aren't sure how I'm keeping up (and i'm struggling to do so). I really > don't want us to ship a bunch of half baked / not entirely thought through > solutions. > > So can we please limit our scope? Let's start by fixing the stuff we have > now, punting on fixing some other problems by using the existing tooling and > then let's come back to the things we've punted once we've closed the loop on > some of these other outstanding things and fix them better. I feel your pain. We might as well allow happy setuptools users to continue using setuptools. I don't care about making a pkg_resources console_scripts handler that does the same thing because we can just use the existing one. The more important contribution is to provide an alternative for people who are not happy setuptools users. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 8:36 PM, Donald Stufft wrote: > > On Jul 18, 2013, at 8:33 PM, Daniel Holth wrote: > > We might as well allow happy setuptools users to continue using > setuptools. I don't care about making a pkg_resources console_scripts > handler that does the same thing because we can just use the existing > one. The more important contribution is to provide an alternative for > people who are not happy setuptools users. > > > I generally agree with this :) I just think that we need to close the loop > on > our current efforts before adding more things into the fray. The only major > change to the eco system we've made so far that has actually *shipped* > to end users is the distribute/setuptools merge and that's causing a lot > of pain to people. > > Soon we'll at least have a pip version with prelim wheel support but I don't > even know if it supports metadata 2.0 at all or not yet? I think there's a > pre-release of wheel that does though? bdist_wheel will produce json metadata that generally conforms to the current PEP but no consumer takes advantage of it just yet. I added the "generator" key to the metadata so it would be easy to throw out outdated or buggy json metadata. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 8:33 PM, Daniel Holth wrote: > We might as well allow happy setuptools users to continue using > setuptools. I don't care about making a pkg_resources console_scripts > handler that does the same thing because we can just use the existing > one. The more important contribution is to provide an alternative for > people who are not happy setuptools users. I generally agree with this :) I just think that we need to close the loop on our current efforts before adding more things into the fray. The only major change to the eco system we've made so far that has actually *shipped* to end users is the distribute/setuptools merge and that's causing a lot of pain to people. Soon we'll at least have a pip version with prelim wheel support but I don't even know if it supports metadata 2.0 at all or not yet? I think there's a pre-release of wheel that does though? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 8:15 PM, Donald Stufft wrote: > So can we please limit our scope? Let's start by fixing the stuff we have > now, punting on fixing some other problems by using the existing tooling and > then let's come back to the things we've punted once we've closed the loop on > some of these other outstanding things and fix them better. Let me just specify though that i'm not stating exactly where that line should be drawn. I just see things heading in this direction and I think we're letting scope creep hit us hard and it will absolutely kill our efforts. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On Tue, Jul 16, 2013 at 8:23 AM, Paul Moore wrote: > > On 16 July 2013 12:42, Oscar Benjamin wrote: >> >> I thought that 64 bit Windows could run 32 bit Windows .exe files >> (although I don't have a way to check this). > > > Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is > because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I > said, I don't know for sure at the moment, but it needs investigating. That's not why they exist; the .exe's don't load the Python DLL, they just invoke python.exe. The existence of separate 32- and 64-bit .exe's is a Distribute innovation, actually; setuptools 0.6 doesn't use them. Instead, setuptools writes a manifest file to tell Windows that it doesn't need privilege escalation or to create a separate console. This meant that only one (32-bit) .exe was needed. I forget what happened with the Distribute approach or why the 64-bit version was kept after the merge; ISTM there was some other use for it, or at least Jason thought so. But DLL loading is not the reason. (Actually, after searching my email, my guess is that there actually *isn't* any need for the 64-bit .exe; ISTM that it was introduced solely as a false fix for the privilege escalation problem, that only fixes it for 64-bit Windows and doesn't help 32-bit.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 7:37 PM, Vinay Sajip wrote: >> I think the point is that people might be dependent on this functionality >> and > >> changing it out from underneath them could break their world. > > > I got the point that Daniel made, and my question was about *how* their world > would break, and whether we really need to support multiple versions of > something installed side-by-side, with on-the-fly sys.path manipulation. If > that is a real requirement which should be supported, shouldn't there be a > PEP for it, if it's coming into Python? It's not supported by distutils, and > it has been a point of contention. > > A PEP would allow standardisation of the multiple-versions feature it it's > considered desirable, rather than definition by implementation (which I > understand you're not in favour of, in general). > > If it's not considered desirable and doesn't need support, then we only need > to consider if it's undeclared setuptools dependencies that we're concerned > with, or some other failure mode not yet identified - hence, my questions. I > like to get into specifics :-) Yes I'm against implementation defined features. However this is already the status quo for this particular implementation. Basically I'm worried we are trying to fix too much at once. One of the major reasons for distutils/packaging failing was it tried to fix the world in one fell swoop. I see this same pattern starting to happen here. The problem is each solution has a bunch of corner cases and gotchas and the more things we try to fix at once the less eyes we'll have on each individual one and the more rushed the entire toolchain is going to be. I think it's *really* important we limit the scope of what we fix at any one time. Right now we have PEP426, PEP440, PEP439, PEP427, Nick is talking about an Sdist 2.0 PEP, Daniel just posted another PEP I haven't looked at yet, this is going to be another PEP. On top of that we have a number of issues related to those PEPs but not specifically part of those PEPs. A lot of things is being done right now and I personally have having trouble keeping up and keeping things straight. I know i'm not the only one because I've had a number of participants of these discussions privately tell me that they aren't sure how I'm keeping up (and i'm struggling to do so). I really don't want us to ship a bunch of half baked / not entirely thought through solutions. So can we please limit our scope? Let's start by fixing the stuff we have now, punting on fixing some other problems by using the existing tooling and then let's come back to the things we've punted once we've closed the loop on some of these other outstanding things and fix them better. > > Regards, > > Vinay Sajip - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 7:09 PM, Vinay Sajip wrote: > PJ Eby telecommunity.com> writes: >> While other strategies are definitely possible, distlib's approach is >> not backward-compatible, as it means installing new versions of a > > Correct, because distlib does not support multiple installed versions of the > same distribution, nor does it do the sys.path manipulations on the fly which > have caused many people to have a problem with setuptools. > > Do people see this as a problem? I would have thought that venvs would allow > people to deal with multiple versions in a less magical way. So does buildout, which doesn't need venvs; it just (if you configure it that way) puts all your eggs in a giant cache directory and writes scripts with hardcoded sys.path to include the right ones. This is actually more explicit than venvs, since it doesn't depend on environment variables or on installation state. IOW, there are other choices available besides "implicit environment-based path" and "dynamically generated path". Even setuptools doesn't require that you have a dynamic path. > If that is a real requirement which should be supported, shouldn't there be a > PEP for it, if it's coming into Python? It's not supported by distutils, and > it has been a point of contention. Distutils lets you install things wherever you want; in the naive case you could use install --root to install every package to a version-specific directory and then use something like Gnu Stow to create symlink farms. Python supports explicit sys.path construction and modification, and of course people certainly "vendor" (i.e. bundle) their dependencies directly in order to have a specific version of them. So, I don't think it's accurate to consider multi-version installation a totally new feature. (And AFAIK, the point of contention isn't that setuptools *supports* multi-version installation, it's that it's the *default* implementation.) In any event, wheels are designed to be able to be used in the same way as eggs for multi-version installs. The question of *how* has been brought up by Nick before, and I've thrown out some counter-proposals. It's still an open issue as to how much *active* support will be provided, but my impression of the discussion is that even if the stdlib isn't exactly *encouraging* multi-version installs, we don't want to *break* them. Hence my suggestion that if you want to drop pkg_resources use from generated scripts, you should use buildout's approach (explicit sys.path baked into the script) rather than distlib's current laissez-faire approach. Or you can just check versions, I'm not that picky. All I want is that if you install a new version of a package and still have an old copy of the script, the old script should still run the old version, or at least give you an error telling you the script wasn't updated, rather than silently running a different version. Buildout's approach accomplishes this by hardcoding egg paths, so as long as you don't delete the eggs, everything is fine, and if you do delete any of them, you can see what's wrong by looking at the script source. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> I think the point is that people might be dependent on this functionality and > changing it out from underneath them could break their world. I got the point that Daniel made, and my question was about *how* their world would break, and whether we really need to support multiple versions of something installed side-by-side, with on-the-fly sys.path manipulation. If that is a real requirement which should be supported, shouldn't there be a PEP for it, if it's coming into Python? It's not supported by distutils, and it has been a point of contention. A PEP would allow standardisation of the multiple-versions feature it it's considered desirable, rather than definition by implementation (which I understand you're not in favour of, in general). If it's not considered desirable and doesn't need support, then we only need to consider if it's undeclared setuptools dependencies that we're concerned with, or some other failure mode not yet identified - hence, my questions. I like to get into specifics :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 7:20 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> PJE is correct that if we surprise people with non-pkg_resources >> console_scripts then we will break things for people who are more >> interested in a working packaging experience. > > Do you mean that you think multiple versions have to be supported, and that's > why console scripts should remain pkg_resources - dependent? > > If you don't think that multiple version support is needed, then the non- > pkg_resources versions of the script should be able to locate the function to > call from the script, assuming it can import the module. Are you saying that > the import or function call will fail, because the distribution didn't > reference setuptools as a dependency, and yet expects it to be there? > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig I think the point is that people might be dependent on this functionality and changing it out from underneath them could break their world. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
Daniel Holth gmail.com> writes: > PJE is correct that if we surprise people with non-pkg_resources > console_scripts then we will break things for people who are more > interested in a working packaging experience. Do you mean that you think multiple versions have to be supported, and that's why console scripts should remain pkg_resources - dependent? If you don't think that multiple version support is needed, then the non- pkg_resources versions of the script should be able to locate the function to call from the script, assuming it can import the module. Are you saying that the import or function call will fail, because the distribution didn't reference setuptools as a dependency, and yet expects it to be there? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
PJ Eby telecommunity.com> writes: > As others have already mentioned, this is not a bug but a feature. > Setuptools-generated scripts are linked to a specific version of the > project, which means that you can install more than one version by > renaming the scripts or installing the scripts to different > directories. > > While other strategies are definitely possible, distlib's approach is > not backward-compatible, as it means installing new versions of a Correct, because distlib does not support multiple installed versions of the same distribution, nor does it do the sys.path manipulations on the fly which have caused many people to have a problem with setuptools. Do people see this as a problem? I would have thought that venvs would allow people to deal with multiple versions in a less magical way. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> > I'll see if I can do a patch. I don't think it will be hard at all, > and I do think it's work that will eventually become necessary. > patch for which solution? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
OH -scripts would be the distutils-style scrips. On Thu, Jul 18, 2013 at 6:51 PM, Daniel Holth wrote: > On Thu, Jul 18, 2013 at 6:42 PM, Nick Coghlan wrote: >> I actually now plan to make scripts and exports first class citizens in PEP >> 426, with pydist-scripts.json and pydist-exports.json as extracted summary >> files (like the existing pydist-dependencies.json). >> >> They're important enough to include directly. >> >> Cheers, >> Nick. > > Must they be two separate features? One of the reasons I use > entry_points scripts is that I forget that the scripts= command to > setup() exists at all. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Thu, Jul 18, 2013 at 6:42 PM, Nick Coghlan wrote: > I actually now plan to make scripts and exports first class citizens in PEP > 426, with pydist-scripts.json and pydist-exports.json as extracted summary > files (like the existing pydist-dependencies.json). > > They're important enough to include directly. > > Cheers, > Nick. Must they be two separate features? One of the reasons I use entry_points scripts is that I forget that the scripts= command to setup() exists at all. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On 19 Jul 2013 08:42, "Nick Coghlan" wrote: > > I actually now plan to make scripts and exports first class citizens in PEP 426, with pydist-scripts.json and pydist-exports.json as extracted summary files (like the existing pydist-dependencies.json). > > They're important enough to include directly. The PEP that will define the updated dist-info contents will be the sdist 2.0 PEP. Things are probably stable enough for me to write that, now. Cheers, Nick. > > Cheers, > Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 5:56 PM, Paul Moore wrote: > On 18 July 2013 22:08, Marcus Smith wrote: >> >> it's a practical problem for users, due to being currently responsible for >> fulfilling the setuptools dependency themselves in non-virtualenv >> environments >> IMO, we need to bundle or install it for them (through dynamic installs, >> or add the logic to get-pip) > > > Seriously, we're talking here about bundling pip with the Python installer. > Why not just bundle setuptools as well? Don't vendor it, don't jump through > hoops, just bundle it too, so that all Python environments can be assumed to > have pip and setuptools present. (Note that I'm one of the least likely > people to advocate setuptools around here, and yet even I don't see why > we're working so hard to avoid just having the thing available...) > > It seems to me that by bundling pip but not setuptools, we're just making > unnecessary work for ourselves. I'll see if I can do a patch. I don't think it will be hard at all, and I do think it's work that will eventually become necessary. PJE is correct that if we surprise people with non-pkg_resources console_scripts then we will break things for people who are more interested in a working packaging experience. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
I actually now plan to make scripts and exports first class citizens in PEP 426, with pydist-scripts.json and pydist-exports.json as extracted summary files (like the existing pydist-dependencies.json). They're important enough to include directly. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 4:36 PM, Paul Moore wrote: > As regards console scripts, I think they should be rewritten to remove the > dependency on pkg_resources. That should be a setuptools fix, As others have already mentioned, this is not a bug but a feature. Setuptools-generated scripts are linked to a specific version of the project, which means that you can install more than one version by renaming the scripts or installing the scripts to different directories. While other strategies are definitely possible, distlib's approach is not backward-compatible, as it means installing new versions of a project will change *existing scripts'* semantics, even if you installed the previous version's scripts to different locations and intended them to remain accessible. If you want an example of doing it right, see buildout, which hardcodes the entire sys.path of a script to refer to the exact versions of all dependencies; while this has different failure modes (i.e., dependence on absolute paths), it is more stable as to script semantics even than setuptools' default behavior. > maybe triggered by a flag (possibly implied by > --single-version-externally-managed, as the pkg_resources complexity is only > needed when multi-versions are involved). That option does not preclude the existence of multiple versions, or the possibility of installing the same script to different directories for different installed versions. If you *must* do this, I suggest using buildout's approach of hardwiring sys.path in the script, only strengthened by checking for the actual existence and versions, rather than distlib's anything-goes approach. (Of course, as Donald points out, this won't do anything for those scripts that themselves make use of other packages' entry points: they will have a runtime dependency on pkg_resources anyway.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 5:56 PM, Paul Moore wrote: > On 18 July 2013 22:08, Marcus Smith wrote: > it's a practical problem for users, due to being currently responsible for > fulfilling the setuptools dependency themselves in non-virtualenv environments > IMO, we need to bundle or install it for them (through dynamic installs, or > add the logic to get-pip) > > Seriously, we're talking here about bundling pip with the Python installer. > Why not just bundle setuptools as well? Don't vendor it, don't jump through > hoops, just bundle it too, so that all Python environments can be assumed to > have pip and setuptools present. (Note that I'm one of the least likely > people to advocate setuptools around here, and yet even I don't see why we're > working so hard to avoid just having the thing available...) > > It seems to me that by bundling pip but not setuptools, we're just making > unnecessary work for ourselves. > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Because a significant number of people have had issues with things breaking because their setuptools install got messed up. Typically some combination of things convinced pip to uninstall setuptools which then breaks pip completely (due to a reliance on pkg_resources) and breaks installing from sdists (due to a reliance on setuptools). This isn't a problem for most tools because they could just use pip to fix their dependencies. However when it's the package manager that breaks you're stuck fixing things manually. While it's obvious to you or I what the problem is I've found that the bulk of people who have these issues have no idea why they are getting the error and how to fix it. Bundling this means that pip is either installed and works, or it isn't installed. It makes it much simpler for end users to deal with and makes it much more robust. Right now this is particularly troublesome because there's a huge bug that's causing this to happen and I think i've not gone a day without having someone different run into the problem. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 22:08, Marcus Smith wrote: > it's a practical problem for users, due to being currently responsible for > fulfilling the setuptools dependency themselves in non-virtualenv > environments > IMO, we need to bundle or install it for them (through dynamic installs, > or add the logic to get-pip) > Seriously, we're talking here about bundling pip with the Python installer. Why not just bundle setuptools as well? Don't vendor it, don't jump through hoops, just bundle it too, so that all Python environments can be assumed to have pip and setuptools present. (Note that I'm one of the least likely people to advocate setuptools around here, and yet even I don't see why we're working so hard to avoid just having the thing available...) It seems to me that by bundling pip but not setuptools, we're just making unnecessary work for ourselves. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Thu, Jul 18, 2013 at 1:50 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: >> For one thing you can have more than one mysql = in the same >> sqlalchemy.dialects. I think in this instance the string parsing is > > Don't you say in the PEP about the key that "It must be locally unique > within this distribution’s group."? Setuptools requires this per-distribution uniqueness, but note that uniqueness is not required across distributions. So more than one distribution can export a 'mysql' in the 'sqlalchemy.dialects' group. It's up to the application to decide how to handle multiple definitions; typically one either uses all of them or the first one found on sys.path, or some other tie-breaking mechanism. The pkg_resources entry point APIs just provide operations for iterating over entry points defined on either a single distribution, or across all distributions on a specified set of directories. (Via the WorkingSet API.) > Note that I don't see necessarily a connection between extras and flags, > though > you've mentioned that they're extras. Does setuptools store, against an > installed distribution, the extras it was installed with? AFAIK it doesn't. > (Even if it did, it would need to keep that updated if one of the extras' > dependencies were later uninstalled.) And if not, how would having extras in > the specification help, since you can't test the "must be installed" part? The pkg_resources implementation does a require() for the extras at the time the entry point is loaded, i.e., just before importing. This allows it to dynamically add requirements to sys.path, or alternatively raise an error to indicate the extras aren't available. In addition, various entry point API functions take an 'installer' keyword argument, specifying a callback to handle installation of missing extras. Setuptools uses this feature internally, so that if you use a setup.py command whose entry point needs additional dependencies, those will be fetched on-the-fly. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
fyi, I'm updating donald's original setuptools bundle issue with all of this as the choices become clearer https://github.com/pypa/pip/issues/1049 On Thu, Jul 18, 2013 at 2:08 PM, Marcus Smith wrote: > > virtualenv /tmp/builder >> /tmp/builder/bin/pip wheel -w /tmp/wheels -r requirements.txt >> > > people will expect to be able to do this globally (i.e not in a > virtualenv). that's when the headache starts > > It does not try to avoid the non-problem of installing setuptools when you >> actually need it >> > > it's a practical problem for users, due to being currently responsible for > fulfilling the setuptools dependency themselves in non-virtualenv > environments > IMO, we need to bundle or install it for them (through dynamic installs, > or add the logic to get-pip) > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> virtualenv /tmp/builder > /tmp/builder/bin/pip wheel -w /tmp/wheels -r requirements.txt > people will expect to be able to do this globally (i.e not in a virtualenv). that's when the headache starts It does not try to avoid the non-problem of installing setuptools when you > actually need it > it's a practical problem for users, due to being currently responsible for fulfilling the setuptools dependency themselves in non-virtualenv environments IMO, we need to bundle or install it for them (through dynamic installs, or add the logic to get-pip) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 21:41, Donald Stufft wrote: > Console scripta aren't the only use of entry points fwiw. THere's other > entry points programs use. I don't know if they all depend on setuptools or > if just assume it's there. Technically the should depend but that would > break things for those people. > > I think either way pkg_resources is going to need to be installed, but > setuptools won't. > If a project uses setuptools features at runtime, it should declare setuptools as a dependency. The difference with script wrappers is that the project didn't write the code, setuptools itself did. Any other use of entry points requires "import pkg_resources" in the user-written code, and should therefore be supported by having setuptools in the runtime dependency list. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 4:36 PM, Paul Moore wrote: > On 18 July 2013 18:24, Marcus Smith wrote: > > I think it's still useful to have pip vendor just pkg_resources (as > pip.pkg_resources). It's easy, it gives you enough to install wheels, > and it's not the only thing you would do. > > I agree. there's 2 problems to be solved here > > 1) making pip a self-sufficient wheel installer (which requires some > internal pkg_resources equivalent) > 2) removing the user headache of a setuptools build *dependency* for > practically all current pypi distributions > > for #2, we have a few paths I think > > 1) bundle setuptools (and have pip install "pkg_resources" for console > scripts, if it existed as a separate project) > 2) bundle setuptools (and rewrite the console script wrapper logic to not > need pkg_resources?) > 3) dynamic install of setuptools from wheel when pip needs to instal sdists > (which is 99.9% of the time, so this feels a bit silly) > 4) just be happy that the pip bootstrap/bundle efforts will alleviate the > pain in new versions of python (by pre-installing setuptools?) > > As you say, for #1 using an internal pkg_resources (probably distlib's, why > bother vendoring a second one?) works. > > Given that pip forces use of setuptools for *all* sdist builds, I think we > have to bundle it for that purpose. I really dislike the need to do this, but > I don't see a way round it. And if we do, we can just as easily use the real > pkg_resources as distlib's emulation. > > As regards console scripts, I think they should be rewritten to remove the > dependency on pkg_resources. That should be a setuptools fix, maybe triggered > by a flag (possibly implied by --single-version-externally-managed, as the > pkg_resources complexity is only needed when multi-versions are involved). If > Jason's not comfortable with the change, then we'll probably have to find > some way of doing it within pip, which will likely to be a fairly gross hack > (unless we go really bleeding-edge and don't pit scripts into a wheel *at > all* (or just omit exes and -script.py files, I don't know) and put the > exports metadata in the wheel assuming that it's the wheel installer's job to > create the wrappers. We can do that for pip install, and we just have to > assume that other tools (wheel install, distlib) will do the same. > > TBH, my preference is for the metadata approach, do it correctly from the > start. > > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Console scripta aren't the only use of entry points fwiw. THere's other entry points programs use. I don't know if they all depend on setuptools or if just assume it's there. Technically the should depend but that would break things for those people. I think either way pkg_resources is going to need to be installed, but setuptools won't. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 18:24, Marcus Smith wrote: > > I think it's still useful to have pip vendor just pkg_resources (as >> pip.pkg_resources). It's easy, it gives you enough to install wheels, >> and it's not the only thing you would do. > > > I agree. there's 2 problems to be solved here > > 1) making pip a self-sufficient wheel installer (which requires some > internal pkg_resources equivalent) > 2) removing the user headache of a setuptools build *dependency* for > practically all current pypi distributions > > for #2, we have a few paths I think > > 1) bundle setuptools (and have pip install "pkg_resources" for console > scripts, if it existed as a separate project) > 2) bundle setuptools (and rewrite the console script wrapper logic to not > need pkg_resources?) > 3) dynamic install of setuptools from wheel when pip needs to instal > sdists (which is 99.9% of the time, so this feels a bit silly) > 4) just be happy that the pip bootstrap/bundle efforts will alleviate the > pain in new versions of python (by pre-installing setuptools?) > As you say, for #1 using an internal pkg_resources (probably distlib's, why bother vendoring a second one?) works. Given that pip forces use of setuptools for *all* sdist builds, I think we have to bundle it for that purpose. I really dislike the need to do this, but I don't see a way round it. And if we do, we can just as easily use the real pkg_resources as distlib's emulation. As regards console scripts, I think they should be rewritten to remove the dependency on pkg_resources. That should be a setuptools fix, maybe triggered by a flag (possibly implied by --single-version-externally-managed, as the pkg_resources complexity is only needed when multi-versions are involved). If Jason's not comfortable with the change, then we'll probably have to find some way of doing it within pip, which will likely to be a fairly gross hack (unless we go really bleeding-edge and don't pit scripts into a wheel *at all* (or just omit exes and -script.py files, I don't know) and put the exports metadata in the wheel assuming that it's the wheel installer's job to create the wrappers. We can do that for pip install, and we just have to assume that other tools (wheel install, distlib) will do the same. TBH, my preference is for the metadata approach, do it correctly from the start. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 1:24 PM, Marcus Smith wrote: > >> I think it's still useful to have pip vendor just pkg_resources (as >> pip.pkg_resources). It's easy, it gives you enough to install wheels, >> and it's not the only thing you would do. > > > I agree. there's 2 problems to be solved here > > 1) making pip a self-sufficient wheel installer (which requires some > internal pkg_resources equivalent) > 2) removing the user headache of a setuptools build *dependency* for > practically all current pypi distributions > > for #2, we have a few paths I think > > 1) bundle setuptools (and have pip install "pkg_resources" for console > scripts, if it existed as a separate project) > 2) bundle setuptools (and rewrite the console script wrapper logic to not > need pkg_resources?) > 3) dynamic install of setuptools from wheel when pip needs to instal sdists > (which is 99.9% of the time, so this feels a bit silly) > 4) just be happy that the pip bootstrap/bundle efforts will alleviate the > pain in new versions of python (by pre-installing setuptools?) virtualenv /tmp/builder /tmp/builder/bin/pip wheel -w /tmp/wheels -r requirements.txt virtualenv /tmp/no-setuptools /tmp/no-setuptools/bin/pip install --use-wheel --find-links=/tmp/wheels --no-index -r requirements.txt That is the anti-setuptools workflow I envision. The build environment has an appropriate amount of setuptools and the no-setuptools environment has none. This gives you the option of not having setuptools if you don't want it, something that some people will appreciate. It does not try to avoid the non-problem of installing setuptools when you actually need it. Eventually there may be more sophisticated build requirements handling, for whatever that's worth, so that you might not have to have an explicit setuptools virtualenv. System packaging certainly doesn't install build requirements into their own isolated environment. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 2:19 PM, Marcus Smith wrote: > >> >> Perhaps you're forgetting that there already is an internal pkg_resources >> equivalent in my pip-distlib branch - this is a pkg_resources >> compatibility >> shim using pip.vendor.distlib which passed all the pip tests when it was >> submitted as a PR. > > > : ) no I haven't forgotten. I actually bring it up with others pretty > often. > my use of "pkg_resource equivalent" was actually a reference to your PR > work. > > Marcus > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API for registering/managing URLs for a package
On Jul 18, 2013, at 12:42 PM, Noah Kantrowitz wrote: > > On Jul 18, 2013, at 8:06 AM, Noah Kantrowitz wrote: > >> >> On Jul 18, 2013, at 7:10 AM, M.-A. Lemburg wrote: >> >>> I would like to write a script to automatically register release URLs >>> for PyPI packages. >>> >>> Is the REST API documented somewhere, or is the implementation the >>> spec ? ;-) >>> >>> And related to this: >>> >>> Will there be an option to tell PyPI's CDN to cache the release >>> URL's contents ? >> >> I think you are perhaps confused, the use of external URLs on PyPI is >> formally deprecated. The way you inform the PyPI and the CDN network about >> your package is you upload it to PyPI. pip 1.4 effectively disables "unsafe" >> external URLs, and all external URLs will follow soon. > > Someone reminded me that I'm only partially correct, the external URL stuffs > will continue to be supported, but only as a convenience during package > registration/upload. From the PoV of clients (and the CDN) everything will be > local. > > --Noah > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Noah, External urls are still supported (Although discouraged). Marc-Andre, There is documentation in the PEP, however I have another PEP coming up for a more streamlined upload process that also contains a much nicer method of sending external urls as well. So you might want to wait for that. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distribute 0.7.3 causing installation error?
On Thu, Jul 18, 2013 at 10:58 AM, Liam Kirsher wrote: > Marcus, > > Thanks! After reading that I think I can fix this by installing pip 1.4. > you can also make your recipe run "pip install -U setuptools" separately before moving on to the supervisor upgrade. that will work and is maybe easier > However, some questions remain. Pip is currently being installed via > distribute_setup.py, which is retrieved from here: > http://python-distribute.org/distribute_setup.py > a setuptools person should probably speak to this, but I would say, don't use this anymore. use "ez_setup.py" which is for setuptools. if you're starting an environment from scratch, just use the new setuptools. https://pypi.python.org/pypi/setuptools/0.9.6#installation-instructions Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> > Perhaps you're forgetting that there already is an internal pkg_resources > equivalent in my pip-distlib branch - this is a pkg_resources compatibility > shim using pip.vendor.distlib which passed all the pip tests when it was > submitted as a PR. > : ) no I haven't forgotten. I actually bring it up with others pretty often. my use of "pkg_resource equivalent" was actually a reference to your PR work. Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> The load_entry_point needs the dist name because of how it's implemented - > it > defers to the distribution instance. AFAICT there are no actual checks. > > def load_entry_point(dist, group, name): > """Return `name` entry point of `group` for `dist` or raise > ImportError""" > return get_distribution(dist).load_entry_point(group, name) > > it checks the version. you get this. I have pip-1.5dev1 in this case, but a script wrapper referencing 1.4rc5 (pip)qwcode@qwcode:~/p/pypa/pip$ pip --version Traceback (most recent call last): File "/home/qwcode/.qwdev/pip/bin/pip", line 5, in from pkg_resources import load_entry_point File "/home/qwcode/.qwdev/pip/lib/python2.6/site-packages/pkg_resources.py", line 3011, in parse_requirements(__requires__), Environment() File "/home/qwcode/.qwdev/pip/lib/python2.6/site-packages/pkg_resources.py", line 626, in resolve raise DistributionNotFound(req) pkg_resources.DistributionNotFound: pip==1.4rc5 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Thu, Jul 18, 2013 at 1:50 PM, Vinay Sajip wrote:
> Daniel Holth gmail.com> writes:
>
>> On Thu, Jul 18, 2013 at 9:27 AM, Paul Moore gmail.com>
> wrote:
>
>> It is an extension so it can be a separate PEP, since there's enough
>> to talk about in the main PEP. The document tries to write down what
>> setuptools does in a straightforward json way.
>
> If the JSON we're talking about goes in the main metadata dictionary,
> perhaps
> it should go into PEP 426, so that everything is in one place. As we're
> talking
> about special handling of script generation by installers, it may make sense
> to
> not consider them as extensions but as core metadata.
>
>> > 2. distlib calls these "exports" and I think that's a better name. But
> if
>> > names are all that we argue over, I'm happy
>
> The reason for picking "exports" is that you can export data, not just code,
> and "entry point" has a connotation of being code. PJE suggested "exports"
> and
> I think it fits the bill better than "entry_points".
>
>> > 3. Someone (I think it was PJE) pointed out that having entry points in
> a
>> > separate metadata file was good because it allowed a fast check of "does
>> > this distribution expose entry points?" Note that this isn't a useful
> thing
>> > to check for script wrappers, which again argues that those should be
>> > handled separately.
>
> That seems generally true, except that there might be applications out there
> that want to walk over the scripts associated with different installed
> distributions. That seems a legitimate, though uncommon, use case.
>
> In any case, I think the script metadata should be structured as
>
> "scripts": {
> "console": [spec1, spec2]
> "gui": [spec1, spec2]
> }
>
> Because that allows a sensible place for e.g. script generation options, as
> in
>
> "scripts": {
> "console": [spec1, spec2]
> "gui": [spec1, spec2]
> "options": { ... }
> }
>
>
>> I am more interested in seeing the installer update some kind of index
>> like a sqlite database. I don't know if it would be faster since I
>> haven't tried it.
>
> That (a SQLite installation database) is probably some way off, and would
> require more significant changes elsewhere. The big advantage of the current
> setup with text files is that every thing is human readable - very handy
> when
> things go wrong. I don't know whether this area is a performance bottleneck,
> but we will be able to deal with it using a separate exports.json file in
> the
> short term.
Who knows. On some filesystems stat() is painfully slow and you could
be better off just parsing the single metadata file.
>> For one thing you can have more than one mysql = in the same
>> sqlalchemy.dialects. I think in this instance the string parsing is
>
> Don't you say in the PEP about the key that "It must be locally unique
> within
> this distribution’s group."?
The kind of entry point has to be unique, but the name inside the spec does not:
dialects : [
"mysql = first mysql driver...",
"mysql = second mysql driver..."
]
>> probably simpler than defining a more JSONy version.
>
> I think Paul's point is that if it was JSON, you wouldn't need to parse
> anything. The current format of entries is
>
> name = module:attrs [flag1,flag2]
>
> which could be { "name": name, "module": module, "attrs": attrs, "flags":
> ["flag1", "flag2"] } which is obviously more verbose.
>
> Note that I don't see necessarily a connection between extras and flags,
> though
> you've mentioned that they're extras. Does setuptools store, against an
> installed distribution, the extras it was installed with? AFAIK it doesn't.
> (Even if it did, it would need to keep that updated if one of the extras'
> dependencies were later uninstalled.) And if not, how would having extras in
> the specification help, since you can't test the "must be installed" part?
> On
> the other hand, you might want to use generalised flags that provide control
> over how the exported entry is processed.
You might mean the document's mention that in setuptools loading an
entry point can require a particular extra. In setuptools this would
mean additional eggs could be added to sys.path as a result of loading
the entry point.
> One reason for keeping the format as-is might be in case any migration
> issues
> come up (i.e. people depending on this specific format in some way), but I'm
> not sure whether there are any such issues or what they might be.
>
>> FWIW the PEP 426 metadata is also full of structured strings.
>
> True - the dependency specifier, if nothing else.
>
> One other thing is that the PEP needs to state that the exports metadata
> must
> be written to exports.json in the .dist-info folder by an installer - that
> isn't mentioned anywhere. Also, whether it should contain the scripts part,
> or
> just the other exports (but see my comment above as to why scripts might be
> considered exports worth iterating over, like any other).
I would like to see it timed with and without exports.jso
Re: [Distutils] Q about best practices now (or near future)
Marcus Smith gmail.com> writes: > > > > I think it's still useful to have pip vendor just pkg_resources (as > pip.pkg_resources). It's easy, it gives you enough to install wheels, > and it's not the only thing you would do. > > I agree. there's 2 problems to be solved here > > 1) making pip a self-sufficient wheel installer (which requires some internal pkg_resources equivalent) > 2) removing the user headache of a setuptools build *dependency* for practically all current pypi distributions > > for #2, we have a few paths I think > > 1) bundle setuptools (and have pip install "pkg_resources" for console scripts, if it existed as a separate project) > 2) bundle setuptools (and rewrite the console script wrapper logic to not need pkg_resources?) > 3) dynamic install of setuptools from wheel when pip needs to instal sdists (which is 99.9% of the time, so this feels a bit silly) > 4) just be happy that the pip bootstrap/bundle efforts will alleviate the pain in new versions of python (by pre-installing setuptools?) If setuptools changes the script generation, the need for pkg_resources is gone at least from that part of the picture. Perhaps you're forgetting that there already is an internal pkg_resources equivalent in my pip-distlib branch - this is a pkg_resources compatibility shim using pip.vendor.distlib which passed all the pip tests when it was submitted as a PR. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
Marcus Smith gmail.com> writes: > pkg_resources scripts confirm the version. don't see that here? not necessary? The load_entry_point needs the dist name because of how it's implemented - it defers to the distribution instance. AFAICT there are no actual checks. def load_entry_point(dist, group, name): """Return `name` entry point of `group` for `dist` or raise ImportError""" return get_distribution(dist).load_entry_point(group, name) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 1:01 PM, Marcus Smith wrote: > > > On Thu, Jul 18, 2013 at 9:49 AM, Vinay Sajip > wrote: >> >> Marcus Smith gmail.com> writes: >> >> > the idea to have pip vendor setuptools crumbles a bit due to console >> scripts >> needing pkg_resources. >> >> They don't *need* pkg_resources. All they're doing is taking a module name >> and the name of a nested object in the form 'a.b.c', and distlib-generated >> scripts show that no external references are needed. Here's the template >> for >> a distlib-generated script: > > > pkg_resources scripts confirm the version. don't see that here? not > necessary? It's useful when you have more than one version of things installed as eggs. pkg_resources does the full dependency resolution and adds everything to the sys.path. When you are not doing that then it's not needed. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
Daniel Holth gmail.com> writes:
> On Thu, Jul 18, 2013 at 9:27 AM, Paul Moore gmail.com>
wrote:
> It is an extension so it can be a separate PEP, since there's enough
> to talk about in the main PEP. The document tries to write down what
> setuptools does in a straightforward json way.
If the JSON we're talking about goes in the main metadata dictionary,
perhaps
it should go into PEP 426, so that everything is in one place. As we're
talking
about special handling of script generation by installers, it may make sense
to
not consider them as extensions but as core metadata.
> > 2. distlib calls these "exports" and I think that's a better name. But
if
> > names are all that we argue over, I'm happy
The reason for picking "exports" is that you can export data, not just code,
and "entry point" has a connotation of being code. PJE suggested "exports"
and
I think it fits the bill better than "entry_points".
> > 3. Someone (I think it was PJE) pointed out that having entry points in
a
> > separate metadata file was good because it allowed a fast check of "does
> > this distribution expose entry points?" Note that this isn't a useful
thing
> > to check for script wrappers, which again argues that those should be
> > handled separately.
That seems generally true, except that there might be applications out there
that want to walk over the scripts associated with different installed
distributions. That seems a legitimate, though uncommon, use case.
In any case, I think the script metadata should be structured as
"scripts": {
"console": [spec1, spec2]
"gui": [spec1, spec2]
}
Because that allows a sensible place for e.g. script generation options, as
in
"scripts": {
"console": [spec1, spec2]
"gui": [spec1, spec2]
"options": { ... }
}
> I am more interested in seeing the installer update some kind of index
> like a sqlite database. I don't know if it would be faster since I
> haven't tried it.
That (a SQLite installation database) is probably some way off, and would
require more significant changes elsewhere. The big advantage of the current
setup with text files is that every thing is human readable - very handy
when
things go wrong. I don't know whether this area is a performance bottleneck,
but we will be able to deal with it using a separate exports.json file in
the
short term.
> For one thing you can have more than one mysql = in the same
> sqlalchemy.dialects. I think in this instance the string parsing is
Don't you say in the PEP about the key that "It must be locally unique
within
this distribution’s group."?
> probably simpler than defining a more JSONy version.
I think Paul's point is that if it was JSON, you wouldn't need to parse
anything. The current format of entries is
name = module:attrs [flag1,flag2]
which could be { "name": name, "module": module, "attrs": attrs, "flags":
["flag1", "flag2"] } which is obviously more verbose.
Note that I don't see necessarily a connection between extras and flags,
though
you've mentioned that they're extras. Does setuptools store, against an
installed distribution, the extras it was installed with? AFAIK it doesn't.
(Even if it did, it would need to keep that updated if one of the extras'
dependencies were later uninstalled.) And if not, how would having extras in
the specification help, since you can't test the "must be installed" part?
On
the other hand, you might want to use generalised flags that provide control
over how the exported entry is processed.
One reason for keeping the format as-is might be in case any migration
issues
come up (i.e. people depending on this specific format in some way), but I'm
not sure whether there are any such issues or what they might be.
> FWIW the PEP 426 metadata is also full of structured strings.
True - the dependency specifier, if nothing else.
One other thing is that the PEP needs to state that the exports metadata
must
be written to exports.json in the .dist-info folder by an installer - that
isn't mentioned anywhere. Also, whether it should contain the scripts part,
or
just the other exports (but see my comment above as to why scripts might be
considered exports worth iterating over, like any other).
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> I think it's still useful to have pip vendor just pkg_resources (as > pip.pkg_resources). It's easy, it gives you enough to install wheels, > and it's not the only thing you would do. I agree. there's 2 problems to be solved here 1) making pip a self-sufficient wheel installer (which requires some internal pkg_resources equivalent) 2) removing the user headache of a setuptools build *dependency* for practically all current pypi distributions for #2, we have a few paths I think 1) bundle setuptools (and have pip install "pkg_resources" for console scripts, if it existed as a separate project) 2) bundle setuptools (and rewrite the console script wrapper logic to not need pkg_resources?) 3) dynamic install of setuptools from wheel when pip needs to instal sdists (which is 99.9% of the time, so this feels a bit silly) 4) just be happy that the pip bootstrap/bundle efforts will alleviate the pain in new versions of python (by pre-installing setuptools?) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
I tried it out. pip can install setuptools when only pkg_resources is installed. The only thing stopping it is a small check for whether the current setuptools is of the distribute variety. On Thu, Jul 18, 2013 at 12:53 PM, Daniel Holth wrote: > On Thu, Jul 18, 2013 at 12:12 PM, Marcus Smith wrote: >> >>> >>> It would actually be nice if "pkg_resources" and "setuptools-core" >>> were available as separate PyPI distributions, and setuptools bundled >>> them together with easy_install. It's a *long* way down the priority >>> list thing (and will likely never make it to the top, although it may >>> be more practical once pip vendors the bits it needs). >> >> >> the idea to have pip vendor setuptools crumbles a bit due to console scripts >> needing pkg_resources. >> you're left with 2 poor solutions: 1) rewriting script import lines, or 2) >> still installing setuptools anyway >> >> so, having a separate pkg_resources is higher up on the list I think for >> that reason. >> without a separate pkg_resources, I think the "dynamic install of >> setuptools" idea wins out, or no change at all. > > I think it's still useful to have pip vendor just pkg_resources (as > pip.pkg_resources). It's easy, it gives you enough to install wheels, > and it's not the only thing you would do. It shouldn't make much > difference whether the vendoring happens before or after > pkg_resource's separation. The trickiest parts might be adding the > undeclared pkg_resources / setuptools dependency when appropriate and > figuring out whether we can install setuptools even if it's not > available as a wheel. > > Meanwhile someone might add a flag or a plugin to setuptools' > console_scripts handler to generate them in a different way. > > I am not worried that 99.9% of pypi-hosted packages depend on > setuptools or distutils. It is enough to introduce only the > possibility of getting along without it. For the rest it is > appropriate to install and use setuptools to build packages that were > in fact designed to use it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 9:49 AM, Vinay Sajip wrote: > Marcus Smith gmail.com> writes: > > > the idea to have pip vendor setuptools crumbles a bit due to console > scripts > needing pkg_resources. > > They don't *need* pkg_resources. All they're doing is taking a module name > and the name of a nested object in the form 'a.b.c', and distlib-generated > scripts show that no external references are needed. Here's the template > for > a distlib-generated script: > pkg_resources scripts confirm the version. don't see that here? not necessary? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 12:12 PM, Marcus Smith wrote: > >> >> It would actually be nice if "pkg_resources" and "setuptools-core" >> were available as separate PyPI distributions, and setuptools bundled >> them together with easy_install. It's a *long* way down the priority >> list thing (and will likely never make it to the top, although it may >> be more practical once pip vendors the bits it needs). > > > the idea to have pip vendor setuptools crumbles a bit due to console scripts > needing pkg_resources. > you're left with 2 poor solutions: 1) rewriting script import lines, or 2) > still installing setuptools anyway > > so, having a separate pkg_resources is higher up on the list I think for > that reason. > without a separate pkg_resources, I think the "dynamic install of > setuptools" idea wins out, or no change at all. I think it's still useful to have pip vendor just pkg_resources (as pip.pkg_resources). It's easy, it gives you enough to install wheels, and it's not the only thing you would do. It shouldn't make much difference whether the vendoring happens before or after pkg_resource's separation. The trickiest parts might be adding the undeclared pkg_resources / setuptools dependency when appropriate and figuring out whether we can install setuptools even if it's not available as a wheel. Meanwhile someone might add a flag or a plugin to setuptools' console_scripts handler to generate them in a different way. I am not worried that 99.9% of pypi-hosted packages depend on setuptools or distutils. It is enough to introduce only the possibility of getting along without it. For the rest it is appropriate to install and use setuptools to build packages that were in fact designed to use it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
Marcus Smith gmail.com> writes:
> the idea to have pip vendor setuptools crumbles a bit due to console
scripts
needing pkg_resources.
They don't *need* pkg_resources. All they're doing is taking a module name
and the name of a nested object in the form 'a.b.c', and distlib-generated
scripts show that no external references are needed. Here's the template for
a distlib-generated script:
SCRIPT_TEMPLATE = '''%(shebang)s
if __name__ == '__main__':
import sys, re
def _resolve(module, func):
__import__(module)
mod = sys.modules[module]
parts = func.split('.')
result = getattr(mod, parts.pop(0))
for p in parts:
result = getattr(result, p)
return result
try:
sys.argv[0] = re.sub('-script.pyw?$', '', sys.argv[0])
func = _resolve('%(module)s', '%(func)s')
rc = func() # None interpreted as 0
except Exception as e: # only supporting Python >= 2.6
sys.stderr.write('%%s\\n' %% e)
rc = 1
sys.exit(rc)
'''
I don't see any reason why setuptools couldn't be updated to use this
approach.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API for registering/managing URLs for a package
On Jul 18, 2013, at 8:06 AM, Noah Kantrowitz wrote: > > On Jul 18, 2013, at 7:10 AM, M.-A. Lemburg wrote: > >> I would like to write a script to automatically register release URLs >> for PyPI packages. >> >> Is the REST API documented somewhere, or is the implementation the >> spec ? ;-) >> >> And related to this: >> >> Will there be an option to tell PyPI's CDN to cache the release >> URL's contents ? > > I think you are perhaps confused, the use of external URLs on PyPI is > formally deprecated. The way you inform the PyPI and the CDN network about > your package is you upload it to PyPI. pip 1.4 effectively disables "unsafe" > external URLs, and all external URLs will follow soon. Someone reminded me that I'm only partially correct, the external URL stuffs will continue to be supported, but only as a convenience during package registration/upload. From the PoV of clients (and the CDN) everything will be local. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> > It would actually be nice if "pkg_resources" and "setuptools-core" > were available as separate PyPI distributions, and setuptools bundled > them together with easy_install. It's a *long* way down the priority > list thing (and will likely never make it to the top, although it may > be more practical once pip vendors the bits it needs). > the idea to have pip vendor setuptools crumbles a bit due to console scripts needing pkg_resources. you're left with 2 poor solutions: 1) rewriting script import lines, or 2) still installing setuptools anyway so, having a separate pkg_resources is higher up on the list I think for that reason. without a separate pkg_resources, I think the "dynamic install of setuptools" idea wins out, or no change at all. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API for registering/managing URLs for a package
On Jul 18, 2013, at 7:10 AM, M.-A. Lemburg wrote: > I would like to write a script to automatically register release URLs > for PyPI packages. > > Is the REST API documented somewhere, or is the implementation the > spec ? ;-) > > And related to this: > > Will there be an option to tell PyPI's CDN to cache the release > URL's contents ? I think you are perhaps confused, the use of external URLs on PyPI is formally deprecated. The way you inform the PyPI and the CDN network about your package is you upload it to PyPI. pip 1.4 effectively disables "unsafe" external URLs, and all external URLs will follow soon. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 13:13, Nick Coghlan wrote: > > On 18 Jul 2013 21:48, "Oscar Benjamin" wrote: > >> In another thread you mentioned the idea that someone would build >> without using distutils/setuptools by using a setup.py that simply >> invokes an alternate build system that is build-required by the sdist. >> That's fine for simple cases but how many 'python setup.py 's >> should the setup.py support? > > Please read PEP 426, as I cover this in detail. If anything needs further > clarification, please let me know. Okay, I have actually read that before but I forgot about that bit. It says: ''' In the meantime, the above operations will be handled through the distutils/setuptools command system: python setup.py dist_info python setup.py sdist python setup.py build_ext --inplace python setup.py test python setup.py bdist_wheel ''' That seems a sufficiently minimal set of commands. What I wonder when reading it is whether any other command line options are expected to be supported. For example if the setup.py is using distutils/setuptools then you could do something like: python setup.py sdist --dist-dir=some_dir Should it be explicitly not required that the setup.py should support any other invocation than those listed and should just report success/failure by error code? Also in the event of failure is it the job of setup.py to clean up after itself (since there's no clean command)? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] API for registering/managing URLs for a package
I would like to write a script to automatically register release URLs for PyPI packages. Is the REST API documented somewhere, or is the implementation the spec ? ;-) And related to this: Will there be an option to tell PyPI's CDN to cache the release URL's contents ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Thu, Jul 18, 2013 at 9:27 AM, Paul Moore wrote:
> On 18 July 2013 14:03, Daniel Holth wrote:
>>
>> Abstract
>>
>> This PEP proposes a way to represent the setuptools “entry points”
>> feature in standard Python metadata. Entry points are a useful
>> mechanism for advertising or discovering plugins or other exported
>> functionality without having to depend on the module namespace. Since
>> the feature is used by many existing Python distributions and not
>> everyone wants to use setuptools, it is useful to have a way to
>> represent the functionality that is not tied to setuptools itself.
>
>
> 1. I think that console (and GUI) scripts should be top-level metadata, not
> an extension. Installers need to be able to create wrappers based on these,
> and it is useful data for introspection.
It is an extension so it can be a separate PEP, since there's enough
to talk about in the main PEP. The document tries to write down what
setuptools does in a straightforward json way.
> 2. distlib calls these "exports" and I think that's a better name. But if
> names are all that we argue over, I'm happy :-)
>
> 3. Someone (I think it was PJE) pointed out that having entry points in a
> separate metadata file was good because it allowed a fast check of "does
> this distribution expose entry points?" Note that this isn't a useful thing
> to check for script wrappers, which again argues that those should be
> handled separately.
I am more interested in seeing the installer update some kind of index
like a sqlite database. I don't know if it would be faster since I
haven't tried it.
> 4. You seem to have an extra set of curly braces in a few places. You say
> the value of "entry_points" is a dictionary, but you show it as a set
> containing one dictionary in the set (and of course sets aren't valid JSON).
> I'll assume this is just a typo, and you meant
>
> { …
> “extensions”: {
> “entry_points” : {
> “sqlalchemy.dialects” : [ “mysql = sqlalchemy_mysql:base.dialect”] ,
> ...
> }
> }
> }
>
> 5. What's the logic for having the values (I don't see a good term for these
> - the "mysql = ..." bit above) be a structured string that the user has to
> parse? Either it should be completely free format (which I suspect makes it
> of limited use for introspection, if nothing else) or it should be broken up
> into JSON - no point in having a blob of data that needs parsing in the
> middle of an already structured format!
For one thing you can have more than one mysql = in the same
sqlalchemy.dialects. I think in this instance the string parsing is
probably simpler than defining a more JSONy version.
FWIW the PEP 426 metadata is also full of structured strings.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On 18 July 2013 14:03, Daniel Holth wrote:
> Abstract
>
> This PEP proposes a way to represent the setuptools “entry points”
> feature in standard Python metadata. Entry points are a useful
> mechanism for advertising or discovering plugins or other exported
> functionality without having to depend on the module namespace. Since
> the feature is used by many existing Python distributions and not
> everyone wants to use setuptools, it is useful to have a way to
> represent the functionality that is not tied to setuptools itself.
>
1. I think that console (and GUI) scripts should be top-level metadata, not
an extension. Installers need to be able to create wrappers based on these,
and it is useful data for introspection.
2. distlib calls these "exports" and I think that's a better name. But if
names are all that we argue over, I'm happy :-)
3. Someone (I think it was PJE) pointed out that having entry points in a
separate metadata file was good because it allowed a fast check of "does
this distribution expose entry points?" Note that this isn't a useful thing
to check for script wrappers, which again argues that those should be
handled separately.
4. You seem to have an extra set of curly braces in a few places. You say
the value of "entry_points" is a dictionary, but you show it as a set
containing one dictionary in the set (and of course sets aren't valid
JSON). I'll assume this is just a typo, and you meant
{ …
“extensions”: {
“entry_points” : {
“sqlalchemy.dialects” : [ “mysql = sqlalchemy_mysql:base.dialect”] ,
...
}
}
}
5. What's the logic for having the values (I don't see a good term for
these - the "mysql = ..." bit above) be a structured string that the user
has to parse? Either it should be completely free format (which I suspect
makes it of limited use for introspection, if nothing else) or it should be
broken up into JSON - no point in having a blob of data that needs parsing
in the middle of an already structured format!
Paul
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] entry points PEP
Abstract
This PEP proposes a way to represent the setuptools “entry points”
feature in standard Python metadata. Entry points are a useful
mechanism for advertising or discovering plugins or other exported
functionality without having to depend on the module namespace. Since
the feature is used by many existing Python distributions and not
everyone wants to use setuptools, it is useful to have a way to
represent the functionality that is not tied to setuptools itself.
The proposed feature defines an extension field for the standard
Python distribution metadata and some basic semantics for its use.
Overview
Entry points are represented as an extension in the metadata:
{ …
“extensions”: { “entry_points” : { … } }
}
The extension contains the data in a dictionary format similar to that
accepted by setuptools’ setup(entry_points={}) keyword argument. It is
a dictionary of “group” : [ “key=module.name:attrs.attrs [extra,
extra2, ...]”, ].
“group” is the name of this class of entry points. Values in common
use include “console_scripts” and “sqlalchemy.dialects”. During
discovery, clients usually iterate over all the entry points in a
particular group.
“key” is the name of a particular entry point in the group. It must be
locally unique within this distribution’s group.
“module.name” is the Python module that defines the entry point.
Client code would import the module to use the entry point.
“:attrs.attrs” are the optional attributes of the module that define
the entry point (the module itself can be the entry point). Client
code uses normal attribute access on the imported module to use the
entry point.
“[extra, extra2, ...]” are any optional features of the distribution
(declared with “extras”) that must be installed to use the declared
entry point. This is not common.
Complete example
{ …
“extensions”: { “entry_points” : {
{ “sqlalchemy.dialects” : [ “mysql = sqlalchemy_mysql:base.dialect”] }
} }
}
Use
Client code reads every distribution’s metadata file on sys.path,
filtering for the entry point group name desired, and, if applicable,
the individual entry point name. Once the desired entry point has been
found, a utility function imports the necessary module and attribute
to return an object which can be inspected or called. distlib and
setuptools’ pkg_resources provide APIs for this functionality.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 Jul, 2013, at 13:48, Oscar Benjamin wrote: > On 17 July 2013 22:43, Nick Coghlan wrote: >> >> On 18 Jul 2013 01:46, "Daniel Holth" wrote: >>> >>> On Wed, Jul 17, 2013 at 11:12 AM, Brett Cannon wrote: I'm going to be pushing an update to one of my projects to PyPI this week and so I figured I could use this opportunity to help with patches to the User Guide's packaging tutorial. But to do that I wanted to ask what the current best practices are. * Are we even close to suggesting wheels for source distributions? >>> >>> No, wheels don't replace source distributions at all. They just let >>> you install something without having to have whatever built the wheel >>> from its sdist. It is currently nice to have them available. >>> >>> I'd like to see an ambitious person begin uploading wheels that have >>> no traditional sdist. >> >> Argh, don't even suggest that. Such projects could never be included in a >> Linux distribution - we need the original source to push into a trusted >> build system. > > What do you mean by this? > > I interpret Daniel's comment as meaning that there's no setup.py in > the sdist. And I think it's a great idea and that lots of others would > be very happy to ditch the setup.py concept in favour of something > entirely different from the distutils way of doing things. > > In another thread you mentioned the idea that someone would build > without using distutils/setuptools by using a setup.py that simply > invokes an alternate build system that is build-required by the sdist. > That's fine for simple cases but how many 'python setup.py 's > should the setup.py support? I don't think that's clear at the moment. It could be as little as "bdist_wheel", that could be enough to interface to get from an extracted sdist to a wheel. The current focus is on defining a common metadata format (the metadata 2.0 JSON files) and a binary distribution format, and that's enough to keep the folks doing the actual work occupied for now. In the long run we'll probably end up with something like this: * Sources from a VCS (that is, project in the layout used by those doing development) | [tool specific] | V * sdist archive (sources + metadata.json + ???, to be specified) | [to be specified interface] | V * wheel archive | ["pip", PEP 376(?)] * installed package If I recall correctly the transformation from sdist to wheel is currently not specified because getting the last steps (binary distribution and installation) right is more important right now. The exact format of an sdist, and the interface for specifying how to build a wheel from an sdist is still open for discussion and experimentation. That is, what's the minimal tool that could be used to create wheels for distributions that contain one or more python packages with dependency information? And what would be needed for a more complex distribution with (optional) C extensions, data files, custom compilers, ...? The initial interface to the build system could well be a setup.py file that the build system will only invoke as "python setup.py bdist_wheel --bdist-dir=DIR" (with build-time depedencies specified in the metdata file) because that's easy to arrange for distutils/setuptools, and it should be easy enough to provide a dummy setup.py file with just that interface for alternative build systems. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 Jul 2013 21:48, "Oscar Benjamin" wrote: > > On 17 July 2013 22:43, Nick Coghlan wrote: > > > > On 18 Jul 2013 01:46, "Daniel Holth" wrote: > >> > >> On Wed, Jul 17, 2013 at 11:12 AM, Brett Cannon wrote: > >> > I'm going to be pushing an update to one of my projects to PyPI this > >> > week > >> > and so I figured I could use this opportunity to help with patches to > >> > the > >> > User Guide's packaging tutorial. > >> > > >> > But to do that I wanted to ask what the current best practices are. > >> > > >> > * Are we even close to suggesting wheels for source distributions? > >> > >> No, wheels don't replace source distributions at all. They just let > >> you install something without having to have whatever built the wheel > >> from its sdist. It is currently nice to have them available. > >> > >> I'd like to see an ambitious person begin uploading wheels that have > >> no traditional sdist. > > > > Argh, don't even suggest that. Such projects could never be included in a > > Linux distribution - we need the original source to push into a trusted > > build system. > > What do you mean by this? > > I interpret Daniel's comment as meaning that there's no setup.py in > the sdist. And I think it's a great idea and that lots of others would > be very happy to ditch the setup.py concept in favour of something > entirely different from the distutils way of doing things. No, that's not what he said, he said no sdist at all. Wheel fills the role of a prebuilt binary format, it's not suitable as the *sole* upload format for a project. Tarball, sdist, wheel. Three different artifacts for three different phases of distribution. > In another thread you mentioned the idea that someone would build > without using distutils/setuptools by using a setup.py that simply > invokes an alternate build system that is build-required by the sdist. > That's fine for simple cases but how many 'python setup.py 's > should the setup.py support? Please read PEP 426, as I cover this in detail. If anything needs further clarification, please let me know. Cheers, Nick. > > Setuptools setup() supports the following: > build, build_py, build_ext, build_clib, build_scripts, clean, install, > install_lib, install_headers, install_scripts, install_data, sdist, > register, bdist, bdist_dumb, bdist_rpm, bdist_wininst, upload, check, > rotate, develop, setopt, saveopts, egg_info, upload_docs, > install_egg_info, alias, easy_install, bdist_egg, test > > (Presumably bdist_wheel would be there if I had a newer setuptools). > > > Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 17 July 2013 22:43, Nick Coghlan wrote: > > On 18 Jul 2013 01:46, "Daniel Holth" wrote: >> >> On Wed, Jul 17, 2013 at 11:12 AM, Brett Cannon wrote: >> > I'm going to be pushing an update to one of my projects to PyPI this >> > week >> > and so I figured I could use this opportunity to help with patches to >> > the >> > User Guide's packaging tutorial. >> > >> > But to do that I wanted to ask what the current best practices are. >> > >> > * Are we even close to suggesting wheels for source distributions? >> >> No, wheels don't replace source distributions at all. They just let >> you install something without having to have whatever built the wheel >> from its sdist. It is currently nice to have them available. >> >> I'd like to see an ambitious person begin uploading wheels that have >> no traditional sdist. > > Argh, don't even suggest that. Such projects could never be included in a > Linux distribution - we need the original source to push into a trusted > build system. What do you mean by this? I interpret Daniel's comment as meaning that there's no setup.py in the sdist. And I think it's a great idea and that lots of others would be very happy to ditch the setup.py concept in favour of something entirely different from the distutils way of doing things. In another thread you mentioned the idea that someone would build without using distutils/setuptools by using a setup.py that simply invokes an alternate build system that is build-required by the sdist. That's fine for simple cases but how many 'python setup.py 's should the setup.py support? Setuptools setup() supports the following: build, build_py, build_ext, build_clib, build_scripts, clean, install, install_lib, install_headers, install_scripts, install_data, sdist, register, bdist, bdist_dumb, bdist_rpm, bdist_wininst, upload, check, rotate, develop, setopt, saveopts, egg_info, upload_docs, install_egg_info, alias, easy_install, bdist_egg, test (Presumably bdist_wheel would be there if I had a newer setuptools). Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> It would actually be nice if "pkg_resources" and > "setuptools-core" > were available as separate PyPI distributions, and setuptools bundled > them together with easy_install. This would seem to require quite a sizeable refactoring of setuptools, since the easy_install command is just an entry point for setuptools.command.easy_install.main(). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Fw: Q about best practices now (or near future)
On 18 July 2013 17:44, Vinay Sajip wrote: > Sorry, accidentally left distutils-sig off when replying. Since I already replied off list, there's no way this dual conversation over the same emails will get confusing, nope, uh-uh :) > - Forwarded Message - >> From: Vinay Sajip >> To: Nick Coghlan >> Cc: >> Sent: Thursday, 18 July 2013, 8:41 >> Subject: Re: [Distutils] Q about best practices now (or near future) >> >>> From: Nick Coghlan >> >> >> >>> Then (help) write the missing PEP! PEP's don't appear out of >> nowhere, >> >> I think I have been helping as and when I can by participating in the various >> discussions, I apologise for that crack - you deserve better than that. > but the PEP has to be written by a champion. I clearly can't be >> a champion for this, As I said off-list, I think you might make a good champion for an explicit bootstrapping PEP. I've already stated I'm no longer a fan of implicit bootstrapping at first use, but some valid concerns have been raised about the bundling approach as well. A middle ground where: 1. We provide an explicit bootstrapping script as a "getpip" module 2. This is added to the standard library in 3.4+ 3. This is added to the Python Launcher for Windows for earlier versions 4. The following installers gain a "Boostrap pip?" option: * Python Launcher for Windows * CPython Windows installer * CPython Mac OS X installer >>> these approaches (up to and including misstating that dislike as "not >> >>> going to happen"), that just means the arguments in favour would need >>> to be a bit more persuasive to convince me I am wrong. >> >> That's not how "not going to happen" comes across. You're >> saying it's a misstatement in this off-list mail, but as you are the >> packaging BDFL, some people on-list would just give up when they saw that. Agreed, that's why I'm correcting the record now :) >>> The problem statement also needs to be updated to cover the use case >>> of an instructor running a class and wanting to offer a local PyPI >>> server (or other cache) without a reliable network connection to the >>> outside world, since *that* is the main argument against the >>> bootstrapping based solutions. >> >> >> How widespread is that scenario, really, in this day and age? I consider >> this a >> straw man. If that really is a case to cover, you can make a getpip script >> cover >> this contingency with a command-line argument, the pip and setuptools >> packages >> can be stored on the local PyPI cache, and so on. It's no more onerous than >> explaining to the students, for example, the pip command line parameters you >> would need to specify to access a local PyPI cache. From my experience, over >> the >> course of a class students will run many commands, some of which they don't >> fully understand, under the guidance of the instructor. Yes, an explicit bootstrapping PEP could definitely make a case for being able to handle that scenario. The reason it's important is that I really want for us to be able to provide relatively straightforward instructions to handle the following two cases: * Tutorials, etc, with unreliable conference and other venue uplinks (but reasonable local connectivity or just passing USB keys around) * Remote events where the uplink has a strict download quota, so minimising bandwidth usage is important >> I have to say, I'm not comfortable with the *level* of some of the >> arguments/points put forward - for example, that "we already had a get-pip >> command, using curl URL | python". They come across as unconsidered, more >> like rationalisations for a course already set, and it's hard to engage in a >> debate which doesn't feel right. I think a large part of that is the natural disappointment from thinking we were close to having everything resolved through the implicit bootstrap mechanism, and then realising that wasn't going to happen after all and we still have more work to do. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [tuf] Re: vetting, signing, verification of release files
On Wed, Jul 17, 2013 at 21:46 -0400, Donald Stufft wrote: > As I've mentioned before an online key (as is required by PyPI) means > that if someone compromises PyPI they compromise the key. It seems to > me that TUF is really designed to handle the case of the Linux > distribution (or similar) where you have vetted maintainers who are > given a subsection of the total releases. However PyPI does not have > vetted authors nor the man power to sign authors keys offline. If we had a person with a master key present at Pycon conferences, package maintainers could walk up and have their key signed. Given the many activities of the PSF and the community, i don't think it's off-limits. If we have sig-verified installs, there would be an incentive for authors to go for that little effort. best, holger signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 4:22 AM, Nick Coghlan wrote: > On 18 July 2013 18:10, Paul Moore wrote: >> On 18 July 2013 08:57, Nick Coghlan wrote: >>> >>> Shipping an msi installer for pip (perhaps bundling with setuptools) >>> would also be an acceptable alternative. >> >> >> -1. >> >> I would suggest that this approach, if it were considered seriously, should >> be reviewed carefully by someone who understands MSI installers (not me!). >> Specifically, if I install pip via an MSI, then use "python -m pip install >> -U pip", will the "Add/Remove Programs" entry created by the MSI still >> uninstall cleanly? Broken uninstall options and incomplete package removals >> are a perennial problem on Windows, usually caused by messing with installed >> files outside control of the installer. > > This potential problem needs to be taken into account for any bundling > solution as well. Explicit bootstrapping (with an install time option > to invoke it in the CPython and Python launcher for Windows > installers) is looking better all the time :) That's only a problem if we make a MSI installer. Which I don't think we need to do. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 17:50, Vinay Sajip wrote: >> It's for more reasons than it's in the standard library. setuptools has > >> had a lot of misfeatures and a good bit of the angst against not using >> setuptools was due to easy_install not setuptools itself. > > It's hard to disentangle the two - it's not as if the easy_install > functionality is completely separate, and it's possible to change its > behaviour independently. Another thing about setuptools which some don't > especially like is that generated scripts reference pkg_resources, for no > particularly good reason. It would actually be nice if "pkg_resources" and "setuptools-core" were available as separate PyPI distributions, and setuptools bundled them together with easy_install. It's a *long* way down the priority list thing (and will likely never make it to the top, although it may be more practical once pip vendors the bits it needs). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 18:10, Paul Moore wrote: > On 18 July 2013 08:57, Nick Coghlan wrote: >> >> Shipping an msi installer for pip (perhaps bundling with setuptools) >> would also be an acceptable alternative. > > > -1. > > I would suggest that this approach, if it were considered seriously, should > be reviewed carefully by someone who understands MSI installers (not me!). > Specifically, if I install pip via an MSI, then use "python -m pip install > -U pip", will the "Add/Remove Programs" entry created by the MSI still > uninstall cleanly? Broken uninstall options and incomplete package removals > are a perennial problem on Windows, usually caused by messing with installed > files outside control of the installer. This potential problem needs to be taken into account for any bundling solution as well. Explicit bootstrapping (with an install time option to invoke it in the CPython and Python launcher for Windows installers) is looking better all the time :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 08:57, Nick Coghlan wrote: > Shipping an msi installer for pip (perhaps bundling with setuptools) > would also be an acceptable alternative. > -1. I would suggest that this approach, if it were considered seriously, should be reviewed carefully by someone who understands MSI installers (not me!). Specifically, if I install pip via an MSI, then use "python -m pip install -U pip", will the "Add/Remove Programs" entry created by the MSI still uninstall cleanly? Broken uninstall options and incomplete package removals are a perennial problem on Windows, usually caused by messing with installed files outside control of the installer. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 16:45, Paul Moore wrote:
> On 18 July 2013 02:03, Donald Stufft wrote:
>>
>> it's simple to upgrade the pip if the user requires a newer version
>> of pip using ``pip install --upgrade pip`
>
>
> Please don't gloss over the potential issues with upgrading in the face of
> in-use exe wrappers. We have a design for a solution, but as yet no working
> code. I expect to work on this, but my time is limited and I'm not at all
> sure there won't be issues still to resolve. (Obviously, anyone else is
> welcome to help, but it's a "windows issue", so I don't know how much
> interest there will be from non-Windows developers).
>
> Prior to the setuptools move away from 2to3, my standard response to anyone
> reporting issues with in-place upgrades of setuptools or pip (certainly on
> Windows, and in general anywhere else too) was "well, don't do that - remove
> and reinstall manually". Things are better now, but not yet perfect and I
> don't believe that there is a consensus that this is acceptable for a
> bundled pip.
Making in-place upgrades using "pip install --upgrade pip" reliable on
Windows is definitely the preferred solution, but it isn't a show
stopper if it isn't ready for 3.4.
Requiring that in-place upgrades be run as "python -m pip install
--upgrade pip" would be acceptable, so long as the direct invocation
("pip install --upgrade pip") was detected and a clear error thrown
suggesting the other command (this would be mildly annoying, but it's
still a substantial improvement over the status quo). Something like:
"Due to an unfortunate limitation of pip on Windows, direct upgrades
are not supported. Please run 'python -m pip install --upgrade pip' to
work around the problem."
Shipping an msi installer for pip (perhaps bundling with setuptools)
would also be an acceptable alternative. Bundling both with the
"Python launcher for Windows" installer is definitely something we
should consider for older versions (rather than updating the CPython
installer). Either way, Windows users are used to downloading and
running installers to get Python upgrades :)
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
> It's for more reasons than it's in the standard library. setuptools has > had a lot of misfeatures and a good bit of the angst against not using > setuptools was due to easy_install not setuptools itself. It's hard to disentangle the two - it's not as if the easy_install functionality is completely separate, and it's possible to change its behaviour independently. Another thing about setuptools which some don't especially like is that generated scripts reference pkg_resources, for no particularly good reason. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 18 July 2013 08:29, Donald Stufft wrote: > Personally I think pip should be vendoring setuptools regardless. A > package manager with dependencies is strange and there have been quite a > few problems caused by setuptools getting in a bad state. Agrred on the dependency point (but I don't consider "depends on something bundled with Python" as being an external dependency, hence my question). As regards vendoring, I'm reserving judgement until I see the code - I think getting something working is more important than discussing what might be hard to implement... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Fw: Q about best practices now (or near future)
Sorry, accidentally left distutils-sig off when replying. - Forwarded Message - > From: Vinay Sajip > To: Nick Coghlan > Cc: > Sent: Thursday, 18 July 2013, 8:41 > Subject: Re: [Distutils] Q about best practices now (or near future) > >> From: Nick Coghlan > > > >> Then (help) write the missing PEP! PEP's don't appear out of > nowhere, > > I think I have been helping as and when I can by participating in the various > discussions, but the PEP has to be written by a champion. I clearly can't be > a champion for this, else why would I be working on distlib? That's what I > currently see as the way forward, obviously, but it's premature to look at a > PEP for it because it hasn't had enough exposure or peer review. > > I have no particular axe to grind against pip - I did a lot of the core work > for > the single code-base port, speeded up the test suite a fair bit, and have > contributed other bits and bobs. However, it is the past and present of > packaging, as I see it, and not a worthy long-term future - it has too much > technical debt. As the de facto installer for Python, pip needs no additional > new endorsement, in my view. If I had to choose, I would say I find none of > the > choices especially appetising, but I would choose an explicit bootstrap over > the > others. Note that installing Distribute/pip was explicitly removed from the > pyvenv script before 3.3 beta, because of python-dev concerns about promoting > specific third-party solutions in the stdlib (even though they were the > defacto > tools for Python 3.x, and endorsed as such by python-dev).. > > Nothing has essentially changed from the 3.3 beta time frame. People still > use > pip, just as they always did. The recommendation from python-dev is as it > always > was (use pip), with a slight alteration on the Distribute front due to the > merge > with setuptools. Neither pip nor setuptools are *significantly* better than > they > were in functional terms, and if they weren't the right solution when > distutils2/packaging was mooted, I don't see why that should have changed > now. > >> these approaches (up to and including misstating that dislike as "not > >> going to happen"), that just means the arguments in favour would need >> to be a bit more persuasive to convince me I am wrong. > > That's not how "not going to happen" comes across. You're > saying it's a misstatement in this off-list mail, but as you are the > packaging BDFL, some people on-list would just give up when they saw that. > >> The problem statement also needs to be updated to cover the use case >> of an instructor running a class and wanting to offer a local PyPI >> server (or other cache) without a reliable network connection to the >> outside world, since *that* is the main argument against the >> bootstrapping based solutions. > > > How widespread is that scenario, really, in this day and age? I consider this > a > straw man. If that really is a case to cover, you can make a getpip script > cover > this contingency with a command-line argument, the pip and setuptools > packages > can be stored on the local PyPI cache, and so on. It's no more onerous than > explaining to the students, for example, the pip command line parameters you > would need to specify to access a local PyPI cache. From my experience, over > the > course of a class students will run many commands, some of which they don't > fully understand, under the guidance of the instructor. > > I have to say, I'm not comfortable with the *level* of some of the > arguments/points put forward - for example, that "we already had a get-pip > command, using curl URL | python". They come across as unconsidered, more > like rationalisations for a course already set, and it's hard to engage in a > debate which doesn't feel right. > > Regards, > > Vinay Sajip > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Jul 18, 2013, at 3:20 AM, Paul Moore wrote: > On 17 July 2013 23:18, Brett Cannon wrote: > As of right now the User's Guide doesn't mention using setuptools for > building (beyond an empty header listing) and goes with the old distutils > setup.py approach. It also words things like you don't know how to really use > Python and are starting a project entirely from scratch. > > Just picking up on this question: > 1. As Brett says, is the recommendation that everyone should use setuptools? > 2. If that's the case, why aren't we bundling setuptools in the same way that > we are bundling pip? > 3. If we were bundling setuptools, pip wouldn't need to go through the > rigmarole of vendoring it. Personally I think pip should be vendoring setuptools regardless. A package manager with dependencies is strange and there have been quite a few problems caused by setuptools getting in a bad state. > > Paul. > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On 17 July 2013 23:18, Brett Cannon wrote: > As of right now the User's Guide doesn't mention using setuptools for > building (beyond an empty header listing) and goes with the old distutils > setup.py approach. It also words things like you don't know how to really > use Python and are starting a project entirely from scratch. > Just picking up on this question: 1. As Brett says, is the recommendation that everyone should use setuptools? 2. If that's the case, why aren't we bundling setuptools in the same way that we are bundling pip? 3. If we were bundling setuptools, pip wouldn't need to go through the rigmarole of vendoring it. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
