Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Nick Coghlan
Thanks for pointing that out Noah - I was planning to come back and check if that wording was considered more acceptable. On 31 Jan 2014 07:12, "Donald Stufft" wrote: > > The updated text is fine with me as it at least accurately documents the fact that using that > feature is fraught with peril

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Donald Stufft
On Jan 30, 2014, at 1:21 PM, Noah Kantrowitz wrote: > > On Jan 30, 2014, at 1:09 AM, Vinay Sajip wrote: > >> >> >> On Thu, 30/1/14, Ralf Gommers wrote: >> >>> Also end user. >>> If, as a user, I want to use inplace builds and PYTHONPATH >>> inst

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Noah Kantrowitz
On Jan 30, 2014, at 1:09 AM, Vinay Sajip wrote: > > > On Thu, 30/1/14, Ralf Gommers wrote: > >> Also end user. >> If, as a user, I want to use inplace builds and PYTHONPATH >> instead of virtualenvs for whatever reason, that should be >> supported.

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Vinay Sajip
On Thu, 30/1/14, Ralf Gommers wrote: > Also end user. > If, as a user, I want to use inplace builds and PYTHONPATH > instead of virtualenvs for whatever reason, that should be > supported. Setuptools inserting stuff to sys.path that come > before PYTH

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Ralf Gommers
On Wed, Jan 29, 2014 at 5:22 PM, Vinay Sajip wrote: > > On Wed, 29/1/14, Paul Moore wrote: > > > That package installation utilities should not dabble in sys.path > manipulation. > > The import path is the user's responsibility. > > User as in develope

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 30 Jan 2014 07:50, "Greg Ewing" wrote: > > Nick Coghlan wrote: >> >> Otherwise we'd have to define a whole new format for something that >> can be adequately handled by a wheel that meets certain restrictions, >> and that would be pointless (we already have too many formats, and we >> wanted th

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Greg Ewing
Nick Coghlan wrote: Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg format'

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Matthew Iversen
On 30/01/14 00:59, Nick Coghlan wrote: > On 29 January 2014 23:48, Donald Stufft wrote: >> So what did you mean when you said "We discussed it extensively before >> PEP 427 was approved" if you're now saying that it wasn't discussed. > "Explicitly" would be a better word: > https://mail.python.org

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
On Wed, 29/1/14, Paul Moore wrote: > That designing modules to support zipimport correctly is non-trivial. It's not trivial, but it's not especially hard, either. Mostly, it's about remembering to consider zipimport, since that hasn't been a mainstrea

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 14:25, Vinay Sajip wrote: > What *else* have we learned from eggs? That designing modules to support zipimport correctly is non-trivial. And that assuming that things work unless told otherwise is a bad default behaviour. That packaging solutions should not depend on runtime s

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 10:00 AM, Vinay Sajip wrote: > > > On Wed, 29/1/14, Donald Stufft wrote: > >> It’s hard to pin down because the failure modes of zipped >> eggs are nebulous themselves. > >> For instance take pip. I just recently redid the get-

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 13:31, Donald Stufft wrote: > Here's Paul explicitly mentioning that Wheels being used with zip import is > an incidental benefit and not a core feature. > > https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html That's quoted out of context. It was in a thread

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
> I went through this with Chris McDonough back when packaging was > dropped from 3.3, and he really helped me focus on what I found to be > the two closely related core problems: - implicit sys.path manipulation - installing as eggs by default > That was due to easy_install defaults being > ch

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
On Wed, 29/1/14, Donald Stufft wrote: > It’s hard to pin down because the failure modes of zipped > eggs are nebulous themselves. > For instance take pip. I just recently redid the get-pip.py > installer to use a zip file (not a Wheel or Egg) that c

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 30 Jan 2014 00:28, "Vinay Sajip" wrote: > > > It may be useful to understand that wheel has *political features* or > > if you prefer *setting the defaults based on what we have learned from > > eggs*. I don't recommend that they be zip-imported generally > > but if you are a consenting adult w

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Brett Cannon
On Wed, Jan 29, 2014 at 9:40 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 9:25 AM, Vinay Sajip wrote: > > >> It may be useful to understand that wheel has *political features* or > >> if you prefer *setting the defaults based on what we have learned from > >> eggs*. I don't recommend that th

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
On Wed, 29/1/14, Donald Stufft wrote: > Mitre’s rules for CVEs are not entirely obvious to people who are not > familiar with them. Generally if the feature *can* be used securely or > there was no evidence that the author intended that the code be s

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 9:25 AM, Vinay Sajip wrote: >> It may be useful to understand that wheel has *political features* or >> if you prefer *setting the defaults based on what we have learned from >> eggs*. I don't recommend that they be zip-imported generally >> but if you are a consenting adult

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
> It may be useful to understand that wheel has *political features* or > if you prefer *setting the defaults based on what we have learned from > eggs*. I don't recommend that they be zip-imported generally > but if you are a consenting adult who understands the caveats you > may do so. What *exa

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:59 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> I believe Paul's concern is with anything that suggests that arbitrary >> *third party* code can be run from wheel files, when the reality is >> that it is fairly easy to accidentally write code that assumes

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:59, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: >> goes further than the current EXTENSIONS approach - this proposal >> would be akin to *requiring* an empty EXTENSIONS file, and/or the >> setuptools zip_safe flag in order to allow mounting of even the pure >> Pytho

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:59 AM, Nick Coghlan wrote: > On 29 January 2014 23:48, Donald Stufft wrote: >> >> So what did you mean when you said "We discussed it extensively before >> PEP 427 was approved" if you're now saying that it wasn't discussed. > > "Explicitly" would be a better word: > htt

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
Nick Coghlan gmail.com> writes: > I believe Paul's concern is with anything that suggests that arbitrary > *third party* code can be run from wheel files, when the reality is > that it is fairly easy to accidentally write code that assumes it is > installed on the filesystem in a way that isn't e

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:48, Donald Stufft wrote: > > So what did you mean when you said "We discussed it extensively before > PEP 427 was approved" if you're now saying that it wasn't discussed. "Explicitly" would be a better word: https://mail.python.org/pipermail/distutils-sig/2012-September/0189

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:34, Donald Stufft wrote: > > If I'm wrong, then by all means show me where it was discussed so I can admit > I was wrong. You have the burden of proof backwards there. You're the one asking me to break backwards compatibility, and to let people continue to believe wheels are

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Daniel Holth
On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: > >> On 29 January 2014 12:58, Nick Coghlan wrote: >>> You can campaign to deprecate that feature *if* we ever want to make a >>> change to the wheel format that is incompatible with it. >> >

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:46 AM, Nick Coghlan wrote: > On 29 January 2014 23:34, Donald Stufft wrote: >> >> If I'm wrong, then by all means show me where it was discussed so I can admit >> I was wrong. > > You have the burden of proof backwards there. You're the one asking me > to break backwards

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Daniel Holth
On Wed, Jan 29, 2014 at 8:44 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 8:43 AM, Daniel Holth wrote: > >> On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft wrote: >>> >>> On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: >>> On 29 January 2014 12:58, Nick Coghlan wrote: > You can camp

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:43 AM, Daniel Holth wrote: > On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft wrote: >> >> On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: >> >>> On 29 January 2014 12:58, Nick Coghlan wrote: You can campaign to deprecate that feature *if* we ever want to make a

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:34 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 8:32 AM, Nick Coghlan wrote: > >> On 29 January 2014 23:31, Donald Stufft wrote: >>> >>> Here's Paul explicitly mentioning that Wheels being used with zip import is >>> an incidental benefit and not a core feature. >>

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Matthew Iversen
On the one hand, it's easy to see the allure of a zipimport-able format. Grab a file, import it, use functionality straight away. It has an attraction of self-containedness. On the other hand, python's dynamic nature means that things are not as simple as Java jar's as Donald points out. I agree th

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:32 AM, Nick Coghlan wrote: > On 29 January 2014 23:31, Donald Stufft wrote: >> >> Here's Paul explicitly mentioning that Wheels being used with zip import is >> an incidental benefit and not a core feature. >> >> https://mail.python.org/pipermail/distutils-sig/2013-March

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:31 AM, Nick Coghlan wrote: > On 29 January 2014 23:16, Donald Stufft wrote: >> So basically even though the text of the PEP specifically points out that a >> difference >> of Wheel and Egg is that Eggs are importable it somehow supports that? Can >> you >> point to a sin

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:31, Donald Stufft wrote: > > Here's Paul explicitly mentioning that Wheels being used with zip import is > an incidental benefit and not a core feature. > > https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html Donald, I was *there*. I know what we discussed

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:16, Donald Stufft wrote: > So basically even though the text of the PEP specifically points out that a > difference > of Wheel and Egg is that Eggs are importable it somehow supports that? Can you > point to a single line in the PEP that supports this besides the ones you've

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:10 AM, Paul Moore wrote: > On 29 January 2014 12:58, Nick Coghlan wrote: >> You can campaign to deprecate that feature *if* we ever want to make a >> change to the wheel format that is incompatible with it. > > Note that virtualenv uses the ability to run wheels from sys.

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:17 AM, Nick Coghlan wrote: > On 29 January 2014 23:10, Paul Moore wrote: >> On 29 January 2014 12:58, Nick Coghlan wrote: >>> You can campaign to deprecate that feature *if* we ever want to make a >>> change to the wheel format that is incompatible with it. >> >> Note th

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:10, Paul Moore wrote: > On 29 January 2014 12:58, Nick Coghlan wrote: >> You can campaign to deprecate that feature *if* we ever want to make a >> change to the wheel format that is incompatible with it. > > Note that virtualenv uses the ability to run wheels from sys.path.

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:14 AM, Nick Coghlan wrote: > On 29 January 2014 22:57, Donald Stufft wrote: >> Also to be specific, what I believe should be done is that the change should >> be >> reverted, and if it is desired that Wheels officially support zip import >> then in the >> next version of

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 22:57, Donald Stufft wrote: > Also to be specific, what I believe should be done is that the change should > be > reverted, and if it is desired that Wheels officially support zip import then > in the > next version of the Wheel spec it should be added when everyone has a chan

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 12:58, Nick Coghlan wrote: > You can campaign to deprecate that feature *if* we ever want to make a > change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv should be a "special

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 7:58 AM, Nick Coghlan wrote: > On 29 January 2014 21:50, Donald Stufft wrote: >> >> As a follow up, even if this becomes something we document and actually >> support, then it should be done in the next version of Wheel when people >> have the chance to discuss it. It shoul

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 21:50, Donald Stufft wrote: > > As a follow up, even if this becomes something we document and actually > support, then it should be done in the next version of Wheel when people > have the chance to discuss it. It should not be added ex post facto by > a committer to the PEP re

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 7:43 AM, Paul Moore wrote: > On 29 January 2014 12:18, Donald Stufft wrote: >> But more importantly, while I'm against officially supporting importable >> Wheels >> because of various reasons, my biggest problem is that this was added in >> without any chance for discussion

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 12:18, Donald Stufft wrote: > But more importantly, while I'm against officially supporting importable > Wheels > because of various reasons, my biggest problem is that this was added in > without any chance for discussion. Fair point. I'm sure Nick did just think of it as cla

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 6:59 AM, Paul Moore wrote: > On 29 January 2014 11:41, Donald Stufft wrote: >> >> On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: >> >>> As I recall, it was in the original version of the PEP/spec and it was >>> always an intended feature. The wheel file for the wheel proj

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 11:41, Donald Stufft wrote: > > On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: > >> As I recall, it was in the original version of the PEP/spec and it was >> always an intended feature. The wheel file for the wheel project >> itself is (deliberately) runnable as a zip file, so

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
That doesn't really speak to the fact they were designed for that. If I read that it looks like some commenting that they are importable (which as it stands they are) and not someone calling it a supported feature of the format. It even states that the format is designed primarily for distributi

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Chris Jerdonek
On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: > >> As I recall, it was in the original version of the PEP/spec and it was >> always an intended feature. The wheel file for the wheel project >> itself is (deliberately) runnable as a zip fi

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 6:47 AM, Donald Stufft wrote: > > On Jan 29, 2014, at 2:32 AM, Nick Coghlan wrote: > >> All of those arguments are against *recommending* directly importing >> from wheels. Yes, there are lots of problems with running directly >> from a zip archive, which is why the fact t

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 2:32 AM, Nick Coghlan wrote: > All of those arguments are against *recommending* directly importing > from wheels. Yes, there are lots of problems with running directly > from a zip archive, which is why the fact that easy_install *actively > encourages* potentially inexperie

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 10:36, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> 1. It does *not* just use the fact that wheels are importable. It goes >> beyond that and *extracts* C extensions to make them importable, too. >> That is a workaround for a known and accepted limitation of zipimpor

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 4:23 AM, Paul Moore wrote: > As I recall, it was in the original version of the PEP/spec and it was > always an intended feature. The wheel file for the wheel project > itself is (deliberately) runnable as a zip file, so that you can > bootstrap into the wheel world using the

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 20:36, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > > You don't want it to be there, even if it might be useful to others, > just because it isn't useful to you? It's not as if distlib is forcing > that functionality on anyone. I believe Paul's concern is with anything

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
Paul Moore gmail.com> writes: > 1. It does *not* just use the fact that wheels are importable. It goes > beyond that and *extracts* C extensions to make them importable, too. > That is a workaround for a known and accepted limitation of zipimport, > and as a workaround, it has issues. If C extens

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 04:14, Vinay Sajip wrote: > When this topic came up before, I asked for specific failure modes which > were causing concern, but I never got a response IIRC. >From what I recall, the topic came up before in relation to distlib's wheel mount functionality. My specific concerns

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 05:52, Donald Stufft wrote: > Which further shows that at the time it was "cool that it worked" but that > the "weird failures" being a reason that Wheel was an installation format. Having "cool" features in a format is not a bad thing. Zip imports are the "cool " feature here

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
Donald Stufft stufft.io> writes: > 1. That unpacking into the home directory is problematic because users > that run services often don’t have home directories. This you waved > away by saying that the home directory isn’t a required place, to > which Jim responded that unpacking them

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Nick Coghlan
All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic. Howeve

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Donald Stufft
On Jan 28, 2014, at 11:14 PM, Vinay Sajip wrote: > Donald Stufft stufft.io> writes: > >> I think we need to reconsider this. I cannot stress how badly an idea >> I think it is for Wheels to concern itself with the ability to add the >> Wheel to sys.path and import it. > > I know that people h

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Donald Stufft
On Jan 28, 2014, at 11:54 PM, Nick Coghlan wrote: > > On 29 Jan 2014 11:19, "Daniel Holth" wrote: > > > > On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft wrote: > > > > > > On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > > > > > > I think you're reading too much into that comment. Putting

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Nick Coghlan
On 29 Jan 2014 11:19, "Daniel Holth" wrote: > > On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft wrote: > > > > On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > > > > I think you're reading too much into that comment. Putting a wheel file > > directly on sys.path is no different from putting any

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Vinay Sajip
Donald Stufft stufft.io> writes: > I think we need to reconsider this. I cannot stress how badly an idea > I think it is for Wheels to concern itself with the ability to add the > Wheel to sys.path and import it. I know that people have had bad experiences in the past with eggs, for the reasons

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Daniel Holth
On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft wrote: > > On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > > I think you're reading too much into that comment. Putting a wheel file > directly on sys.path is no different from putting any other zipfile directly > on sys.path - whether or not it w

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Donald Stufft
On Jan 28, 2014, at 6:38 PM, Nick Coghlan wrote: > I think you're reading too much into that comment. Putting a wheel file > directly on sys.path is no different from putting any other zipfile directly > on sys.path - whether or not it will work depends on the context, but it's a > useful cap

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Nick Coghlan
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip implemen