Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-09-19 Thread barry croff
___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-03 Thread Brett Cannon
On Sat, Feb 1, 2014 at 3:00 PM, Vinay Sajip wrote: > On Sat, 1/2/14, Brett Cannon wrote: > > > No one is talking about solving this problem overnight. > > But if we don't want to bother putting the effort in now > > then the zipimport module might as well get tossed as > > this is a constant iss

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Brett Cannon wrote: > No one is talking about solving this problem overnight. > But if we don't want to bother putting the effort in now > then the zipimport module might as well get tossed as > this is a constant issue that people are not designing APIs > to support. Whoa - nobo

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Brett Cannon
On Sat, Feb 1, 2014 at 2:00 PM, Vinay Sajip wrote: > On Sat, 1/2/14, Brett Cannon wrote: > > > Yes, that is definitely a design flaw in the ssl module that should > > get remedied. Did you file a bug to add a new API (whether new > > function or new parameters) to accept a file-like object or st

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Christian Heimes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 01.02.2014 19:06, Brett Cannon wrote: > Yes, that is definitely a design flaw in the ssl module that should > get remedied. Did you file a bug to add a new API (whether new > function or new parameters) to accept a file-like object or string > ins

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Donald Stufft wrote: > distil isn't licensed so that I can even legally *use* it as > far as I understand the law. >From a quick Google search, I got the following from Daniel J. Bernstein's >website at http://cr.yp.to/softwarelaw.html - the page entitled "Software >user's righ

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Donald Stufft wrote: > distil isn't licensed so that I can even legally *use* it as far as I > understand the law. Oh really? I hadn't realised. I had thought my announcement in March inviting people to try it out would suffice, but IANAL. Having a single-file deployment means ha

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Brett Cannon wrote: > Yes, that is definitely a design flaw in the ssl module that should > get remedied. Did you file a bug to add a new API (whether new > function or new parameters) to accept a file-like object or string > instead? While that might improve flexibility in how

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Noah Kantrowitz wrote: > Junctions on Windows are actually more flexible than POSIX symlinks, > and I have in fact used them for Python packages before when doing > Django dev on Windows XP. Why do you say they're more flexible? They work as directories/mount points and AFAIK you

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Brett Cannon
On Sat, Feb 1, 2014 at 1:16 PM, Brett Cannon wrote: > > > > On Sat, Feb 1, 2014 at 1:00 PM, Paul Moore wrote: > >> On 1 February 2014 17:53, Vinay Sajip wrote: >> >> My pref is [...] and really pushing people to use the loader APIs >> >> for reading intra-package "files". >> > >> > +1 >> > >> >

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Brett Cannon
On Sat, Feb 1, 2014 at 1:16 PM, Brett Cannon wrote: > > > > On Sat, Feb 1, 2014 at 1:00 PM, Paul Moore wrote: > >> On 1 February 2014 17:53, Vinay Sajip wrote: >> >> My pref is [...] and really pushing people to use the loader APIs >> >> for reading intra-package "files". >> > >> > +1 >> > >> >

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Noah Kantrowitz
On Feb 1, 2014, at 1:36 AM, Vinay Sajip wrote: > On Sat, 1/2/14, Noah Kantrowitz wrote: > >> In all but a tiny number of cases, you could use a symlink for this. >> Much less magic :-) > > That's "POSIX is all there is" myopia, right there. While recent versions > of Windows have symlinks mor

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Brett Cannon
On Sat, Feb 1, 2014 at 1:00 PM, Paul Moore wrote: > On 1 February 2014 17:53, Vinay Sajip wrote: > >> My pref is [...] and really pushing people to use the loader APIs > >> for reading intra-package "files". > > > > +1 > > > >> Maybe the Packaging Users Guide could have a > >> Recommended Deploy

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Donald Stufft
-- Donald Stufft don...@stufft.io On Sat, Feb 1, 2014, at 01:06 PM, Brett Cannon wrote: On Sat, Feb 1, 2014 at 12:34 PM, Donald Stufft <[1]don...@stufft.io> wrote: On Sat, Feb 1, 2014, at 12:26 PM, Brett Cannon wrote: On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip <[2]vinay_sa...@yah

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Brett Cannon
On Sat, Feb 1, 2014 at 12:34 PM, Donald Stufft wrote: > On Sat, Feb 1, 2014, at 12:26 PM, Brett Cannon wrote: > > > > > On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip wrote: > > > On Fri, 31/1/14, Brian Wickman wrote: > > > There are myriad other practical reasons. Here are some: > > Thanks for

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Donald Stufft
On Sat, Feb 1, 2014, at 12:40 PM, Vinay Sajip wrote: > On Sat, 1/2/14, Donald Stufft wrote: > > > I never said anything gets magically better on GitHub, I > > told you the reason why *I* haven’t bothered to try it. > > And I'm OK with that. > > > Distlib is a significantly different case, it’s

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Paul Moore
On 1 February 2014 17:40, Vinay Sajip wrote: >> I never said anything gets magically better on GitHub, I >> told you the reason why *I* haven't bothered to try it. > > And I'm OK with that. FWIW, the difficulty of seeing the source of distlil is one of the reasons I've not done more with it. Not

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Donald Stufft wrote: > There's also the problem when you need to give a file that you > have packaged as part of your thing to an API that only accepts > filenames. The standard ssl module is one of these that i've run > into recently. Yes, this is one of the pkgutil gaps I refer

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Paul Moore
On 1 February 2014 17:53, Vinay Sajip wrote: >> My pref is [...] and really pushing people to use the loader APIs >> for reading intra-package "files". > > +1 > >> Maybe the Packaging Users Guide could have a >> Recommended Deployment Strategies section > [...] >> (especially if the instructions w

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Paul Moore
On 1 February 2014 17:26, Brett Cannon wrote: > PEP 302 tried to unify this with get_data() and set_data() on loaders, but > prior to Python 3.3 you just didn't have any guarantee that __loader__ would > even be set, let alone have a loader with those methods. Paul can tell me if > my hunch is off

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Brett Cannon wrote: > PEP 302 tried to unify this with get_data() and set_data() > on loaders, but prior to Python 3.3 you just didn't have any > guarantee that __loader__ would even be set, let alone > have a loader with those methods. Paul can tell me if my > hunch is off, but I

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Donald Stufft wrote: > I never said anything gets magically better on GitHub, I > told you the reason why *I* haven’t bothered to try it. And I'm OK with that. > Distlib is a significantly different case, it’s a library that is going to be > directly useful for a very small pop

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Donald Stufft
On Sat, Feb 1, 2014, at 12:26 PM, Brett Cannon wrote: On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip <[1]vinay_sa...@yahoo.co.uk> wrote: On Fri, 31/1/14, Brian Wickman <[2]wick...@gmail.com> wrote: > There are myriad other practical reasons. Here are some: Thanks for taking the time to res

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Brett Cannon
On Sat, Feb 1, 2014 at 3:23 AM, Vinay Sajip wrote: > On Fri, 31/1/14, Brian Wickman wrote: > > > There are myriad other practical reasons. Here are some: > > Thanks for taking the time to respond with the details - they are good > data points > to think about! > > > Lastly, there are social rea

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Donald Stufft
On Feb 1, 2014, at 11:50 AM, Vinay Sajip wrote: > On Sat, 1/2/14, Donald Stufft wrote: > >> Additionally I'd point out that, looking at how you've >> implemented it, it seems to depend on the fact that distil >> is a single file and thus doesn't need installed into >> site-packages. > > No, I

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Donald Stufft wrote: > Additionally I'd point out that, looking at how you've > implemented it, it seems to depend on the fact that distil > is a single file and thus doesn't need installed into > site-packages. No, I don't believe that's the case - I developed (and continue to d

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Donald Stufft wrote: > So perhaps it isn't that pip is lacking a useful feature, it's that > pip tried it, as you did with distil, and due to pip's much larger > user base it got a much more thorough real world experience > with it and it turned out to be ultimately more trouble t

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Donald Stufft
On Feb 1, 2014, at 10:45 AM, Donald Stufft wrote: > It predates my involvement with pip, but pip used to have the feature where it > did not require being installed into the virtualenv. It had the -E > path/to/venv > flag, and also had a PIP_RESPECT_VIRTUALENV env var that would have it > autom

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Donald Stufft
It predates my involvement with pip, but pip used to have the feature where it did not require being installed into the virtualenv. It had the -E path/to/venv flag, and also had a PIP_RESPECT_VIRTUALENV env var that would have it automatically install into the activated virtualenv. This was remove

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Nick Coghlan wrote: > I have no idea how most of the internal design decisions on > pip were made (and I neither need nor particularly want to > know - that would be rather missing the point of collaborative > development). However, you're trying to make out that the > distil appr

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Nick Coghlan
On 1 February 2014 22:20, Vinay Sajip wrote: > > On Sat, 1/2/14, Nick Coghlan wrote: > >> My point is that doing it the way virtualenv/pip did avoided a bunch >> of design work and associated testing, and reduced the >> opportunities for bugs - when you're trying to get things done with >> limite

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Nick Coghlan wrote: > My point is that doing it the way virtualenv/pip did avoided a bunch > of design work and associated testing, and reduced the > opportunities for bugs - when you're trying to get things done with > limited resources, that's a sensible engineering trade-off t

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Paul Moore wrote: > I would like it if pip worked the same way (even though the > tide of installing pip into each virtualenv may be unstoppable > by now). Would you be interested in writing a patch to do this? > I don't have the time to commit to it and nobody else seems > partic

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Nick Coghlan
On 1 February 2014 20:49, Vinay Sajip wrote: > So, I am at a loss to see your point. My point is that doing it the way virtualenv/pip did avoided a bunch of design work and associated testing, and reduced the opportunities for bugs - when you're trying to get things done with limited resources, t

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Paul Moore
On 1 February 2014 09:29, Vinay Sajip wrote: >> However, it's substantially *harder* to explain to people how >> to use it correctly that way. > > But distil installs into venvs (both PEP 405 and virtualenv) > without needing to be in there. It's as simple as > > distil -e path/to/venv install XYZ

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Paul Moore
On 1 February 2014 09:36, Vinay Sajip wrote: > While recent versions > of Windows have symlinks more like POSIX symlinks, XP only has a > stunted version called "reparse points" or "junction points" which are > not really fit for purpose. I think you'll find that XP environments are found > in rat

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Paul Moore
1 February 2014 08:50, Noah Kantrowitz wrote: > In all but a tiny number of cases, you could use a symlink for this. Much > less magic :-) Windows isn't a "tiny number" of cases, unfortunately :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@pyth

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Nick Coghlan wrote: > I'm talking about easing the cognitive burden on developers. > The pip/virtualenv model is that once an environment is > activated, developers don't need to think about it anymore - > they're in that environment until they explicitly switch it off. But dist

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Nick Coghlan
On 1 February 2014 19:29, Vinay Sajip wrote: > On Sat, 1/2/14, Nick Coghlan wrote: > >> FWIW, installing into a venv from outside it works fine > > That's what I'm saying :-) > >> However, it's substantially *harder* to explain to people how >> to use it correctly that way. > > But distil install

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Noah Kantrowitz wrote: > In all but a tiny number of cases, you could use a symlink for this. > Much less magic :-) That's "POSIX is all there is" myopia, right there. While recent versions of Windows have symlinks more like POSIX symlinks, XP only has a stunted version called "r

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Nick Coghlan
On 1 February 2014 18:50, Noah Kantrowitz wrote: > On Feb 1, 2014, at 12:43 AM, Nick Coghlan wrote: >> That said, something I mentioned to the OpenStack folks a while ago >> (and I think on this list, but potentially not), is that I have now >> realised the much-reviled (for good reason) *.pth fi

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Sat, 1/2/14, Nick Coghlan wrote: > FWIW, installing into a venv from outside it works fine That's what I'm saying :-) > However, it's substantially *harder* to explain to people how > to use it correctly that way. But distil installs into venvs (both PEP 405 and virtualenv) without needing

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Noah Kantrowitz
On Feb 1, 2014, at 12:43 AM, Nick Coghlan wrote: > On 1 February 2014 18:23, Vinay Sajip wrote: >> On Fri, 31/1/14, Brian Wickman wrote: >> >>> There are myriad other practical reasons. Here are some: >> >> Thanks for taking the time to respond with the details - they are good data >> poin

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Nick Coghlan
On 1 February 2014 18:23, Vinay Sajip wrote: > On Fri, 31/1/14, Brian Wickman wrote: > >> There are myriad other practical reasons. Here are some: > > Thanks for taking the time to respond with the details - they are good data > points > to think about! > >> Lastly, there are social reasons. It

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-02-01 Thread Vinay Sajip
On Fri, 31/1/14, Brian Wickman wrote: > There are myriad other practical reasons.  Here are some: Thanks for taking the time to respond with the details - they are good data points to think about! > Lastly, there are social reasons. It's just hard to convince most engineers > to use things l

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-01-31 Thread Nick Coghlan
On 1 February 2014 05:31, Brian Wickman wrote: > This is in response to Vinay's thread but since I wasn't subscribed to > distutils-sig, I couldn't easily respond directly to it. > > Vinay's right, the technology here isn't revolutionary but what's notable is > that we've been using it in producti

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-01-31 Thread Brian Wickman
> > > For practical reasons we've always needed "not zip-safe" PEX files > > where all code is written to disk prior to execution (ex: legacy > > Django applications that have __file__-relative business logic) > > so we decided to just throw away the magical zipimport stuff and > > embrace using d

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-01-31 Thread Daniel Holth
On Fri, Jan 31, 2014 at 3:52 PM, Vinay Sajip wrote: > Thanks for the info. > >> For practical reasons we've always needed "not zip-safe" PEX files >> where all code is written to disk prior to execution (ex: legacy >> Django applications that have __file__-relative business logic) >> so we decide

Re: [Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-01-31 Thread Vinay Sajip
Thanks for the info. > For practical reasons we've always needed "not zip-safe" PEX files > where all code is written to disk prior to execution (ex: legacy > Django applications that have __file__-relative business logic) > so we decided to just throw away the magical zipimport stuff and > embra

[Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

2014-01-31 Thread Brian Wickman
This is in response to Vinay's thread but since I wasn't subscribed to distutils-sig, I couldn't easily respond directly to it. Vinay's right, the technology here isn't revolutionary but what's notable is that we've been using it in production for almost 3 years at Twitter. It's also been open-so