Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 29 Mar, 2013, at 21:11, Nick Coghlan wrote: > On Fri, Mar 29, 2013 at 8:43 AM, Daniel Holth wrote: >> WinZip will ignore anything in the front of the file since the zip >> directory doesn't reference it. The #! shebang is for Unix, would >> point to the correct Python, and the +x flag would make it executable. >> The mini PEP is for the .pyz registration and for publicity. > > The two big reasons almost nobody knows about the executable zip files > and directories is we forgot to mention it in the original 2.6 What's > New (it's there now, but was added much later), and it was done in a > tracker issue [1] (with Guido participating) rather than as a PEP. > > A new PEP to: > > * register the .pyz and .pyzw extensions in the 3.4 Windows installer Also: * add support for .pyz and .pyzw support to Python Launcher on OSX > * ship a tool for creating an executable pyz or pyzw file from a > directory of pure-Python files (warning if any extension files are > noticed, and with the option of bytecode precompilation) Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Mar 29, 2013 4:12 PM, "Nick Coghlan" wrote: > > On Fri, Mar 29, 2013 at 8:43 AM, Daniel Holth wrote: > > WinZip will ignore anything in the front of the file since the zip > > directory doesn't reference it. The #! shebang is for Unix, would > > point to the correct Python, and the +x flag would make it executable. > > The mini PEP is for the .pyz registration and for publicity. > > The two big reasons almost nobody knows about the executable zip files > and directories is we forgot to mention it in the original 2.6 What's > New (it's there now, but was added much later), and it was done in a > tracker issue [1] (with Guido participating) rather than as a PEP. > > A new PEP to: > > * register the .pyz and .pyzw extensions in the 3.4 Windows installer > * ship a tool for creating an executable pyz or pyzw file from a > directory of pure-Python files (warning if any extension files are > noticed, and with the option of bytecode precompilation) > > Would be great. And that pre-compilation could even do it for multiple versions of Python thanks to __pycache__. I've actually contemplated creating a distutils command to do this exact thing. Been thinking about this since 2010: http://sayspy.blogspot.ca/2010/03/various-ways-of-distributing-python.html > > Cheers, > Nick. > > [1] http://bugs.python.org/issue1739468 > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Nick Coghlan gmail.com> writes: > * ship a tool for creating an executable pyz or pyzw file from a > directory of pure-Python files This could be a variant of my pyzzer.py tool: https://gist.github.com/vsajip/5276787 It doesn't print warnings for extensions or byte-compile anything, but otherwise does more or less what you mentioned. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 29 March 2013 22:15, Daniel Holth wrote: > Would pyzw be much better than reading the shebang line for Windows? Yes. A different executable has to be run (console or windows). A .pyz file with pythonw in the shebang would run py.exe and flash up a console window before starting pythonw.exe. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Would pyzw be much better than reading the shebang line for Windows? On Mar 29, 2013 4:11 PM, "Nick Coghlan" wrote: > On Fri, Mar 29, 2013 at 8:43 AM, Daniel Holth wrote: > > WinZip will ignore anything in the front of the file since the zip > > directory doesn't reference it. The #! shebang is for Unix, would > > point to the correct Python, and the +x flag would make it executable. > > The mini PEP is for the .pyz registration and for publicity. > > The two big reasons almost nobody knows about the executable zip files > and directories is we forgot to mention it in the original 2.6 What's > New (it's there now, but was added much later), and it was done in a > tracker issue [1] (with Guido participating) rather than as a PEP. > > A new PEP to: > > * register the .pyz and .pyzw extensions in the 3.4 Windows installer > * ship a tool for creating an executable pyz or pyzw file from a > directory of pure-Python files (warning if any extension files are > noticed, and with the option of bytecode precompilation) > > Would be great. > > Cheers, > Nick. > > [1] http://bugs.python.org/issue1739468 > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Fri, Mar 29, 2013 at 8:43 AM, Daniel Holth wrote: > WinZip will ignore anything in the front of the file since the zip > directory doesn't reference it. The #! shebang is for Unix, would > point to the correct Python, and the +x flag would make it executable. > The mini PEP is for the .pyz registration and for publicity. The two big reasons almost nobody knows about the executable zip files and directories is we forgot to mention it in the original 2.6 What's New (it's there now, but was added much later), and it was done in a tracker issue [1] (with Guido participating) rather than as a PEP. A new PEP to: * register the .pyz and .pyzw extensions in the 3.4 Windows installer * ship a tool for creating an executable pyz or pyzw file from a directory of pure-Python files (warning if any extension files are noticed, and with the option of bytecode precompilation) Would be great. Cheers, Nick. [1] http://bugs.python.org/issue1739468 -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 6:19 PM, Steve Dower wrote: > Daniel Holth wrote: >> On Thu, Mar 28, 2013 at 3:49 PM, PJ Eby wrote: >>> On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower >>> wrote: And, I'm almost certain that most if not all existing ZIP tools on Windows will fail to open files with a shebang, since they've never had to deal with them. >>> >>> Actually, the opposite is true, at least for 3rd-party (non-Microsoft) >>> archiving tools: they work even when there's a whole .exe file stuck >>> on the front. ;-) >>> >>> Some of them require you to rename from .exe to .zip first, but some >>> actually detect that an .exe is a stub in front of a zip file and give >>> you extraction options in an Explorer right-click. >>> >>> So, no worries on the prepended data front, even if the extension is >>> .zip. What you probably can't safely do is *modify* a .zip with >>> prepended data... and there I'm just guessing, because I've never >>> actually tried. >> >> It will all work. ZIP is cool! Every offset is relative from the end of the >> file. >> >> The wheel distribution even has a ZipFile subclass that lets you pop() files >> off the end by truncating the file and rewriting the index. This will work >> on any ordinary zip file that is just the concatenation of the compressed >> files in zip directory order, without data or extra space between the >> compressed files. > > Ah of course, I totally forgot that it works from the end of the file. > > I'd still rather they got a new extension though, since nobody is going to > teach WinZip what "#! python" means. That's probably an issue for the handful > of Windows devs on python-dev rather than here, though. > > > Cheers, > Steve WinZip will ignore anything in the front of the file since the zip directory doesn't reference it. The #! shebang is for Unix, would point to the correct Python, and the +x flag would make it executable. The mini PEP is for the .pyz registration and for publicity. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Daniel Holth wrote: > On Thu, Mar 28, 2013 at 3:49 PM, PJ Eby wrote: >> On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower >> wrote: >>> And, I'm almost certain that most if not all existing ZIP tools on >>> Windows will fail to open files with a shebang, since they've >>> never had to deal with them. >> >> Actually, the opposite is true, at least for 3rd-party (non-Microsoft) >> archiving tools: they work even when there's a whole .exe file stuck >> on the front. ;-) >> >> Some of them require you to rename from .exe to .zip first, but some >> actually detect that an .exe is a stub in front of a zip file and give >> you extraction options in an Explorer right-click. >> >> So, no worries on the prepended data front, even if the extension is >> .zip. What you probably can't safely do is *modify* a .zip with >> prepended data... and there I'm just guessing, because I've never >> actually tried. > > It will all work. ZIP is cool! Every offset is relative from the end of the > file. > > The wheel distribution even has a ZipFile subclass that lets you pop() files > off the end by truncating the file and rewriting the index. This will work > on any ordinary zip file that is just the concatenation of the compressed > files in zip directory order, without data or extra space between the > compressed files. Ah of course, I totally forgot that it works from the end of the file. I'd still rather they got a new extension though, since nobody is going to teach WinZip what "#! python" means. That's probably an issue for the handful of Windows devs on python-dev rather than here, though. Cheers, Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 3:49 PM, PJ Eby wrote: > On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower > wrote: >> And, I'm almost certain that most if not all existing ZIP tools on Windows >> will fail to open files with a shebang, since they've never had to deal with >> them. > > Actually, the opposite is true, at least for 3rd-party (non-Microsoft) > archiving tools: they work even when there's a whole .exe file stuck > on the front. ;-) > > Some of them require you to rename from .exe to .zip first, but some > actually detect that an .exe is a stub in front of a zip file and give > you extraction options in an Explorer right-click. > > So, no worries on the prepended data front, even if the extension is > .zip. What you probably can't safely do is *modify* a .zip with > prepended data... and there I'm just guessing, because I've never > actually tried. It will all work. ZIP is cool! Every offset is relative from the end of the file. The wheel distribution even has a ZipFile subclass that lets you pop() files off the end by truncating the file and rewriting the index. This will work on any ordinary zip file that is just the concatenation of the compressed files in zip directory order, without data or extra space between the compressed files. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 1:54 PM, Steve Dower wrote: > And, I'm almost certain that most if not all existing ZIP tools on Windows > will fail to open files with a shebang, since they've never had to deal with > them. Actually, the opposite is true, at least for 3rd-party (non-Microsoft) archiving tools: they work even when there's a whole .exe file stuck on the front. ;-) Some of them require you to rename from .exe to .zip first, but some actually detect that an .exe is a stub in front of a zip file and give you extraction options in an Explorer right-click. So, no worries on the prepended data front, even if the extension is .zip. What you probably can't safely do is *modify* a .zip with prepended data... and there I'm just guessing, because I've never actually tried. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Daniel Holth gmail.com> writes:
> file.write(module.random_attribute.decode('base64')). The runnable zip
> feature is awesome, not well enough known, and totally worth promoting
> over the shar pattern; with some minimal tooling you'd be good to go.
Runnable zips sound great - I certainly haven't come across them before (or if
I have, I didn't see the potential at the time).
That said, from a Windows perspective, shebangs and mixed text/binary files
worry me. The better approach on Windows would be to take a new extension
(.pyz? .pyp[ackage]?) and associate that with the launcher. (File extensions on
Windows are the moral equivalent of shebang lines.)
Changing .zip in any way will upset anyone who has a utility for opening ZIP
files (i.e. everyone) and there's no way to launch files differently based on
content without changing that association. And, I'm almost certain that most if
not all existing ZIP tools on Windows will fail to open files with a shebang,
since they've never had to deal with them.
I also think that a runnable zip may be a better package installation option
than MSIs, but that's another issue :)
Cheers,
Steve
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Daniel Holth gmail.com> writes:
> file.write(module.random_attribute.decode('base64')). The runnable zip
> feature is awesome, not well enough known, and totally worth promoting
> over the shar pattern; with some minimal tooling you'd be good to go.
Well, maybe the promoting would be better done by actually shipping something
this way that shows how well it works, rather than just talking about it. I
don't see that the user experience is any better with runnable zips, though I'm
not saying it's any worse. After that, it just comes down to individual
developer
taste, and there's no accounting for that :-)
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Philippe Ombredanne nexb.com> writes: > Conceptually I find these no different from setup.py scripts, and > these have been mostly normalized (or at the minimum have a > conventional name and a conventional if not specified interface.) Except that you programmatically interface (to distutils or setuptools) with setup.py, which is not the case with virtualenv or distil. > I feel this is worth discussing as bootstrapping is where everything begins :) Oh, certainly it's worthy of discussion - I wasn't meaning to imply otherwise. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
Not really trying to tell Vinay to rewrite his script, but IMHO if you
expect it unzip is a lot easier than
file.write(module.random_attribute.decode('base64')). The runnable zip
feature is awesome, not well enough known, and totally worth promoting
over the shar pattern; with some minimal tooling you'd be good to go.
On Thu, Mar 28, 2013 at 10:44 AM, Philippe Ombredanne
wrote:
> On Thu, Mar 28, 2013 at 2:33 PM, Vinay Sajip wrote:
>>> From: Philippe Ombredanne
>>> On the other hand, I find it somewhat discomforting as an emerging
>>> best way to package and distribute self-contained bootstrap scripts.
>
>>> Virtualenv does it, distil is doing it now, pip tried some of it here
>>> https://github.com/pypa/pip/blob/develop/contrib/get-pip.py
>>> In contrast, buildout, distribute and setuptools bootstrap scripts do
>>> not embed their dependencies and either try to get them satisfied
>>> locally or attempt to download the requirements.
>>
>> And all this time, they would have been vulnerable to a MITM attack
>> on PyPI because PyPI didn't support verifiable SSL connections
>> until recently. It's good to be cautious, but Bruce Schneier has
>> plenty of stories about caution directed in the wrong directions.
>
> I am not so worried about security... I brought the point here because
> this is the packaging and distribution list, and I see this as an
> emerging pattern for the packaging and distribution of bootstrap
> scripts and this is something that has not been discussed much before.
>
> Conceptually I find these no different from setup.py scripts, and
> these have been mostly normalized (or at the minimum have a
> conventional name and a conventional if not specified interface.)
>
> Yet today, for the all important core package and environment
> management tools, we have bootstrap scripts each with different
> interfaces and different approaches to self containment or no
> containment.
>
> I feel this is worth discussing as bootstrapping is where everything begins :)
>
> --
> Philippe Ombredanne
>
> +1 650 799 0949 | pombreda...@nexb.com
> DejaCode Enterprise at http://www.dejacode.com
> nexB Inc. at http://www.nexb.com
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 2:33 PM, Vinay Sajip wrote: >> From: Philippe Ombredanne >> On the other hand, I find it somewhat discomforting as an emerging >> best way to package and distribute self-contained bootstrap scripts. >> Virtualenv does it, distil is doing it now, pip tried some of it here >> https://github.com/pypa/pip/blob/develop/contrib/get-pip.py >> In contrast, buildout, distribute and setuptools bootstrap scripts do >> not embed their dependencies and either try to get them satisfied >> locally or attempt to download the requirements. > > And all this time, they would have been vulnerable to a MITM attack > on PyPI because PyPI didn't support verifiable SSL connections > until recently. It's good to be cautious, but Bruce Schneier has > plenty of stories about caution directed in the wrong directions. I am not so worried about security... I brought the point here because this is the packaging and distribution list, and I see this as an emerging pattern for the packaging and distribution of bootstrap scripts and this is something that has not been discussed much before. Conceptually I find these no different from setup.py scripts, and these have been mostly normalized (or at the minimum have a conventional name and a conventional if not specified interface.) Yet today, for the all important core package and environment management tools, we have bootstrap scripts each with different interfaces and different approaches to self containment or no containment. I feel this is worth discussing as bootstrapping is where everything begins :) -- Philippe Ombredanne +1 650 799 0949 | pombreda...@nexb.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
> From: Daniel Holth > Also if you are looking for tweakability you can run a directory with > the same contents of the .zip exactly the same as if it was a zip. Sure, but my smoke testing involved copying the tweaked distil.py to a network share, then running that file from other Windows, Linux and OS X machines - of course I could have copied whole directory trees, but doing it the way I've done works well enough for me :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
> From: Philippe Ombredanne > On the other hand, I find it somewhat discomforting as an emerging > best way to package and distribute self-contained bootstrap scripts. But what is the root cause of that discomfort? The distil approach is slightly more discoverable than a pure zip would be, but for the security conscious all the code is there and available for inspection (unlike installing a distribution directly from PyPI, which will pull you-know-not-what from the network). > Virtualenv does it, distil is doing it now, pip tried some of it here > https://github.com/pypa/pip/blob/develop/contrib/get-pip.py > In contrast, buildout, distribute and setuptools bootstrap scripts do > not embed their dependencies and either try to get them satisfied > locally or attempt to download the requirements. And all this time, they would have been vulnerable to a MITM attack on PyPI because PyPI didn't support verifiable SSL connections until recently. It's good to be cautious, but Bruce Schneier has plenty of stories about caution directed in the wrong directions. > Having some support to do self-contained bootstrap scripts (as in > requiring no network access and embedding all their dependencies) > using this shar style could be something to consider normalizing? It seems like a decision for individual developers or developer teams to make on a case-by-case basis - it doesn't seem like something that needs to be "officially" encouraged or discouraged. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 13:11, Vinay Sajip wrote: > I don't know if it's that important to distinguish between the two. I found > the approach I'm using with distil to be a tad more flexible in my case. A > runnable zip has the advantage that it's harder to tinker with, but with the > way distil.py is at the moment, you can tweak e.g. its logging just by > changing distil.py. (At some point soon it will have an optional > configuration file to control some aspects of its behaviour, but that's by > the by.) It also does a bit of processing to process -e and -p and relaunches > with a new Python interpreter if needed - developing this logic was quicker > because I didn't have to add my changes to the .zip each time I tweaked > something. Good point. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 9:11 AM, Vinay Sajip wrote: >> From: Paul Moore > >> >> Yes, my point was that Vinay's usage could be covered by distributing >> distil as a zip file. All it is doing is decoding it's blob of data >> (which is an encoded zip file) and then adding the resulting zip to >> sys.path. > [snip] >> I hope that "embedded binary blobs" does not become a common approach. >> I'd much rather that "runnable zip files" became the norm. > > I don't know if it's that important to distinguish between the two. I found > the approach I'm using with distil to be a tad more flexible in my case. A > runnable zip has the advantage that it's harder to tinker with, but with the > way distil.py is at the moment, you can tweak e.g. its logging just by > changing distil.py. (At some point soon it will have an optional > configuration file to control some aspects of its behaviour, but that's by > the by.) It also does a bit of processing to process -e and -p and relaunches > with a new Python interpreter if needed - developing this logic was quicker > because I didn't have to add my changes to the .zip each time I tweaked > something. > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Also if you are looking for tweakability you can run a directory with the same contents of the .zip exactly the same as if it was a zip. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
> From: Paul Moore > > Yes, my point was that Vinay's usage could be covered by distributing > distil as a zip file. All it is doing is decoding it's blob of data > (which is an encoded zip file) and then adding the resulting zip to > sys.path. [snip] > I hope that "embedded binary blobs" does not become a common approach. > I'd much rather that "runnable zip files" became the norm. I don't know if it's that important to distinguish between the two. I found the approach I'm using with distil to be a tad more flexible in my case. A runnable zip has the advantage that it's harder to tinker with, but with the way distil.py is at the moment, you can tweak e.g. its logging just by changing distil.py. (At some point soon it will have an optional configuration file to control some aspects of its behaviour, but that's by the by.) It also does a bit of processing to process -e and -p and relaunches with a new Python interpreter if needed - developing this logic was quicker because I didn't have to add my changes to the .zip each time I tweaked something. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 12:45, Daniel Holth wrote: > On Thu, Mar 28, 2013 at 8:43 AM, Paul Moore wrote: >> On 28 March 2013 12:26, Daniel Holth wrote: >>> The launcher will be updated to understand this format and Python will >>> register this filename association when it is installed. >> >> The launcher should need no changes. The Python msi installer would >> need a change to register the new extension, though. >> >> And *creating* such zips is mildly annoying on Windows, due to a >> general lack of tool support for manipulating binary files in text >> editors. >> >> Oh, and wouldn't "#!/usr/bin/env python(w)" be a better header? That >> would work on Unix, and the launcher recognises that format. >> >> But +1 on the idea in general. >> Paul > > There is no 'cat header zip > newzip'? There are multiple options. And text file vs binary file issues to cover. CMD.EXE: copy /b header+zip newzip Powershell: get-content header,zip -enc Byte | set-content newzip -enc Byte Powershell: cmd /c copy /b header+zip newzip (because the previous version is so ugly...) Or write a Python script, which is what I did. Yes, I know :-( Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On Thu, Mar 28, 2013 at 8:43 AM, Paul Moore wrote: > On 28 March 2013 12:26, Daniel Holth wrote: >> The launcher will be updated to understand this format and Python will >> register this filename association when it is installed. > > The launcher should need no changes. The Python msi installer would > need a change to register the new extension, though. > > And *creating* such zips is mildly annoying on Windows, due to a > general lack of tool support for manipulating binary files in text > editors. > > Oh, and wouldn't "#!/usr/bin/env python(w)" be a better header? That > would work on Unix, and the launcher recognises that format. > > But +1 on the idea in general. > Paul There is no 'cat header zip > newzip'? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 12:26, Daniel Holth wrote: > The launcher will be updated to understand this format and Python will > register this filename association when it is installed. The launcher should need no changes. The Python msi installer would need a change to register the new extension, though. And *creating* such zips is mildly annoying on Windows, due to a general lack of tool support for manipulating binary files in text editors. Oh, and wouldn't "#!/usr/bin/env python(w)" be a better header? That would work on Unix, and the launcher recognises that format. But +1 on the idea in general. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
If it was distributed as such virtualenv could read blobs out of its own zip file, use the get_data() API to read non-module, or add subdirectories inside its zip file to sys.path On Thu, Mar 28, 2013 at 8:32 AM, Paul Moore wrote: > On 28 March 2013 11:40, Philippe Ombredanne wrote: >> This is not a zip, not an egg, not a wheel but some egg-in-py, >> zip-in-py or wheel-in-py and is similar to a shar shell archive. >> >> My point was that on the one hand, I like the fact that everything is >> self contained in one single .py file that you can execute right away. >> On the other hand, I find it somewhat discomforting as an emerging >> best way to package and distribute self-contained bootstrap scripts. >> Yet I cannot think of a better way atm: for instance splitting things >> in non-encoded non-binary plain strings would be quite weird too. > > Yes, my point was that Vinay's usage could be covered by distributing > distil as a zip file. All it is doing is decoding it's blob of data > (which is an encoded zip file) and then adding the resulting zip to > sys.path. > > The virtualenv situation is different, as there we are trying to > ensure that we remain single-file while embedding things that are > *not* modules to add to sys.path. And we don't want to download our > dependencies because we need to be able to run with no internet > connection. But you are right, the embedded script approach is not > ideal. > > I hope that "embedded binary blobs" does not become a common approach. > I'd much rather that "runnable zip files" became the norm. It's > certainly possible now, but I don't think it's well enough known (and > there are administrative issues like the file extension question on > Windows that make it more awkward than it should be). Hence my > comments, trying to raise awareness a bit. > > Thanks for the feedback, and in particular the reminder that > virtualenv could do with looking at this... I've added a virtualenv > issue to remind me to think some more about it. > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
On 28 March 2013 11:40, Philippe Ombredanne wrote: > This is not a zip, not an egg, not a wheel but some egg-in-py, > zip-in-py or wheel-in-py and is similar to a shar shell archive. > > My point was that on the one hand, I like the fact that everything is > self contained in one single .py file that you can execute right away. > On the other hand, I find it somewhat discomforting as an emerging > best way to package and distribute self-contained bootstrap scripts. > Yet I cannot think of a better way atm: for instance splitting things > in non-encoded non-binary plain strings would be quite weird too. Yes, my point was that Vinay's usage could be covered by distributing distil as a zip file. All it is doing is decoding it's blob of data (which is an encoded zip file) and then adding the resulting zip to sys.path. The virtualenv situation is different, as there we are trying to ensure that we remain single-file while embedding things that are *not* modules to add to sys.path. And we don't want to download our dependencies because we need to be able to run with no internet connection. But you are right, the embedded script approach is not ideal. I hope that "embedded binary blobs" does not become a common approach. I'd much rather that "runnable zip files" became the norm. It's certainly possible now, but I don't think it's well enough known (and there are administrative issues like the file extension question on Windows that make it more awkward than it should be). Hence my comments, trying to raise awareness a bit. Thanks for the feedback, and in particular the reminder that virtualenv could do with looking at this... I've added a virtualenv issue to remind me to think some more about it. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Self-contained boostrap scripts [was: Re: A new, experimental packaging tool: distil]
PEP XXX Yet Another Python .ZIP File Extension One reason Python ZIP applications have not been more widely adopted is because they have no Windows file association and would have to be confusingly named .py to be invoked by the interpreter. Henceforth, files with the extension .pyz shall be known as Python ZIP files. They consist of a ZIP format archive containing at minimum __main__.py, concatenated to two lines #!python or #!pythonw (or the full path to the interpreter), and an explanation # This is a ZIP format archive executable by the Python interpreter. The launcher will be updated to understand this format and Python will register this filename association when it is installed. On Thu, Mar 28, 2013 at 7:40 AM, Philippe Ombredanne wrote: > On Tue, Mar 26, 2013 at 11:08 AM, Paul Moore wrote: >> On 26 March 2013 09:49, Philippe Ombredanne wrote: >>> Would anyone know of a better way to package things in a single >>> python-executable bootstrapping script file without obfuscating the >>> source contents in compressed/encoded/obfuscated byte arrays? >> >> Packaging as a zip file is a good way - but on Windows the file needs >> to be named xxx.py (which is surprising, to say the least :-)) for the >> relevant file association to be triggered (and on Unix, a #! line >> needs to be prepended). > Paul: > I was not talking about this type of zips, but rather the same used in > virtualenv, i.e. a string in a .py file that contains an encoded zip. > That string is then decoded and unzipped at runtime as in here: > https://github.com/pypa/virtualenv/blob/develop/virtualenv.py#L1933 > > This is not a zip, not an egg, not a wheel but some egg-in-py, > zip-in-py or wheel-in-py and is similar to a shar shell archive. > > My point was that on the one hand, I like the fact that everything is > self contained in one single .py file that you can execute right away. > On the other hand, I find it somewhat discomforting as an emerging > best way to package and distribute self-contained bootstrap scripts. > Yet I cannot think of a better way atm: for instance splitting things > in non-encoded non-binary plain strings would be quite weird too. > > Virtualenv does it, distil is doing it now, pip tried some of it here > https://github.com/pypa/pip/blob/develop/contrib/get-pip.py > In contrast, buildout, distribute and setuptools bootstrap scripts do > not embed their dependencies and either try to get them satisfied > locally or attempt to download the requirements. > Having some support to do self-contained bootstrap scripts (as in > requiring no network access and embedding all their dependencies) > using this shar style could be something to consider normalizing? > > -- > Philippe Ombredanne > > +1 650 799 0949 | pombreda...@nexb.com > DejaCode Enterprise at http://www.dejacode.com > nexB Inc. at http://www.nexb.com > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
