Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
Just to update everyone on the status of this, the next thing on my list is to figure out the Windows build and set up the the file association in the installer. Actually, I should ask if there's anything else that I should pay attention to here, e.g. do I have to add an icon association for Windows or something like that? Is there any documentation like a wiki page on this? I looked at the README in the PC* directories and it doesn't seem to talk about the installer. Maybe it will become clearer when I get Visual Studio. Andy On 7/23/07, Paul Moore <[EMAIL PROTECTED]> wrote: > On 23/07/07, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > Actually, it isn't, because you can't start a zipfile with a Python > > script. Lord knows I've *tried*, but the Python interpreter just > > won't accept arbitrary binary data as part of a script. :) > > That bit me a while back, hard enough that I thought of putting > together a patch for it (probably just to stop processing the script > at a NUL byte), but never did as I didn't think I could put a > convincing enough case for it being *useful*. > > Anyway, I'd be happy enough with the -z patch as it stands, or if > someone comes up with something better, that would suit me too... > > Paul. > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andychup%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Add a -z interpreter flag to execute a zip file
I'd like to request comments on this patch I submitted: https://sourceforge.net/tracker/index.php?func=detail&aid=1739468&group_id=5470&atid=305470 There are many details given in the comments on that page. This can be used to deploy Python programs in a very lightweight and cross-platform way. You could imagine a cgi script or a light web app server being deployed like this. I have personally deployed Python programs using zip files and this would get rid of the need for boilerplate needed for each platform. The good thing about this is that it's extremely simple -- basically 20 lines of C code to add a -z flag that calls a 3-line Python function in the runpy module. I don't believe it overlaps with anything that already exists. py2exe and py2app are platform specific and bundle the Python interpreter. This will be a cross platform binary that doesn't bundle the Python interpreter. It doesn't require eggs but I think it would work fine with eggs, and could help fix a little bug as I mentioned on the patch page. Nick Coghlan has reviewed the patch and seems to think it's a good idea. Thomas Wouters also said he likes it, and I ran it by Guido earlier and he seemed to think the idea is good, although I don't think he has seen the implementation. thanks, Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/13/07, Jim Jewett <[EMAIL PROTECTED]> wrote: > Andy C wrote: > >... a .zip file with a __zipmain__.py module at its root? > > Why not just an __init__.py, which you would normally execute if you > tried to import/run a directory? > > > * Magically looking at the first argument to see if it's a zip file > > seems problematic to me. I'd rather be explicit with the -z flag. > > Likewise, I'd rather be explicit and call it __zipmain__ rather than > > __main__. > > Treating zip files (and only zip files) as a special case equivalent > to uncompressed files seems like a wart; I would prefer not to > special-case zips any more than they already are. Just to clarify, my patch already works with uncompressed directory trees just fine. It's just a matter of naming, I suppose. I don't mind calling it -z and using it for directories. But mainly that's because no one has proprosed another name. : ) I think we've agreed that -p is something totally different. > > while I think it would be a bad practice to > > import __main__, > > I have seen it recommended as the right place to store global > (cross-module) settings. Where? People use __main__.py now? That seems bad, because __ names are reserved, so they should just use main.py, I would think. Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/13/07, Greg Ewing <[EMAIL PROTECTED]> wrote: > Andy C wrote: > > What does "if __name__ == '__main__" mean in > > __main__.py? : ) If someone tries does import __main__ from another > > module in the program, won't that result in an infinite loop? > > Is there a reason not to use __init__.py for this? Well, you might have multiple executable .py files in the same source tree. So then you would want to build multiple .pyz files, each of which has a different __zipmain__. I think of __zipmain__ as part of the format of the .pyz file, not part of the source tree. In particular, it specifies what module/function to run in the zipped source tree (and I imagine it will be adapted for other uses). In this model you're separating your development tree and the thing you deploy, which is not always the case in Python. Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
Another issue I see is that -m and -c have command line parsing semantics, and -z follows those now. python -z foo.zip -z bar As implemented, this would pass sys.argv[0:3] == ['foo.zip', '-z', 'bar'] If you allow multiple -z flags to be meaningful, this gets confusing. The foo.zip program could have a legitimate -z flag. If you overload -z to mean "prepend things to sys.path", then you might also want to do python -z /dir1 -z /foo.zip -c 'import foo; print foo'. Should this execute dir1/__main__.py, foo.zip/__main__.py or print foo? I could be missing what you intend. But I think the patch as implemented doesn't have any of these potentially unconsidered cases and unintended consequences. Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/12/07, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 03:52 PM 7/12/2007 -0700, Andy C wrote: > >On 7/12/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > > > The patch suggests using .pyz and adding a default association to the > > > > installer (much like .py and .pyw have). > > > > > > Ok. It would be good if the patch actually added that extension, rather > > > than merely suggesting that it should be added. > > > >So does everyone agree that there should be a new extension called > >.pyz? And that the definition of this is a .zip file with a > >__zipmain__.py module at its root? If so, I can make the change... I > >haven't looked around the codebase yet but it sounds easy enough. > > Let's use __main__, please. Fewer names to remember, and __main__ is > supposed to be the __name__ of the main program. It Just Makes Sense. I can definitely see why it "just makes sense", and my first thought was indeed to name it __main__. But then you lose the ability to make a distinction: What does "if __name__ == '__main__" mean in __main__.py? : ) If someone tries does import __main__ from another module in the program, won't that result in an infinite loop? There aren't any restrictions on what can be in __main__ (it's just another module), and while I think it would be a bad practice to import __main__, I could see people being tripped up by this in practice. People might start storing silly things like the program version there, for convenience. At Google some people do "import sitecustomize" and get values that were computed earlier by the sitecustomize. I could see the same kind of thing happen with __main__.py. > >* Does anyone else want to change the -z flag to make more sense for > >directories (and possibly change __zipmain__.py to __main__.py)? In > >thinking about this again, I am not sure I can come up with a real use > >case. > > Testing your package before you zip it, would be one. :) My > personal main interest was in being able to add an item to sys.path > without having to set $PYTHONPATH on Windows. That's why I'd like it > to be possible to use -z more than once (or whatever the option ends up as). Where would you do that? Just typing it literally on the command line? Just curious, I have never felt a need to do that. I use Python on Windows frequently. > > I think it's sufficient to treat it as a documented "trick" > >that you can substitute a whole directory for a zip file with the -z > >flag. If there is a concrete suggestion, I'd like to discuss it, but > >otherwise it seems like we'll get bogged down in expanding use cases. > > Eh? First you say there aren't any use cases, now you say there'll be > too many? I'm confused. The only competing proposal besides what > I've suggested was the one to add an option to "runpy", and IMO > that's dead in the water due to shebang argument limits. As implemented the patch is fairly simple, and shouldn't have any unintended consequences. I'm not necessarily opposed to making it more general and thinking about sys.path vs. a zip file specifically. But I don't have a clear enough picture from all the comments of what exactly to implement. -z is not the same as "prepend an item to sys.path", because we replace "" with the -z argument. And we also munge sys.argv[0] (which is what you said should happen). So it's not clear to me at all what multiple -z's should do, exactly. Can you write out the pseudo code? Or modify my patch. I think it would be fine to have both a -z and -p flag, if the functionality is needed. -z accepts a zip file or a directory and does the right thing to run it as an executable. -p could accept multiple arguments and literally prepends them to sys.path. These seem like different things to me. I'll look at adding the file association for .pyz (is there an expert on that for questions?), and in that time hopefully the list can decide on the rest of the issues. Or we can just make Guido decide, which is fine by me. : ) Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/12/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > The patch suggests using .pyz and adding a default association to the > > installer (much like .py and .pyw have). > > Ok. It would be good if the patch actually added that extension, rather > than merely suggesting that it should be added. So does everyone agree that there should be a new extension called .pyz? And that the definition of this is a .zip file with a __zipmain__.py module at its root? If so, I can make the change... I haven't looked around the codebase yet but it sounds easy enough. This makes it seem like a bigger change than it is, but I think it's the right thing to do to support Windows properly. Other points: * I think it's true that the shebang line should only have one argument. * Does anyone else want to change the -z flag to make more sense for directories (and possibly change __zipmain__.py to __main__.py)? In thinking about this again, I am not sure I can come up with a real use case. I think it's sufficient to treat it as a documented "trick" that you can substitute a whole directory for a zip file with the -z flag. If there is a concrete suggestion, I'd like to discuss it, but otherwise it seems like we'll get bogged down in expanding use cases. * Magically looking at the first argument to see if it's a zip file seems problematic to me. I'd rather be explicit with the -z flag. Likewise, I'd rather be explicit and call it __zipmain__ rather than __main__. thanks, Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/12/07, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 10:09 AM 7/12/2007 +0200, Martin v. Löwis wrote: > >"" should not be removed from sys.path. It is *not* meant to be > >the current directory, but the directory where the main script > >lives. > > Right; it should be replaced with the zipfile path instead. That's indeed what the current implementation does, replacing "" with the zip file. > I would personally rather see this option defined as simply placing a > directory at the front of sys.path, and perhaps defining a default -m > value of __main__, unless overrridden. Being able to use the option Actually, that's a good idea, and it does work with my current implementation [1], although we'd have to change the name __zipmain__. Is __main__ a good idea considering that is used for something similar but implemented completely differently (the module name)? I thought about using __main__, but decided on __zipmain__ since seemed to be more explicit and reduce potential conflicts. To be clear to other readers, the convention would be that if a __main__.py file exists at the root of a directory, then the whole directory is considered an executable python program. > more than once would be nice, too. On Windows, you can't set an > environment variable on the same line as a command, so this would > give you a one-liner way of setting sys.path and running an application. > > I do not see a reason to make this option zipfile-specific in any > way, though; it's just as useful (and sometimes more so) to be able > to distribute an application as a directory, since that lets you use > .pyd, .so, .dll etc. without needing the egg cache system for using those. Yes, the dynamic library importing is nice. thanks, Andy 1) andychu testprog$ find . ./__init__.py ./package1 ./package1/__init__.py ./package1/foo.py ./package1/lib.py ./__zipmain__.py andychu testprog$ ../python -z . lib module here argv: ['.'] andychu testprog$ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/12/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > But in the description, you said that you do the same on Windows > by making a file that is both a zip file and a batch file. So my > approach is also cross-platform, no? > > How do you get the -z option to work on Windows? What extension > do you use, and how is the zipfile created? Nick suggested using .pyz, and others seem to like that solution (possibly using pythonw) and that seems logical enough to me. If it's agreed that that's the right solution on Windows, I can put in the work for that. > Couldn't that also be achieved by documenting best practice in > the documentation? Why is the shell script not robust? I think it's pretty clear that it's not robust, and there have been even more anecdotal examples on this thread. Everyone does it slightly differently -- not for any particular reason, but just because the right thing isn't trivial. As I pointed out, the example you came up with (which many others would come up with too) has a fairly serious problem, in that it will import things from outside the .zip file. I could build my .zip file on my system, test it out, and then deploy it to another machine and it will break. Ironically, this happened to *me* while developing the patch! > Why that? Why do eggs fail to process $0 correctly, whereas the > -z option gets it correct? That just sounds like a bug in eggs > to me, that could be fixed - or, if not, I'd expect that -z > cannot fix it, either. > > My understanding of this note is that > pkg_resources uses sys.argv[0] to determine the version number > of the egg; IIUC, -z won't help at all here because sys.argv[0] > will still be the name of the symlink. OK, I could be mistaken here, I haven't actually repro'd this bug. > What are those weird hacks, why are they necessary, and how does the > -z option overcome the need for these hacks? > > That people fail to make it work with /bin/sh doesn't automatically > mean they succeed with -z. Either they are too unexperienced to > make the shell header correct (in which case documenting best > practice would help), or they have deeper problems with that approach, > in which case it isn't at all obvious that the proposed change > improves anything. I don't think this is true at all. I have provided the sample code to make one of these files, and so you basically have to run a command line, rather than write a shell header -- and the shell header is currently not documented anywhere. As mentioned, this approach also prevents you from having to start the shell, and makes it more portable, since people might use #!/bin/myfavoriteshell or use #!/bin/sh and not realize they are using system-specific features of the shell. > > Another example is that the behavior of the zip in your example > > depends on what else is in the current directory [1], which isn't > > desirable. Nick pointed out this issue and I addressed it in the > > patch by removing "" from sys.path, since the -c flag adds that. > > "" should not be removed from sys.path. It is *not* meant to be > the current directory, but the directory where the main script > lives. Regardless of what "" *should* be interpretreted as, the example you gave has the problem mentioned (with current versions of Python) -- that "" *is* the current directory and thus things get imported outside of the zip file when they are not found in the zip file. Right now "" is replaced with the zip file. If there's a better implementation I'm willing to change it. > > As mentioned, it's also a very tiny amount of code, and I don't see > > much potential for bad interactions with other things, the way I've > > written it. > > It's baggage that is rarely needed, and the feature can be readily > implemented in a different way for people who need it. I also disagree with both statements. : ) I think others have said basically the exact same thing as I am saying: that it is *commonly* needed, it's not a lot of baggage in Python since it's so little code, and it's easy to get wrong. Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a -z interpreter flag to execute a zip file
On 7/11/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > Nick Coghlan has reviewed the patch and seems to think it's a good > > idea. Thomas Wouters also said he likes it, and I ran it by Guido > > earlier and he seemed to think the idea is good, although I don't > > think he has seen the implementation. > > See my comment: I must be missing the point of the patch, since > I can do the same thing (make a single executable zip file on > Linux) through a /bin/sh header just fine. Right, but it's supposed to be cross platform, as mentioned in the patch. This will work on Windows. The main problem I see is that a shell script in front of a zip file seems like a relatively common idiom that people use and have different variants on, each of which have their own idiosyncrasies. So it would nice to consolidate them and make it standard and robust. For example, it looks like eggs have an executable format that is similar to this. And see the bug I mentioned where those executable eggs can't be invoked through a symlink (which to me is a relatively severe problem). I think this has to do with some introspection on $0, but you won't run into that with this implementation. Also, I mentioned the program called autopar we use at Google that does the same thing, and it also have a significant number of weird hacks in the shell header. I think Thomas Wouters has also worked on another program to make an executable zip file. Another example is that the behavior of the zip in your example depends on what else is in the current directory [1], which isn't desirable. Nick pointed out this issue and I addressed it in the patch by removing "" from sys.path, since the -c flag adds that. If lots of people reinvent this wheel (and they have), there are going to be other subtleties like this that will be missed. The -z flag also eliminates starting an extra process -- you invoke the Python interpreter directly instead of starting a shell which in turn invokes the Python interpreter. As mentioned, it's also a very tiny amount of code, and I don't see much potential for bad interactions with other things, the way I've written it. Andy 1) andychu test2$ ./foo_exe.zip Traceback (most recent call last): File "", line 1, in ? File "foo.py", line 16, in ? import outside ImportError: No module named outside andychu test2$ touch outside.py andychu test2$ ./foo_exe.zip main andychu test2$ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Add a -z interpreter flag to execute a zip file
(resending this now that I'm subscribed, not sure it made it through the moderation the first time) I'd like to request comments on this patch I submitted: https://sourceforge.net/tracker/index.php?func=detail&aid=1739468&group_id=5470&atid=305470 There are many details given in the comments on that page. This can be used to deploy Python programs in a very lightweight and cross-platform way. You could imagine a cgi script or a light web app server being deployed like this. I have personally deployed Python programs using zip files and this would get rid of the need for boilerplate needed for each platform. The good thing about this is that it's extremely simple -- basically 20 lines of C code to add a -z flag that calls a 3-line Python function in the runpy module. I don't believe it overlaps with anything that already exists. py2exe and py2app are platform specific and bundle the Python interpreter. This will be a cross platform binary that doesn't bundle the Python interpreter. It doesn't require eggs but I think it would work fine with eggs, and could help fix a little bug as I mentioned on the patch page. Nick Coghlan has reviewed the patch and seems to think it's a good idea. Thomas Wouters also said he likes it, and I ran it by Guido earlier and he seemed to think the idea is good, although I don't think he has seen the implementation. thanks, Andy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
