Re: [Distutils] How to handle launcher script importability?
I know I'm bike-shedding here, but my preference for extension would be '.pye' as it indicates something to execute, but without indicating exactly how (i.e. that it's via a separate launcher executable). .pyx that said, both .pye and .pyx are superior to .pyl (is that last character an ell, an eye or a one?) -matt ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
.pyx that said, both .pye and .pyx are superior to .pyl (is that last character an ell, an eye or a one?) I agree that .pyl is less readable/more confusable. We can't use .pyx, though, as that is already used in the Python ecosystem for Pyrex files (a forerunner of Cython). Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 23:30, Chris Barker - NOAA Federal chris.bar...@noaa.govwrote: On Tue, Aug 13, 2013 at 2:27 PM, Paul Moore p.f.mo...@gmail.com wrote: 3) I'd rather not have to mess with PATHEXT, and I particularly don't want to have to tell my students to do it -- environment variables are a pain, and somehow PATHEXT has been fragile for me (and I don't use Cygwin) Nobody is suggesting that end users mess with PATHEXT. The proposal is that the Python installer does this ouch! I don't think I'd want PATHEXT set for *.py files -- I'd rather they get opened by an editor by default than run...or is point+click behavior different than command line -- shows you how well I know Windows. It's OK, PATHEXT has nothing to do with double clicks - that's the file associations (and run the script is the default action set by the installer, and has been for many, many versions). (indeed, that's been done for Python 3.4, I don't know if the installer for the standalone launcher has been updated in the same way yet). There are those of us still in the 2.7 world -- and I suspect for a good while. I understand this. It's the biggest issue here, that any changes have to not forget users of older Pythons. Personally, I tend to err towards a view that we improve things for current Pythons (3.3+) and make sure we don't ruin things for older versions, possibly by providing workarounds the user needs to apply. I view install the standalone launcher to be in this category of workaround. There are others who are more conservative, so don't worry your views are well represented! I'm suggesting that we collect specifics on any fragility (can you provide details of what has gone wrong for you?) well, for PATHEXT, the env variable has to be set right, and Windows kind of hides all that from you -- it's really a pain to edit them by hand, so if the installer doesn't do it, or someone re-builds their registry or profile, or what have you, then it'll break. Oh and the cygwin (and who know what other shell alternatives) issue. At least we can pretty much count on an exe running if the shell can find it.. Thanks. I'd class most of that as relatively non-specific or fixed. Users shouldn't have to set PATHEXT if the installer does it, and in any case I'd like to clearly understand what sort of users are writing command line Python scripts that they want to run transparently as if they were executables, who still wouldn't know how to set an environment variable (or be able to understand whatever documentation we produce on how to do it). The cygwin issue has been mentioned/addressed elsewhere, but thanks for that. It's certainly something that needs to be weighed in the balance. Fragiity for the exe approach -- all I know is that the setuptools binary was proken a while back -- but setuptools itself was in a bit of a void of not-quite-sure-if-its maintained, and not-sure-even-how-to-report-a-bug state. It seems that setuptools (or whatever this will be part of) has now been adopted by the core Python community, so we can expect good support -- let's hope so. Anyway, whether the exe approach was more or less fragile than anyting else, is sure was harder to debug/fix for anyone that doesn't know Windows and non-python development well. My personal experience with the exes: 1. If the shebang line in the script gets corrupted, the resulting error is misleading and extremely hard to debug. 2. The existence of the exe in an installer makes it platform-specific, which is a problem for otherwise portable scripts/packages. 3. To use the exes, you really need to create them via setuptools, so they are not practical for standalone scripts - and having 2 distinct ways of setting scripts up to be runnable is less than ideal. 4. If you are upgrading/reinstalling the script, you cannot do so without workarounds if the exe is currently running (mostly a nice case for pip upgrading itself). Also, not directly related to the exes, but definitely to setuptools wrappers (see point 3 above): 1. The wrappers need pkg_resources installed *at runtime* not just at build time. I cant help thinking a more elegant solution exists, but maybe not. Personally, I believe that executable .py scripts (or maybe a dedicated .pys/.pye/.pya/whatever extension) *is* a more elegant solution - but equally, I concede, maybe not... well, I guess that's the way Windows does things -- *nix has the executable bit, Windows uses extensions, so I suppose that is the way to get an executable script -- but it really SHOULDN'T be *.py! Agreed, we need two concepts Python module (.py) and Python script (the new extension). I've lost track of what needs to be tried -- can't we just associate an extension with py and give it a go? (got to get that Windos VM working) Yes, essentially. I can confidently report that the solution is flawless, and should be adopted at once :-) More seriously, we need
Re: [Distutils] How to handle launcher script importability?
Jason R. Coombs jaraco at jaraco.com writes: This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Just to muddy the waters a little, I'd like to suggest an alternative approach which doesn't appear to have been tried: 1. The installer just installs foo.exe on Windows for script 'foo', where the foo.exe contains the actual script 'foo' as a resource. 2. The launcher, before looking for foo-script.py, examines its resources. If a script resource is found, it is extracted and written to foo-script.py in the same directory as foo.exe. If such a resource isn't found, it continues to the next step. 3. The launcher looks for 'foo-script.py' in its directory, invokes it and waits for it to complete. 4. If a 'foo-script.py' was written in step 2, it is deleted. The launcher comes with an embedded manifest, so no external manifest is needed. The insertion of a script as a resource is easy to do using ctypes: I've tested it using a suitably modified version of the current distlib launcher and it seems to work as expected. Advantages of this approach: 1. No additional file clutter: Just one installed file per script. 2. No need to worry about PATHEXT, or whether the PEP 397 launcher is installed. 3. Since the script is foo-script.py, no import clashes will occur. Disadvantages of this approach: 1. The scripts are hard to inspect because they're in the .exe. OTOH, (a) we don't really want people to mess with the scripts, and (b) they should be stock wrappers for other code which does the real work. (There are developer tools which allow inspection of resources, if really needed.) 2. Requires ctypes in the installer, so perhaps problematic for versions of Python 2.5. 3. Scripts take up more space than a .py-with-PEP 397-launcher solution, but that approach can still give rise to the import issue, as well as requiring PATHEXT setup. Thoughts? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 13 August 2013 20:58, Paul Moore p.f.mo...@gmail.com wrote: On 13 August 2013 18:08, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 13 August 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote: On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. So say I have a ~/bin directory where I put my scripts that I want to be generally available. I install something with python setup.py install --install-scripts=~/bin so that the scripts/script-wrappers go in there because I want to be able to always access that program under that name. Don't be fooled by the unixy tilde: I'm running ordinary Windows Python in that command in git-bash, not Cygwin. Now if that folder is on PATH while I am in Cygwin I can run the program with the same name if an .exe wrapper was added. I can't run it with the same name if it's a .py/,bat file because Cygwin doesn't have the implicit strip-the-extension PATHEXT feature and can't run .bat files. Ah, OK, thanks for the clarification. In that case I can see why you'd prefer exe wrappers (or maybe cygwin bash shell wrappers, or shell aliases...). Maybe an option to still use exe wrappers is worth it - but honestly, I'd say that in that context you probably have enough expertise to understand the issue and make your own solution relatively easily. Yes, but I'd like it if pip install some_cmd would just work. What about having in your .bashrc: for prog in ls ~/bin/*.py; do alias $(basename $prog .py)=$prog done (Excuse me if I got the precise details wrong there). OK, you need to rerun .bashrc if you add new scripts. It's not perfect. But it's not a showstopper either. There are ways to make it work for every different environment where I would type the command. Really though it's a pain to have to set these things up everywhere. Also this still doesn't work with subprocess(..., shell=False). There are a huge range of programs that can invoke subprocesses of a given name and I want them all to work with commands that I install from pypi. There are good reasons to use shell=False: the subprocess documentation contains no less than 5 warning boxes about shell=True! This is not peculiar to Python's subprocess module: it is the underlying Windows API calls regardless of which language the parent process is implemented in. Here's a demo of what happens with Robert Kern's kernprof.py script that doesn't have an .exe wrapper (on my system; it's possible that I didn't install it with pip). $ python Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. import subprocess subprocess.call(['kernprof.py'], shell=True) # Uses file-association Usage: kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... 2 import os os.environ['PATHEXT'] '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PY;.PYC;.PSC1;.RB;.RBW' subprocess.call(['kernprof'], shell=True) # Uses PATHEXT Usage: kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] ... 2 subprocess.call(['kernprof'], shell=False) # Needs an .exe wrapper! Traceback (most recent call last): File stdin, line 1, in module File q:\tools\Python27\lib\subprocess.py, line 524, in call return Popen(*popenargs, **kwargs).wait() File q:\tools\Python27\lib\subprocess.py, line 711, in __init__ errread, errwrite) File q:\tools\Python27\lib\subprocess.py, line 948, in _execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file specified subprocess.call(['kernprof.py'], shell=False) # Needs an .exe wrapper! Traceback (most recent call last): File stdin, line 1, in module File q:\tools\Python27\lib\subprocess.py, line 524, in call return Popen(*popenargs, **kwargs).wait() File q:\tools\Python27\lib\subprocess.py, line 711, in __init__ errread, errwrite) File q:\tools\Python27\lib\subprocess.py, line 948, in _execute_child startupinfo) WindowsError: [Error 193] %1 is not a valid Win32 application Here's what happens if I put kernprof.bat next to kernprof.py (the .bat file just @echos running kernprof): import subprocess subprocess.call(['kernprof'], shell=True) # PATHEXT in action running kernprof 0 subprocess.call(['kernprof'], shell=False) # No PATHEXT Traceback (most recent call last): File stdin, line 1, in module File q:\tools\Python27\lib\subprocess.py, line 524, in call return Popen(*popenargs, **kwargs).wait() File q:\tools\Python27\lib\subprocess.py, line 711, in __init__ errread, errwrite) File q:\tools\Python27\lib\subprocess.py, line 948, in _execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file specified subprocess.call(['kernprof.bat'], shell=False) # Works but
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 7:34 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Jason R. Coombs jaraco at jaraco.com writes: This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Just to muddy the waters a little, I'd like to suggest an alternative approach which doesn't appear to have been tried: 1. The installer just installs foo.exe on Windows for script 'foo', where the foo.exe contains the actual script 'foo' as a resource. 2. The launcher, before looking for foo-script.py, examines its resources. If a script resource is found, it is extracted and written to foo-script.py in the same directory as foo.exe. If such a resource isn't found, it continues to the next step. 3. The launcher looks for 'foo-script.py' in its directory, invokes it and waits for it to complete. 4. If a 'foo-script.py' was written in step 2, it is deleted. The launcher comes with an embedded manifest, so no external manifest is needed. Better suggestion: just append a PEP 441 .pyz to the .exe, and no extraction is necessary; the .exe just reads out the #! part. For Python 2.6 and up, the .exe can simply pass itself as argv[1] to the interpreter. (For older Pythons, a little -c and PYTHONPATH munging is required, but should be doable.) For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) Thoughts? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. In any case, it'd probably be a good idea to offer some command line tools for manipulating such .exes, to e.g. show/change what Python it's set to use, extract/dump/replace the zip, etc. (As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 12:42, Oscar Benjamin oscar.j.benja...@gmail.com wrote: I do think, as I said before, that this needs some sort of policy-type PEP on the standard approach for wrapping scripts, with all the pros and cons of the various approaches documented and reviewed. There have been so many emails on this list that I can't immediately find it but somewhere Steve Dower of Microsoft said something like: ''' Don't worry. Exe wrappers are *definitely* the best solution for this. ''' (quote is approximate and emphasis added by me). My experience has lead me to the same conclusion as Steve. It may be worth documenting the reasons why but make no mistake about it: .exe wrappers of some form are the way to go. I would like it documented, with all the reasons, so that we don't keep rehashing this whole thing over and over. Also, if we do recommend exes, I'd like to see something (in the stdlib ultimately) that makes it trivially easy to wrap an existing .py script (a standalone one, not one with an associated setup.py) into an exe. Having two ways to write a command using Python is icky. Also, I'd like a single-file solution. More times than I care to remember I have put a program on a USB key or in my dropbox/skydrive and forgotten the support stuff (whether that's required DLLs, resource files, or whatever - it's not a Python-specific problem). But I do see your point regarding things like subprocess. It's a shame, but anything other than exes do seem to be second class citizens on Windows. BTW, you mention bat files - it bugs me endlessly that bat files seem to have a more privileged status than other script formats whether that's .py or .ps1 or whatever. I've never managed to 100% convince myself that they are special in a way that you can't replicate with suitable settings (PATHEXT, etc, etc). I think it's that .bat is hard-coded in the OS search algorithm or something, though. The docs are not easy to locate on the various aspects of matter. (If bat files didn't have their horrible nesting and ctrl-C handling behaviours, they'd be a viable solution...) Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 13:57, PJ Eby p...@telecommunity.com wrote: Thoughts? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? Good point. No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. Again, agreed - we have executable zipfiles for Python, and a combined exe/zipfile is a perfectly viable format (it's used by most self-extracting zip formats, as well as wininst formats). In any case, it'd probably be a good idea to offer some command line tools for manipulating such .exes, to e.g. show/change what Python it's set to use, extract/dump/replace the zip, etc. I'd say tools supporting the format are essential. exe/zip formats will never be as user friendly as a pure text file script - we need to make the extra effort as minimal as possible. In particular, see my other post - I don't want to have one format (exe) for installed commands packaged with setuptools, and a separate format for one-file scripts I write myself. Actually, this sounds like a very good solution. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Better suggestion: just append a PEP 441 .pyz to the .exe, and no IIUC PEP 441 is about tooling to create archives; don't we just need a Python-compatible .zip (i.e. with a __main__.py)? For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) This is less helpful; one might have N scripts per project, no need to stick the whole project in with each one, or am I misunderstanding? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good True. idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? That could probably be taken care of with a bit of footwork. No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. How would such an offset be used? Are you saying the -c scriptlet would use that offset to extract the script? Or do you mean something else? (As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.) It's needed only if you use the specific approach I used - which was to use the Windows UpdateResource API to embed the script as a pukka Windows resource. Of course, if you're just appending something to the .exe, you don't need ctypes. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 09:58, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Better suggestion: just append a PEP 441 .pyz to the .exe, and no IIUC PEP 441 is about tooling to create archives; don't we just need a Python-compatible .zip (i.e. with a __main__.py)? For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) This is less helpful; one might have N scripts per project, no need to stick the whole project in with each one, or am I misunderstanding? I believe PJE's suggestion is to expand the scope of PEP 441 a bit to include the ability to generate a valid Windows executable with the archive appended, instead of just the *nix/Python launcher shebang line. If you prepend the shebang line, you get a .pya file (Note: I suggested to Daniel that the extension be changed to pya for Python application, but the PEP hasn't been updated yet), if you prepend the executable, you get an actual Windows executable. That would also let us avoid the need for a separate .pyaw extension - you would just prepend a Windows GUI executable to handle that case, with .pya only handling console applications. The script wrapper case would then just be a particular use of the PEP 441 executable generation features: prepend the console executable for console scripts and the GUI executable for GUI wrappers, with the wrapper itself being a __main__.py file in the attached archive. Given that Vinay already gave the Python launcher the ability to do this (when built with different options), this sounds quite feasible to me. Aside from the lack of embedded C extension support (which could likely be fixed if zipimport was migrated to Python code for 3.5), you'd have the essentials of py2exe right there in the standard library :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Changing the install hooks mechanism for PEP 426
I spent last weekend at Flock to Fedora, mostly due to my day job working on the Beaker integration testing system (http://beaker-project.org) for Red Hat, but also to talk to folks about Fedora and Python interactions. A completely unexpected discovery over the weekend, was that some of the RPM folks are exploring the idea of switching the *user* facing format for the packaging system away from spec files and towards directly executable Python code. Thus, you'd get away from the painful mess that is RPM conditionals and macros and have a real programming language to define what your built packages *should* look like, while still *producing* static metadata for consumption by installers and other software distribution tools. Hmm, does that approach sound familiar to anyone? :) Anyway, we were talking about how they're considering approaching the install hook problem, and their approach gave me an idea for a better solution in PEP 426. Currently, PEP 426 allows a distribution to define install hooks: hooks that will execute after the distribution is installed and before it is uninstalled. I'm now planning to change that to allowing distributions to define export hooks, based on the cleaned up notion of export groups in the latest version of PEP 426. An export hook definition consists of the following fields: * group - name of the export group to hook * preupdate - export to call prior to installing/updating/removing a distribution that exports this export group * postupdate - export to call after installing/updating/removing a distribution that exports this export group * refresh - export to call to resynchronise any caches with the system state. This will be invoked for every distribution on the system that exports this export group any time the distribution defining the export hook is itself installed or upgraded If a distribution exports groups that it also defines hooks for, it will exhibit the following behaviours: Fresh install: * preupdate NOT called (hook not yet registered) * postupdate called * refresh called Upgrade: * preupdate called (old version) * postupdate called (new version) * refresh called (new version) Complete removal: * preupdate called * postupdate NOT called (hook no longer registered) * refresh NOT called (hook no longer registered) This behaviour follows naturally from *not* special casing self-exports: prior to installation, the export hooks won't be registered, so they won't be called, and the same applies following complete removal. The hooks would have the following signatures: def preupdate(current_meta, next_meta): # current_meta==None indicates fresh install # next_meta==None indicates complete removal def postupdate(previous_meta, current_meta): # previous_meta==None indicates fresh install # current_meta==None indicates complete removal def refresh(current_meta): # Used to ensure any caches are consistent with system state # Allows handling of previously installed distributions Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 14:48, Paul Moore p.f.mo...@gmail.com wrote: But I do see your point regarding things like subprocess. It's a shame, but anything other than exes do seem to be second class citizens on Windows. BTW, you mention bat files - it bugs me endlessly that bat files seem to have a more privileged status than other script formats whether that's .py or .ps1 or whatever. I've never managed to 100% convince myself that they are special in a way that you can't replicate with suitable settings (PATHEXT, etc, etc). I think it's that .bat is hard-coded in the OS search algorithm or something, though. I think it is hard-coded into CreateProcess (at least on some versions of Windows). It certainly isn't a documented feature, but as demonstrated in my previous post it does work on XP. The docs are not easy to locate on the various aspects of matter. I just tried to find documentation but all I found was this (with dead-links to MS): http://blog.kalmbachnet.de/?postid=34 (If bat files didn't have their horrible nesting and ctrl-C handling behaviours, they'd be a viable solution...) You were right to cry about these previously. To give an example of where these subprocess issues might matter. sphinx auto-generates Makefiles that call 'sphinx-build' with no extension. The sphinx-build command has a setuptools .exe wrapper so that it will be picked up. I wouldn't confidently assume that for all combinations of Windows version and 'make' implementation that 'make' would know how to find sphinx-build for anything other than an .exe. A quick experiment shows that my own make handles shebangs if present and then falls back to just calling CreateProcess which handles .exe files and (via the undocumented hack above) .bat files . It does not respect PATHEXT and the error when the extension is provided but no shebang is given clearly shows it using the same sys-call as used by Python's subprocess module: Q:\tmpshow main 'show' is not recognized as an internal or external command, operable program or batch file. Q:\tmptype Makefile all: mycmd.py Q:\tmptype mycmd.py print 'hello' Q:\tmpmake mycmd.py process_begin: CreateProcess(Q:\tmp\mycmd.py, mycmd.py, ...) failed. make (e=193): Error 193 make: *** [all] Error 193 Q:\tmpmycmd.py hello Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan ncogh...@gmail.com wrote: I spent last weekend at Flock to Fedora, mostly due to my day job working on the Beaker integration testing system (http://beaker-project.org) for Red Hat, but also to talk to folks about Fedora and Python interactions. A completely unexpected discovery over the weekend, was that some of the RPM folks are exploring the idea of switching the *user* facing format for the packaging system away from spec files and towards directly executable Python code. Thus, you'd get away from the painful mess that is RPM conditionals and macros and have a real programming language to define what your built packages *should* look like, while still *producing* static metadata for consumption by installers and other software distribution tools. Hmm, does that approach sound familiar to anyone? :) Anyway, we were talking about how they're considering approaching the install hook problem, and their approach gave me an idea for a better solution in PEP 426. Currently, PEP 426 allows a distribution to define install hooks: hooks that will execute after the distribution is installed and before it is uninstalled. I'm now planning to change that to allowing distributions to define export hooks, based on the cleaned up notion of export groups in the latest version of PEP 426. An export hook definition consists of the following fields: * group - name of the export group to hook * preupdate - export to call prior to installing/updating/removing a distribution that exports this export group * postupdate - export to call after installing/updating/removing a distribution that exports this export group * refresh - export to call to resynchronise any caches with the system state. This will be invoked for every distribution on the system that exports this export group any time the distribution defining the export hook is itself installed or upgraded If a distribution exports groups that it also defines hooks for, it will exhibit the following behaviours: Fresh install: * preupdate NOT called (hook not yet registered) * postupdate called * refresh called Upgrade: * preupdate called (old version) * postupdate called (new version) * refresh called (new version) Complete removal: * preupdate called * postupdate NOT called (hook no longer registered) * refresh NOT called (hook no longer registered) This behaviour follows naturally from *not* special casing self-exports: prior to installation, the export hooks won't be registered, so they won't be called, and the same applies following complete removal. The hooks would have the following signatures: def preupdate(current_meta, next_meta): # current_meta==None indicates fresh install # next_meta==None indicates complete removal def postupdate(previous_meta, current_meta): # previous_meta==None indicates fresh install # current_meta==None indicates complete removal def refresh(current_meta): # Used to ensure any caches are consistent with system state # Allows handling of previously installed distributions I think I'm okay with this so long as it remains optional. I'm not crazy about executable build specs where they're not necessary. For most cases, especially in pure Python packages, it's frequently overkill and asking for trouble. So I would still want to see a well-accepted static build spec for Python packages too (sort of a la setup.cfg as parsed by d2to1, only better), though I realize that's a separate issue from PEP 426. Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 9:58 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
IIUC PEP 441 is about tooling to create archives; don't we just need a
Python-compatible .zip (i.e. with a __main__.py)?
I meant that it has a #! line before the zip part, so that the
launcher knows what Python to invoke.
There are also some challenges for older Pythons to invoke __main__,
since the normal Python import machinery frowns on reloading __main__.
I expect the zip would need an extra __main.py stub to bootstrap the
loading of __main__, and then invoke python with something like '-c
__import__('sys').path[0:0]=['/path/to','path/to/exe''];
__import__('__main').go()'.
(It can't have the import run the app as a side effect, because
otherwise the import lock will be held, leading to Bad Things in
multi-threaded apps.)
This is less helpful; one might have N scripts per project, no need to stick
the whole project in with each one, or am I misunderstanding?
I just meant that for cases where there's only one script, or where
you are doing a custom-built application. This also becomes The One
Obvious Way to do py2exe-like things.
How would such an offset be used? Are you saying the -c scriptlet would use
that offset to extract the script? Or do you mean something else?
Extract the script by seeking to the offset and reading it. It's far
from ideal, though; the .zip is much better since everything back to
2.3 can support it in some fashion.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP440 and fork versioning
I'm wondering if PEP440 should recommend how to version forks? It's fairly common to fork dependencies temporarily until the change can be released upstream. Ideally, you want to version a fork (and keep the same name) so that it fulfills the requirement, but be obvious that it's a fork. Although pip allows overriding requirement consistency, consistency is preferred, and needed in cases where a `pkg_resources.require` enforces it in a console script. As it is now, the post-release scheme works for this, but it's not the intended use case. Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan ncogh...@gmail.com wrote: * group - name of the export group to hook * preupdate - export to call prior to installing/updating/removing a distribution that exports this export group * postupdate - export to call after installing/updating/removing a distribution that exports this export group * refresh - export to call to resynchronise any caches with the system state. This will be invoked for every distribution on the system that exports this export group any time the distribution defining the export hook is itself installed or upgraded I've reread your post a few times and I'm not sure I understand it. Let me try and spell out a scenario to see if I've got it: * Distribution A defines a refresh hook for group 'foo.bar' -- but doesn't export anything in that group * Distribution B defines an *export* (fka entry point) -- any export -- in export group 'foo.bar' -- but doesn't define any hooks * Distribution A's refresh hook will be notified when B is installed, updated, or removed Is that what this is for? If so, my confusion is probably because of overloading of the term export in this context; among other things, it's unclear whether this is a separate data structure from exports themselves... and if so, why? If I were doing something like this in the existing entry point system, I'd do something like: [mebs.refresh] foo.bar = my.hook.module:refresh i.e., just list the hooks in an export group, using the export name to designate what export group is being monitored. This approach leverages the fact that exports already need to be indexed, so why create a whole new sort of metadata just for the hooks? (But of course if I have misunderstood what you're trying to do in the first place, this and my other thoughts may be moot.) (Oh, and btw, if a distribution has hooks for itself, then how are you going to invoke two different versions of the code? Rollback sys.modules and reload? Spawn another process?) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP440 and fork versioning
I'm noticing the mention of forks in PEP426 for provides. so theoretically, `pkg_resources.WorkingSet.resolve` would be updated at some point to account for provides in PEP426, and this feature would be surfaced as a setup keyword for users to use in their fork projects. On Wed, Aug 14, 2013 at 10:53 AM, Marcus Smith qwc...@gmail.com wrote: I'm wondering if PEP440 should recommend how to version forks? It's fairly common to fork dependencies temporarily until the change can be released upstream. Ideally, you want to version a fork (and keep the same name) so that it fulfills the requirement, but be obvious that it's a fork. Although pip allows overriding requirement consistency, consistency is preferred, and needed in cases where a `pkg_resources.require` enforces it in a console script. As it is now, the post-release scheme works for this, but it's not the intended use case. Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 16:49, Oscar Benjamin oscar.j.benja...@gmail.com wrote: To give an example of where these subprocess issues might matter. sphinx auto-generates Makefiles that call 'sphinx-build' with no extension. The sphinx-build command has a setuptools .exe wrapper so that it will be picked up. I wouldn't confidently assume that for all combinations of Windows version and 'make' implementation that 'make' would know how to find sphinx-build for anything other than an .exe. OK, that's a pretty solid use case, and pretty clearly demonstrates that there will be issues with anything other than an exe. So we come full circle again - I'm pretty sure the last time this came up a month or so ago, someone came up with a scenario that convinced me to give up on executable script files. I definitely will at some point write up *some* sort of document on best practices for wrapping Python code (scripts, apps, whatever) as OS commands, in a cross-platform way. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 14 August 2013 18:46, PJ Eby p...@telecommunity.com wrote: Shouldn't naming the file .pyw already work today for that case? Certainly, the .pyw extension is already suitable for manually creating GUI scripts in a text editor. Unless there's something special about how the 'pythonw' executable processes the command line, it should work just as well for a zipped archive. .pyw files can be imported as modules, just like .py, so you hit the issue of scripts named the same as modules that they import. Naming a zipped archive .pyw is no better or worse than naming it .py - both work most of the time, but are disconcerting to users who think the filetype implies text and both have the problems associated with being both executable and importable. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP440 and fork versioning
Distros actually need to do this fairly regularly for security patches and packaging tweaks, so it may be a good idea. I think local updates were one of the intended uses for post-releases, but that doesn't work if upstream is also using that suffix (and we know some projects do). *If* this was added to the PEP, I would add it as a new optional .localN suffix, with a recommendation that public index servers MUST disallow use of local numbering, since it is intended for downstream integrators to indicate the inclusion of additional changes relative to the upstream version. Outside a given integrators environment, the local numbering is no longer valid. Currently, the PEP assumes the use of an external numbering system to track that kind of local change (e.g. with RPM, one common tactic is to increase the release level rather than the version, so the version continues to match the upstream base). The rationale for adding it directly to the PEP would be to let people accurately track versions for local modifications, even if they're using virtualenv as their integration environment. Provides is a bit different, in that it covers more permanent forks and name changes, rather than the fix things locally for immediate use, submit upstream patch as a good open source citizen and backport selected bug fixes from later versions workflows that are pretty common in larger integration projects like Linux distros. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP440 and fork versioning
*If* this was added to the PEP, I would add it as a new optional .localN suffix, with a recommendation that public index servers MUST disallow use of local numbering, since it is intended for downstream integrators to indicate the inclusion of additional changes relative to the upstream version. Outside a given integrators environment, the local numbering is no longer valid. Seems like you have to add something like this, if you're not wanting provides to cover it. Like you mention, .post won't work in all cases. Otherwise people are left not changing the version, or changing the version in a way that could break consistency. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On 14 August 2013 14:00, PJ Eby p...@telecommunity.com wrote:
On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan ncogh...@gmail.com wrote:
* group - name of the export group to hook
* preupdate - export to call prior to installing/updating/removing a
distribution that exports this export group
* postupdate - export to call after installing/updating/removing a
distribution that exports this export group
* refresh - export to call to resynchronise any caches with the system
state. This will be invoked for every distribution on the system that
exports this export group any time the distribution defining the
export hook is itself installed or upgraded
I've reread your post a few times and I'm not sure I understand it.
Let me try and spell out a scenario to see if I've got it:
* Distribution A defines a refresh hook for group 'foo.bar' -- but
doesn't export anything in that group
* Distribution B defines an *export* (fka entry point) -- any export
-- in export group 'foo.bar' -- but doesn't define any hooks
* Distribution A's refresh hook will be notified when B is installed,
updated, or removed
No, A's preupdate and postupdate hooks would fire when B (or any other
distro exporting the foo.bar group) is installed/updated/removed.
refresh() would fire only when A was installed or updated.
I realised that my proposed signature for the refresh() hook is wrong,
though, since it doesn't deal with catching up on *removed*
distributions properly. Rather than being called multiple times,
refresh() instead needs to be called with an iterable providing the
metadata for all currently installed distributions that export that
group.
Is that what this is for?
If so, my confusion is probably because of overloading of the term
export in this context; among other things, it's unclear whether
this is a separate data structure from exports themselves... and if
so, why?
Where exports is about publishing entries into an export group, the
new export_hooks field would be about *subscribing* to an export
group and being told about changes to it.
While you could use a naming convention to defined these hooks
directly in exports without colliding with the export of the group
itself, but I think it's better to separate them out so you can do
stricter validation on the permitted keys and values (the rationale is
similar to that for separating out commands from more general exports,
and exports from arbitrary metadata extensions).
You're right the name should be a key in a mapping (like exports)
rather than a subfield in a list, though. That means I'm envisioning
something like the following:
Distribution foo:
export_hooks : {
foo.bar: {
preupdate: foo.bar:exporter_update_started
postupdate: foo.bar:exporter_update_completed
refresh: foo.bar:resync_cache
}
}
When foo is installed or updated, then the installer would invoke
foo.bar.resync_cache with an iterable of all currently installed
distributions that export the foo.bar export group.
Distribution notfoo:
exports : {
foo.bar: {}
}
When notfoo is installed, updated or removed, then
foo.bar:exporter_update_started would be called prior to changing
anything, and foo.bar:exporter_update_completed would be called when
the changes had been made.
If I were doing something like this in the existing entry point
system, I'd do something like:
[mebs.refresh]
foo.bar = my.hook.module:refresh
i.e., just list the hooks in an export group, using the export name to
designate what export group is being monitored. This approach
leverages the fact that exports already need to be indexed, so why
create a whole new sort of metadata just for the hooks?
Mostly so you can validate them and display them differently, and
avoid reserving any part of the shared namespace. I find documentation
is also easier when the core use cases aren't wedged into the
extension mechanisms (even if they share implementation details under
the hood).
(But of course if I have misunderstood what you're trying to do in the
first place, this and my other thoughts may be moot.)
I think you mostly understood it, I'm just not explaining it very well yet.
(Oh, and btw, if a distribution has hooks for itself, then how are you
going to invoke two different versions of the code? Rollback
sys.modules and reload? Spawn another process?)
Requiring that all hook invocations happen in a subprocess sounds like
the best plan to me. The arguments all serialise nicely to JSON so
that shouldn't be too hard to arrange.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP440 and fork versioning
On 14 August 2013 15:07, Marcus Smith qwc...@gmail.com wrote: *If* this was added to the PEP, I would add it as a new optional .localN suffix, with a recommendation that public index servers MUST disallow use of local numbering, since it is intended for downstream integrators to indicate the inclusion of additional changes relative to the upstream version. Outside a given integrators environment, the local numbering is no longer valid. Seems like you have to add something like this, if you're not wanting provides to cover it. Like you mention, .post won't work in all cases. Otherwise people are left not changing the version, or changing the version in a way that could break consistency. Yeah, I agree. To help keep track of this stuff (and to make pull requests a possibility), I created a PyPI Metadata Formats repo on BitBucket and filed this as the first issue: https://bitbucket.org/pypa/pypi-metadata-formats/issue/1/add-local-numbering-to-pep-440 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Wed, Aug 14, 2013 at 3:14 PM, Nick Coghlan ncogh...@gmail.com wrote: On 14 August 2013 14:00, PJ Eby p...@telecommunity.com wrote: On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan ncogh...@gmail.com wrote: * group - name of the export group to hook * preupdate - export to call prior to installing/updating/removing a distribution that exports this export group * postupdate - export to call after installing/updating/removing a distribution that exports this export group * refresh - export to call to resynchronise any caches with the system state. This will be invoked for every distribution on the system that exports this export group any time the distribution defining the export hook is itself installed or upgraded I've reread your post a few times and I'm not sure I understand it. Let me try and spell out a scenario to see if I've got it: * Distribution A defines a refresh hook for group 'foo.bar' -- but doesn't export anything in that group * Distribution B defines an *export* (fka entry point) -- any export -- in export group 'foo.bar' -- but doesn't define any hooks * Distribution A's refresh hook will be notified when B is installed, updated, or removed No, A's preupdate and postupdate hooks would fire when B (or any other distro exporting the foo.bar group) is installed/updated/removed. refresh() would fire only when A was installed or updated. Huh? So refresh is only relevant to the package itself? I guess I don't understand the point of that, since you get the same info from postupdate then, no? I realised that my proposed signature for the refresh() hook is wrong, though, since it doesn't deal with catching up on *removed* distributions properly. Rather than being called multiple times, refresh() instead needs to be called with an iterable providing the metadata for all currently installed distributions that export that group. Ah. But then why is it called for A, instead of.. oh, I think I see now. Gotcha. This is the sort of thing that examples are really made for, so you can see the use cases for the different hooks. If so, my confusion is probably because of overloading of the term export in this context; among other things, it's unclear whether this is a separate data structure from exports themselves... and if so, why? Where exports is about publishing entries into an export group, the new export_hooks field would be about *subscribing* to an export group and being told about changes to it. That's not actually a justification for not using exports. While you could use a naming convention to defined these hooks directly in exports without colliding with the export of the group itself, but I think it's better to separate them out so you can do stricter validation on the permitted keys and values (the rationale is similar to that for separating out commands from more general exports, and exports from arbitrary metadata extensions). The separation of commands is (just barely) justifiable because it's not a runtime use, it's installer use. Stricter validation, OTOH, is a completely bogus justification for not using exports, otherwise nobody would ever have any reason to use exports, everybody would have to define their own extensions so they could have stricter validation. ;-) The solution to providing more validation is to use *more* export groups, e.g.: [mebs.export_validators] mebs.refresh = module.that.validates.keys.in.the.refresh.group:somefunc (In other words, define hooks for validating export groups, the way setuptools uses an entry point group for validating setup keywords.) Of course, even without that possibility, the stricter validation concept is kind of bogus here: the only thing you can really validate is that syntactically valid group names are being used as export names, which isn't much of a validation. You can't *semantically* validate them, since there is no global registry of group names. So what's the point? The build system *should* reserve at least one (subdivisible) namespace for itself, and use that mechanism for its own extension, for two reasons: 1. Entities should not be multiplied beyond necessity, 2. It serves as an example of how exports are to be used, and 3. The API is reusable... No, three reasons! Wait, I'll come in again... the API is reusable, it serves as an example, no duplication, and namespaces are a good idea, let's do more of them... no, four reasons... chief amongst the reasonry... Seriously: I can *sort of* see a reason to keep commands separate, but that's a meh. I admittedly just grabbed it as a handy way to shoehorn that functionality into setuptools. But keeping extensions to the build system itself in a separate place? No, a thousand times no. This sort of extensibility is *precisely* what the darn things are *for*. If the build system doesn't use them, what's the point? Mostly so you can validate them and display them differently, and avoid reserving any part of
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 2:14 PM, Paul Moore p.f.mo...@gmail.com wrote: .pyw files can be imported as modules, just like .py, Darn. Okay, so another extension *is* needed if you want to be able to make non-console apps runnable-but-not-importable. IIUC it should by '.pywa' rather than '.pya', though, because of the issue with only the first three characters of an extension working in PowerShell, which means it would be executed by the wrong PyLauncher under some circumstances. (Honestly, though, I'm not sure whether anybody cares about PATH/PATHEXT in relation to GUI apps; ISTM they'd mostly be invoked by shortcuts, and there's also a registry mechanism that's supposed to be used for these things nowadays, rather than PATH... but I think it only works for .exe's, so there we go again back to the land of .exe's just plain Work Better On Windows.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] help requested getting existing project gv into PyPI
On Fri, Aug 9, 2013 at 12:46 PM, PJ Eby p...@telecommunity.com wrote: update the download links whenever a new version is released. Note, too, that unless there is a specific download for the Python binding that uses setup.py to invoke its build process, a PyPI listing won't make the binding pip-installable. Yes it appears setup.py is not used. Thanks, and nevermind! ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Realistic PyPI, pip and TUF demo
Hello everyone, We now have a demonstration of pip that securely and efficiently downloads with TUF any package from a PyPI mirror: https://github.com/theupdateframework/pip/wiki/pip-over-TUF We hope that you will try our demonstration with your favourite packages and tell us about any issue that you find. TUF does not yet work on Microsoft Windows and Apple OS X. This is because it depends for cryptography on a custom Python library (evpy) which binds with OpenSSL. We are planning to fix this by moving to the cross-platform Mozilla Network Security Services (NSS) library. We also welcome your thoughts on features and enhancements that you would like to see. Our next demo will show security flaws in package managers such as pip that do not use TUF. We will then see how pip with TUF addresses those security attacks. -The TUF team ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
PJ Eby pje at telecommunity.com writes: The build system *should* reserve at least one (subdivisible) namespace for itself, and use that mechanism for its own extension, +1 - dog-food :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
