Re: [Distutils] Replacing pip.exe with a Python script
On Tue, Jul 16, 2013 at 8:23 AM, Paul Moore p.f.mo...@gmail.com wrote: On 16 July 2013 12:42, Oscar Benjamin oscar.j.benja...@gmail.com wrote: I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this). Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I said, I don't know for sure at the moment, but it needs investigating. That's not why they exist; the .exe's don't load the Python DLL, they just invoke python.exe. The existence of separate 32- and 64-bit .exe's is a Distribute innovation, actually; setuptools 0.6 doesn't use them. Instead, setuptools writes a manifest file to tell Windows that it doesn't need privilege escalation or to create a separate console. This meant that only one (32-bit) .exe was needed. I forget what happened with the Distribute approach or why the 64-bit version was kept after the merge; ISTM there was some other use for it, or at least Jason thought so. But DLL loading is not the reason. (Actually, after searching my email, my guess is that there actually *isn't* any need for the 64-bit .exe; ISTM that it was introduced solely as a false fix for the privilege escalation problem, that only fixes it for 64-bit Windows and doesn't help 32-bit.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
Am 15.07.2013 19:26, schrieb Donald Stufft: Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues. pip.bat? Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 17 July 2013 14:13, Thomas Heller thel...@ctypes.org wrote: Am 15.07.2013 19:26, schrieb Donald Stufft: Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues. pip.bat? That's my cue to cry :-) If you missed my earlier comments about bat files, then no - bat files have a significant number of failings that make them unsuitable for this sort of thing. The simplest example I can give is that bat files don't nest. So if the pip command is implemented as a bat file, and you have a script to build your virtualenv that looks like this: @echo off virtualenv foo foo\scripts\activate pip install wibble echo Complete! then the final message would never be executed as the pip bat file would not return to its caller. (Actually, the activate command is also a bat file, so pip would never be executed either, but you get my point). That's the worst sort of silent failure, and I have spent significant time in the past debugging scripts that fell foul of this behaviour. To fix this you have to use call pip install wibble. And once you admit the possibility that certain commands could be implemented as bat files you have to either check everything, or use call everywhere, even when not necessary (e.g. call python) just to be safe. If you don't think this problem is sufficient, I can offer more :-( I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 17 July 2013 14:49, Thomas Heller thel...@ctypes.org wrote: Yes, missed them. Sorry that I did not read enough of the discussion, I just stumbled over some messages in this thread and .bat popped up in my head. No problem. It's probably useful to have a summary of the issues with bat files (more accurately, the reasons we need exe wrappers) written down here in any case. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
Am 17.07.2013 15:33, schrieb Paul Moore: On 17 July 2013 14:13, Thomas Heller thel...@ctypes.org mailto:thel...@ctypes.org wrote: Am 15.07.2013 19:26, schrieb Donald Stufft: Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues. pip.bat? That's my cue to cry :-) If you missed my earlier comments about bat files, then no - bat files have a significant number of failings that make them unsuitable for this sort of thing. Yes, missed them. Sorry that I did not read enough of the discussion, I just stumbled over some messages in this thread and .bat popped up in my head. I still use them for quite some things although I have been bitten by their problems myself often enough. I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On Jul 17, 2013, at 3:12 PM, Paul Moore p.f.mo...@gmail.com wrote: The problem issue remaining is recognising when we need to do this. In terms of code paths, pip install -U pip is no different from (for example) pip install -U flask. But it needs to be handled specially just because it's pip. And it *doesn't* need to be handled specially if it's python -m pip install -U pip. That's not a Windows issue, though, I was just using the Windows issue to put off having to think about it :-) OTOH it doesn't *hurt* to do it so if a conditional is hard to sort out (I'm thinking explicitly about pip vs python -mpip) we can just genericize it. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 17/07/2013 20:12, Paul Moore wrote: On 17 July 2013 19:55, Steve Dower steve.do...@microsoft.com wrote: I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results. No reason to be afraid of this, exe wrappers are totally the best option. As for updating .exes while they're running, the best approach is to rename the running one (e.g. 'pip.exe' - 'pip.exe.deleteme') in the same folder and either: * delete any existing .deleteme files on next run, or * delete an existing pip.exe.deleteme file immediately before trying to rename to it Any other approach will also have corner cases, but this will be the most reliable in the context of multiple users/permissions/environment variables. The problem issue remaining is recognising when we need to do this. In terms of code paths, pip install -U pip is no different from (for example) pip install -U flask. But it needs to be handled specially just because it's pip. I don't think it does: in essence, any distribution which installs an .exe wrapper for some entry point can (possibly should) be treated the same way. Assuming that flask installed some kind of run-flask.exe in scripts/ you'd do the same thing: rename in-place so that the old one could keep running; write the new one under the old name; delete the old one (if you can). The issue of deleting the .deleteme versions of arbitrary scripts is that you don't know when they might be delete-able. In the Flask example, assume that someone was actually using run-flask.exe to run a Flask app, you can rename the .exe but you won't be able to delete it until it had finished running. Unless you have a watchdog script which watches the file / folder then you just have to call it .deleteme and clean up as best you can the next time round. TJG ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results. No reason to be afraid of this, exe wrappers are totally the best option. As for updating .exes while they're running, the best approach is to rename the running one (e.g. 'pip.exe' - 'pip.exe.deleteme') in the same folder and either: * delete any existing .deleteme files on next run, or * delete an existing pip.exe.deleteme file immediately before trying to rename to it Any other approach will also have corner cases, but this will be the most reliable in the context of multiple users/permissions/environment variables. Paul Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
The problem issue remaining is recognising when we need to do this. In terms of code paths, pip install -U pip is no different from (for example) pip install -U flask. But it needs to be handled specially just because it's pip. And it *doesn't* need to be handled specially if it's python -m pip install -U pip. That's not a Windows issue, though, I was just using the Windows issue to put off having to think about it :-) Not especially. You wouldn't want to do it for all files, but you can do the rename for all .exe files and then try and delete the .deleteme immediately. (Or you can try and replace and gracefully handle the exception.) As long as failing to delete the old .exe doesn't cause installation to fail, 99% of the time the end result is identical to now (and the other 1% is better, because install succeeded where it would currently fail). One thought, while I have a Windows expert's attention ;-) Is there a way (I'm not too bothered how complex it might be) of doing the equivalent of Unix exec in Windows? That is, start a new process and then have my initial process exit *without* the shell (or whatever started the first process) getting control back until the child completes? I'm assuming not, as otherwise solving the issue would be as easy as exec-ing python -m pip from the wrapper. But no harm in asking :-) I had this discussion with a colleague earlier today, and there really isn't. You'd need to be cooperating with the process that started you initially (cmd.exe/powershell.exe/whatever) and if there is a way to do it, (a) I don't know what it is, and (b) it's unlikely to be consistent/reliable/documented. At best, you could start a new process at the end of installation that keeps trying to delete the .deleteme file until it succeeds (or waits for a specific process to exit). However, my gut says that it's safer just to leave the file around and try and delete it later, especially since pip is going to be used in such a wide variety of contexts. Being clever is more likely to get you into trouble. Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 18 Jul 2013 05:13, Paul Moore p.f.mo...@gmail.com wrote: On 17 July 2013 19:55, Steve Dower steve.do...@microsoft.com wrote: I'm afraid exe files as wrappers are probably the only viable option. The basic reason is that the OS recognises exe files in all context, with no special configuration needed. This is not true of any other file type. So anything else will have corner cases that will give unexpected results. No reason to be afraid of this, exe wrappers are totally the best option. As for updating .exes while they're running, the best approach is to rename the running one (e.g. 'pip.exe' - 'pip.exe.deleteme') in the same folder and either: * delete any existing .deleteme files on next run, or * delete an existing pip.exe.deleteme file immediately before trying to rename to it Any other approach will also have corner cases, but this will be the most reliable in the context of multiple users/permissions/environment variables. Cool. I'm not happy about the clutter of '.deleteme' files, and I'll still look for a way to delete them straight after the upgrade process terminates, but I may have to settle for lazy deletion. Call them .previous and it looks less ugly, in my opinion. You can also just try deletion first and only if that fails do the renaming trick. EAFP and all that :) Cheers, Nick. The problem issue remaining is recognising when we need to do this. In terms of code paths, pip install -U pip is no different from (for example) pip install -U flask. But it needs to be handled specially just because it's pip. And it *doesn't* need to be handled specially if it's python -m pip install -U pip. That's not a Windows issue, though, I was just using the Windows issue to put off having to think about it :-) One thought, while I have a Windows expert's attention ;-) Is there a way (I'm not too bothered how complex it might be) of doing the equivalent of Unix exec in Windows? That is, start a new process and then have my initial process exit *without* the shell (or whatever started the first process) getting control back until the child completes? I'm assuming not, as otherwise solving the issue would be as easy as exec-ing python -m pip from the wrapper. But no harm in asking :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 15 July 2013 23:21, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: I think I'm coming to the conclusion that the best way forward is: 1. Continue using the setuptools exe launcher, but bundle our own copy. Wouldn't that be the case with a bundled setuptools/pip anyway? The launcher executables are part of setuptools AFAIK. 2. Modify setup.py to install our own scripts run via the exe launcher, which don't rely on entry points and pkg_resources. FYI the distlib executable launchers (not based on setuptools, but work the same way) and the distlib script generation approach does not require distlib to run. You might be able to make use of that in some way :-) Yes, the physical executables aren't that much of an issue - grabbing them from somewhere and bundling them is easy enough and as you say we may already have them (I actually thought distlib included the exes, but the version bundled with pip at the moment doesn't have them). The fun bit is having to modify setuptools to do our own script wrapper management, because setuptools doesn't let us customise its process to remove the runtime dependency on a top-level pkg_resources. Having to make project-specific customisations to distutils feels like we're going in precisely the opposite direction from what the whole packaging process is trying to achieve, and it makes me feel vaguely unclean having to consider it :-) Two thoughts for the wider audience: 1. Should pip re-vendor a newer version of distlib, so we have the exe wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI. 2. Would writing a distutils extension class in setup.py to make the exe wrappers using the vendored distlib.scripts package be acceptable to remove the runtime dependency on pkg_resources from the wrappers? Note: This is just one relatively small step towards removing some of our dependencies on an external setuptools. It's not the whole story, and it still leaves the upgrading a running exe wrapper problem to address. (This discussion may be worth migrating to pypa-dev, I'm not sure how much the wider distutils audience cares about this level of detail - feel free to switch lists anyone who thinks it's appropriate). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 16 July 2013 11:28, Paul Moore p.f.mo...@gmail.com wrote: Two thoughts for the wider audience: 1. Should pip re-vendor a newer version of distlib, so we have the exe wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI. In what way would that affect anyone? 2. Would writing a distutils extension class in setup.py to make the exe wrappers using the vendored distlib.scripts package be acceptable to remove the runtime dependency on pkg_resources from the wrappers? Does that mean that an end user would need a C compiler in a situation where they previously didn't? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 16 July 2013 12:04, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On 16 July 2013 11:28, Paul Moore p.f.mo...@gmail.com wrote: Two thoughts for the wider audience: 1. Should pip re-vendor a newer version of distlib, so we have the exe wrappers? We currently have 0.1.1 and 0.1.2 is on PyPI. In what way would that affect anyone? Sorry, you're right - that's really for the pip developers. 2. Would writing a distutils extension class in setup.py to make the exe wrappers using the vendored distlib.scripts package be acceptable to remove the runtime dependency on pkg_resources from the wrappers? Does that mean that an end user would need a C compiler in a situation where they previously didn't? I don't believe so - distlib bundles the compiled code. On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 16 July 2013 12:25, Paul Moore p.f.mo...@gmail.com wrote: 2. Would writing a distutils extension class in setup.py to make the exe wrappers using the vendored distlib.scripts package be acceptable to remove the runtime dependency on pkg_resources from the wrappers? Does that mean that an end user would need a C compiler in a situation where they previously didn't? I don't believe so - distlib bundles the compiled code. On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip... I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this). Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 16 July 2013 12:42, Oscar Benjamin oscar.j.benja...@gmail.com wrote: I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this). Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I said, I don't know for sure at the moment, but it needs investigating. Grumble. Next time the label on the can says worms I need to leave the can opener alone :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 16 Jul, 2013, at 13:25, Paul Moore p.f.mo...@gmail.com wrote: On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip... Couldn't you just ship both variants of the exe wrappers in a single distribution and then use the correct one for the current installation? That's what I'm doing in py2app. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 16 July 2013 22:23, Paul Moore p.f.mo...@gmail.com wrote: On 16 July 2013 12:42, Oscar Benjamin oscar.j.benja...@gmail.com wrote: I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this). Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I said, I don't know for sure at the moment, but it needs investigating. Grumble. Next time the label on the can says worms I need to leave the can opener alone :-) I, for one, am happy you opened it now rather than in a few months time! If Paul Nasrat takes up the challenge of documenting all the practical issues associated with bundling pip with the CPython binary installers as a PEP, I expect he will need to work closely with you to capture the details on the Windows side of things. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 15 July 2013 14:39, Paul Moore p.f.mo...@gmail.com wrote: I'm looking at the possibility of replacing the current setuptools entry point based pip executables with Python scripts. The biggest problem is that a script pip.py shadows the pip package, making import pip fail. I can get round this by deleting sys.path[0] (the location of the currently running script) but how robust is that? Are there any corner cases where it would break? Or alternatively, is there a better way to do this rather than messing directly with sys.path? I suspect this is a fairly common question, but my Google-fu is failing me :-( Can you not just rename the pip module to _pip? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On Mon, Jul 15, 2013 at 9:39 AM, Paul Moore p.f.mo...@gmail.com wrote: I'm looking at the possibility of replacing the current setuptools entry point based pip executables with Python scripts. The biggest problem is that a script pip.py shadows the pip package, making import pip fail. I can get round this by deleting sys.path[0] (the location of the currently running script) but how robust is that? Are there any corner cases where it would break? Or alternatively, is there a better way to do this rather than messing directly with sys.path? I suspect this is a fairly common question, but my Google-fu is failing me :-( Sorry, I know this is a basic Python coding question - in my defence, it's for something related to the current pip discussions :-) As long as you make sure that sys.path[0] is actually the script location then it will work (other things like .pth files, PYTHONSTARTUP, etc. could have changed things before your script started execution). But realize that a) in Python 3.3 the scripts location will be ./pip.py, not just pip.py, and b) if I get my way all paths will be absolute for __file__, so you will have to just associate '' with os.getcwd() and then search for the proper directory on sys.path. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 15 July 2013 14:44, Oscar Benjamin oscar.j.benja...@gmail.com wrote: Can you not just rename the pip module to _pip? That's a far more intrusive change, and I'm not sure I want to go there. It *will* break existing code that (rightly or wrongly) imports pip. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 15 July 2013 14:53, Brett Cannon br...@python.org wrote: As long as you make sure that sys.path[0] is actually the script location then it will work (other things like .pth files, PYTHONSTARTUP, etc. could have changed things before your script started execution). But realize that a) in Python 3.3 the scripts location will be ./pip.py, not just pip.py, and b) if I get my way all paths will be absolute for __file__, so you will have to just associate '' with os.getcwd() and then search for the proper directory on sys.path. OK, that pretty much tells me that this is a bad idea. It's never going to be robust enough to work. I'm amazed actually that there's no way to say don't add the script location to sys.path, even as a command line option. It seems like the sort of thing you'd want to make scripts robust, a bit like -S and -E. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 16 July 2013 00:16, Paul Moore p.f.mo...@gmail.com wrote: On 15 July 2013 14:53, Brett Cannon br...@python.org wrote: As long as you make sure that sys.path[0] is actually the script location then it will work (other things like .pth files, PYTHONSTARTUP, etc. could have changed things before your script started execution). But realize that a) in Python 3.3 the scripts location will be ./pip.py, not just pip.py, and b) if I get my way all paths will be absolute for __file__, so you will have to just associate '' with os.getcwd() and then search for the proper directory on sys.path. OK, that pretty much tells me that this is a bad idea. It's never going to be robust enough to work. Most of the stuff Brett mentioned there shouldn't be relevant for a directly executed script - doing sys.path.remove(os.path.dirname(os.path.abspath(__file__)) should be pretty robust in any scenario. I'm amazed actually that there's no way to say don't add the script location to sys.path, even as a command line option. It seems like the sort of thing you'd want to make scripts robust, a bit like -S and -E. You'd think that, but then you'd look at getpath.c and run away (or write something like PEP 432, as I did) :P Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 15 July 2013 15:32, Nick Coghlan ncogh...@gmail.com wrote: Most of the stuff Brett mentioned there shouldn't be relevant for a directly executed script - doing sys.path.remove(os.path.dirname(os.path.abspath(__file__)) should be pretty robust in any scenario. OK, well apart from the shadowing issue, my initial tests have looked relatively positive. So the questions are now more around whether this is how we want to go with pip, what backward compatibility issues it may have (the launcher isn't available in Python 3.3) etc. So I'll work up a pull request for discussion by the pip devs. I'm amazed actually that there's no way to say don't add the script location to sys.path, even as a command line option. It seems like the sort of thing you'd want to make scripts robust, a bit like -S and -E. You'd think that, but then you'd look at getpath.c and run away (or write something like PEP 432, as I did) :P You're a better man than I, Gunga Din :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On Jul 15, 2013, at 9:39 AM, Paul Moore p.f.mo...@gmail.com wrote: I'm looking at the possibility of replacing the current setuptools entry point based pip executables with Python scripts. The biggest problem is that a script pip.py shadows the pip package, making import pip fail. I can get round this by deleting sys.path[0] (the location of the currently running script) but how robust is that? Are there any corner cases where it would break? Or alternatively, is there a better way to do this rather than messing directly with sys.path? I suspect this is a fairly common question, but my Google-fu is failing me :-( Sorry, I know this is a basic Python coding question - in my defence, it's for something related to the current pip discussions :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On 15 July 2013 18:26, Donald Stufft don...@stufft.io wrote: Maybe this is a crazy idea, but would a windows only extension work? .pye(executable) Then just associate .pye with the launcher. Python won't see .pye as importable so there's no shadow issues. That's actually a very good idea. The only downside is the proliferation of extensions involved, and the need to register them. That puts it into the territory of things the installer needs to do if we're to be able to assume it. But I may propose it to python-dev (Daniel proposed a zipped Python app extension a while back, as well. I'm not sure what happened with that one...) Actually, this and many of the other ideas fall foul of backward compatibility issues - we can't assume the Python launcher is available prior to Python 3.3, so #! support in .py files isn't available either. I think I'm coming to the conclusion that the best way forward is: 1. Continue using the setuptools exe launcher, but bundle our own copy. 2. Modify setup.py to install our own scripts run via the exe launcher, which don't rely on entry points and pkg_resources. 3. Special case the heck out of pip upgrading itself to ignore errors from trying to replace the exe (as long as the exe is unchanged, based on a size/checksum check) This covers the replacing-the-exe issue and the entry point script problems of vendoring setuptools. I don't *like* this option, but at least it's not going to break big chunks of our userbase... There are some other options I'd still like to explore before settling on something, for example making pip install from sdist *always* build a temporary wheel and then always install from wheels - we can then introspect the wheel before installing and catch this type of issue before starting. That lets us easily avoid the overwriting the exe issue, as well as letting us cleanly roll back failed installs. One thing is clear - this is a longer term effort, not a quick fix... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
Paul Moore p.f.moore at gmail.com writes: I think I'm coming to the conclusion that the best way forward is: 1. Continue using the setuptools exe launcher, but bundle our own copy. Wouldn't that be the case with a bundled setuptools/pip anyway? The launcher executables are part of setuptools AFAIK. 2. Modify setup.py to install our own scripts run via the exe launcher, which don't rely on entry points and pkg_resources. FYI the distlib executable launchers (not based on setuptools, but work the same way) and the distlib script generation approach does not require distlib to run. You might be able to make use of that in some way :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
