Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 7:31 PM, Jason R. Coombs wrote: >> -Original Message- >> From: Nick Coghlan [mailto:ncogh...@gmail.com] >> Sent: Sunday, 11 August, 2013 17:14 >> >> We actually have a proposal on import-sig to allow module specific import >> path manipulation (including the ability to say "don't import this module >> from this directory, even though it looks like it is here"). I'd favour that >> mechanism over a new "not importable" file extension. > > I don't believe this mechanism would suffice. My previous example was > over-simplified to the general problem, which is that any script could > potentially be imported as a module of the same name. So if I were to launch > easy_install.py, it would set sys.path[0] to Scripts\ and if it were then to > import cython (which it does), it would import Scripts/cython.py as cython, > unless there were some way to globally declare all installed scripts somewhere > so they're excluded from import. Indeed. It really *does* need to be a "don't import this" extension, though it doesn't much matter what that extension is. Except on Windows, of course, where it has to be something associated with Python that also still works as a console app and is listed in PATHEXT. (As you surmised earlier, my choice of '-script.py' was indeed chosen to prevent accidental importing, as the '-' ensures it's not a valid module name.) >> If that doesn't make it into 3.4, the proposed zipapp extensions would also >> serve a similar purpose, with some straightforward sys.path manipulation in >> __main__.py (as PJE pointed out). > > Regardless what solution might be made available for Python 3.4, I'd prefer to > work toward a solution that leverages pylauncher under older Pythons. After > all, one of the huge benefits of pylauncher is that it supports multiple > Pythons. If zipapp is the preferred mechanism for that, then so be it. For 2.6+, zipapps would work as long as pylauncher supported them and put the requisite extensions in PATHEXT. > This approach also means that the script generation is not congruent with that > on Unix systems. Using a zipapp means that the whole script generation needs > to be special-cased for Windows. One of great benefits of using a simple > script was that the code becomes largely unified (only requiring appending of > an extension when on Windows). That is, unless zipapps can be made executable > on Unix as well. They're already executable on Unix (as of 2.6+), as they contain a #! line. And they don't need a special extension; on both Unix and Windows, Python detects zipapps by inspecting the tail signature, not by the extension. (Of course, you could just continue using the existing wrapper mechanism on Unix, which has the advantage of added transparency, at the cost of having two code paths.) > Given these obstacles, do you still feel that zipapp is the best approach? Long-term, yes. I would slightly prefer the "this is a script" extension, though, as it has the extra transparency benefit. (Still, nobody should be editing an installed script anyway.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
> -Original Message- > From: Nick Coghlan [mailto:ncogh...@gmail.com] > Sent: Sunday, 11 August, 2013 17:14 > > We actually have a proposal on import-sig to allow module specific import > path manipulation (including the ability to say "don't import this module > from this directory, even though it looks like it is here"). I'd favour that > mechanism over a new "not importable" file extension. I don't believe this mechanism would suffice. My previous example was over-simplified to the general problem, which is that any script could potentially be imported as a module of the same name. So if I were to launch easy_install.py, it would set sys.path[0] to Scripts\ and if it were then to import cython (which it does), it would import Scripts/cython.py as cython, unless there were some way to globally declare all installed scripts somewhere so they're excluded from import. > If that doesn't make it into 3.4, the proposed zipapp extensions would also > serve a similar purpose, with some straightforward sys.path manipulation in > __main__.py (as PJE pointed out). Regardless what solution might be made available for Python 3.4, I'd prefer to work toward a solution that leverages pylauncher under older Pythons. After all, one of the huge benefits of pylauncher is that it supports multiple Pythons. If zipapp is the preferred mechanism for that, then so be it. I do agree that we should devise a best approach within the context of Python 3.4, and consider the backward-compatibility implications separately. My feeling is that zipapp is somewhat too convoluted in that it alters the sys.path, but then has to alter it back to simulate not being a zipapp. It's also a file within a file, meaning it can't be readily edited with a text editor, but requires a routine even just to inspect it. I guess that's more transparent than an executable, though. This approach also means that the script generation is not congruent with that on Unix systems. Using a zipapp means that the whole script generation needs to be special-cased for Windows. One of great benefits of using a simple script was that the code becomes largely unified (only requiring appending of an extension when on Windows). That is, unless zipapps can be made executable on Unix as well. Given these obstacles, do you still feel that zipapp is the best approach? smime.p7s Description: S/MIME cryptographic signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
We actually have a proposal on import-sig to allow module specific import path manipulation (including the ability to say "don't import this module from this directory, even though it looks like it is here"). I'd favour that mechanism over a new "not importable" file extension. If that doesn't make it into 3.4, the proposed zipapp extensions would also serve a similar purpose, with some straightforward sys.path manipulation in __main__.py (as PJE pointed out). Regards, Nick. P.S. Has anyone heard from Daniel lately, or know if he's away? I pinged him about getting the zipapp utility module PEP moving again a couple of weeks ago and haven't heard anything back. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 1:58 PM, Jason R. Coombs wrote: > This sounds like a suitable idea, but as you mention in a subsequent > message, this format has issues with sys.path assumptions as well. Meh. It's basically, a one-line fix in the __main__.py, i.e.: import sys,os.path; sys.path[0] = os.path.dirname(sys.path[0]) Not exactly rocket science. ;-) > In this case, I'm inclined to suggest yet another option (7) - create another > extension to specifically represent executable but not importable scripts, > perhaps .pys/.pysw (or .pycs/.pygs to more closely match console script and > gui script). Probably .pys/.pyws or .pws would be needed, due to issues with some Windows shells using extensions longer than three characters. (This came up in PEP 441 discussions on Python-Dev.) > It sounds as if there is a fundamental need for Python to define an > extension that distinguishes a script from a module. Yep. > I am aware of the PATHEXT factor. I personally add .py and .pyw to the > PATHEXT (for all users) on my systems, so I was unsure if Python 3.3 did add > those or if pylauncher would add them (if installed separately). I was > _hoping_ that was the case, but it sounds like it is not. I did include in > the documentation notes about this requirement > (https://bitbucket.org/pypa/setuptools/src/1.0b1/docs/easy_install.txt#cl-98 > ). > > I do want to explore the possibility of setuptools facilitating this > configuration such that it's easy for a Windows user to enable these > settings even if Python does not. It would definitely make sense to have an installer that sets this up, but it would need to be a Windows installer, I think, not a Python program. That is, I don't think setuptools can really do anything about it directly. Personally, I'm not sure I see the point of pushing for early elimination of .exe's - they don't depend on the registry or environment variables or anything else, which makes them great for standalone applications, and they work across all Python versions. Meanwhile, the experimental nature of your change -- and its inability to be the default on versions below 3.4 -- means you're going to be maintaining two sets of code for a very long time. OTOH, implementing a way to deploy an app as a .pyz/.pwz file is a useful feature in its own right, so it might not be doubling up as much. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
> -Original Message- > From: PJ Eby [mailto:p...@telecommunity.com] > Sent: Sunday, 11 August, 2013 12:17 > > > Here's another problem with #1: you will break single-directory standalone > portable app installs, where you use "easy_install -mad somedir" to install all > of an app's dependencies to a single directory that the app can then be run > from (assuming Python is available). > > In order to work around this issue, you'd need to hardcode sys.path entries > for the dependencies, or do something else more complicated in order to > ensure that dependency resolution will pick up the adjacent distributions > before searching anything else on sys.path. > > > > Third, is it possible some users are depending on the presence of > > sys.path[0] > > Absolutely. It's a documented feature of Python that the script directory is > always first on sys.path, so that you can provide modules and packages > adjacent to it. That's how portable app installs work with easy_install. All good points to be considered. Thanks. > May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non- > console apps) as a zipped Python application. .pyz files aren't importable, > but *are* executable. That's basically all that's needed to prevent importing > -- a file extension that's launchable but not importable. This sounds like a suitable idea, but as you mention in a subsequent message, this format has issues with sys.path assumptions as well. In this case, I'm inclined to suggest yet another option (7) - create another extension to specifically represent executable but not importable scripts, perhaps .pys/.pysw (or .pycs/.pygs to more closely match console script and gui script). It sounds as if there is a fundamental need for Python to define an extension that distinguishes a script from a module. > (There's also an option 6, which is to use import system hooks to prevent the > script modules from being found in the sys.path[0] entry, but that's rather > hairier.) Agreed, this sounds like it has its own subtle challenges. > Using option 5 means the feature can only work with versions of Python on > Windows that install the necessary PATHEXT support to allow that extension > to work, but you're kind of limited to that anyway, because by default .py > files aren't findable via PATH on Windows. > > Your post doesn't make it clear whether you're aware of that, btw: > IIUC, on most Windows setups, executing a .py file via PATH doesn't work > unless you've set up PATHEXT to include .py. So your feature's going to break > until that's fixed, and AFAIK there is *no* Windows Python that fixes this, > with the possible exception of 3.4 alpha, possibly a future alpha that hasn't > been released yet, because last I saw on Python-Dev it was still being > discussed *how* to update PATHEXT safely from the installer. > > In short: dropping .exe wrappers is not supportable on *any* current version > of Python for Windows, in the sense that anybody who uses it will not yet be > able to execute the scripts if they are currently doing so via PATH (and > haven't manually fixed their PATHEXT). (This was one of the main reasons > for using .exe wrappers in the first > place.) > > The .pyz approach of course has the same drawback, but at least it should be > viable for future Python versions, and doesn't have the sys.path[0] problems. > I think you are going to have to keep .exe wrappers the default for all Python > versions < 3.4. I am aware of the PATHEXT factor. I personally add .py and .pyw to the PATHEXT (for all users) on my systems, so I was unsure if Python 3.3 did add those or if pylauncher would add them (if installed separately). I was _hoping_ that was the case, but it sounds like it is not. I did include in the documentation notes about this requirement (https://bitbucket.org/pypa/setuptools/src/1.0b1/docs/easy_install.txt#cl-98 ). I do want to explore the possibility of setuptools facilitating this configuration such that it's easy for a Windows user to enable these settings even if Python does not. The nice thing about (7) is it would define an extension that's specifically meant to be executable, so would be an obvious (and potentially less controversial) choice to include in PATHEXT. smime.p7s Description: S/MIME cryptographic signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 12:17 PM, PJ Eby wrote: > May I suggest an option 5 instead? Use the new .pyz (or .pyzw for > non-console apps) as a zipped Python application. .pyz files aren't > importable, but *are* executable. That's basically all that's needed > to prevent importing -- a file extension that's launchable but not > importable. (Details I forgot to mention: the script would be in __main__.py inside the zipped application file, and it would need to change sys.path[0], because sys.path[0] will be the .pyz file itself; it should replace it with the directory containing the .pyz file before doing anything else. That would be the correct way to simulate the existing .exe approach.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 10:38 AM, Jason R. Coombs wrote: > In Setuptools 1.0 (currently in beta), I've added an experimental, opt-in > feature to install pure Python launcher scripts on Windows instead of > installing a launcher executable for each script, with the intention that > these scripts will be launched by pylauncher or Python directly, eventually > obviating the need for a launcher executable in setuptools at all. > > This means that instead of installing, for example: > > Scripts\my-command.exe > Scripts\my-command-script.py > Scripts\my-command.exe.manifest > > Instead Setuptools just installs: > > Scripts\my-command.py > > This technique is much like the scripts that get installed to bin/ on Unix, > except that due to the nature of launching commands on Windows, the .py > extension is essentially required. > > One problem with this technique is that if the script is also a valid module > name, it can be imported, and because Python puts the script's directory at > the top of sys.path, it _will_ be imported if that name is imported. > > This happens, for example, after installing Cython. Cython provides a > command, 'cython', and a (extension) module called 'cython'. If one launches > cython using the script launcher, the 'cython' module will not be importable > (because "import cython" will import the launcher script). Presumably, this > is why '-script' was added to the launcher scripts in previous versions. > > This is a rather unfortunate situation, and I'd like to solicit comments for > a way to avoid this situation. I see a few options: > > 1. Have the setuptools-generated launcher scripts del sys.path[0] before > launching. > 2. Accept the implementation and file bugs with the offending projects like > Cython to have them either rename their script or rename their internal > module. > 3. Continue to generate the script names with '-script.py' appended, > requiring invocation to always include -script.py on Windows. > 4. Continue to generate executables, duplicating the effort of pylauncher, > and dealing with the maintenance burden of that functionality. > > I don't see (2), (3), or (4) as really viable, so my proposal is to move > forward with (1) if there aren't any better suggestions. > > If we move forward with (1), there are a few concerns that come to mind. Here's another problem with #1: you will break single-directory standalone portable app installs, where you use "easy_install -mad somedir" to install all of an app's dependencies to a single directory that the app can then be run from (assuming Python is available). In order to work around this issue, you'd need to hardcode sys.path entries for the dependencies, or do something else more complicated in order to ensure that dependency resolution will pick up the adjacent distributions before searching anything else on sys.path. > Third, is it possible some users are depending on the presence of > sys.path[0] Absolutely. It's a documented feature of Python that the script directory is always first on sys.path, so that you can provide modules and packages adjacent to it. That's how portable app installs work with easy_install. May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non-console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable. (There's also an option 6, which is to use import system hooks to prevent the script modules from being found in the sys.path[0] entry, but that's rather hairier.) Using option 5 means the feature can only work with versions of Python on Windows that install the necessary PATHEXT support to allow that extension to work, but you're kind of limited to that anyway, because by default .py files aren't findable via PATH on Windows. Your post doesn't make it clear whether you're aware of that, btw: IIUC, on most Windows setups, executing a .py file via PATH doesn't work unless you've set up PATHEXT to include .py. So your feature's going to break until that's fixed, and AFAIK there is *no* Windows Python that fixes this, with the possible exception of 3.4 alpha, possibly a future alpha that hasn't been released yet, because last I saw on Python-Dev it was still being discussed *how* to update PATHEXT safely from the installer. In short: dropping .exe wrappers is not supportable on *any* current version of Python for Windows, in the sense that anybody who uses it will not yet be able to execute the scripts if they are currently doing so via PATH (and haven't manually fixed their PATHEXT). (This was one of the main reasons for using .exe wrappers in the first place.) The .pyz approach of course has the same drawback, but at least it should be viable for future Python versions, and doesn't have the sys.path[0] problems. I think you are going to have to keep .exe wrappers the default for all Python versions < 3.4.
[Distutils] How to handle launcher script importability?
In Setuptools 1.0 (currently in beta), I've added an experimental, opt-in feature to install pure Python launcher scripts on Windows instead of installing a launcher executable for each script, with the intention that these scripts will be launched by pylauncher or Python directly, eventually obviating the need for a launcher executable in setuptools at all. This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Instead Setuptools just installs: Scripts\my-command.py This technique is much like the scripts that get installed to bin/ on Unix, except that due to the nature of launching commands on Windows, the .py extension is essentially required. One problem with this technique is that if the script is also a valid module name, it can be imported, and because Python puts the script's directory at the top of sys.path, it _will_ be imported if that name is imported. This happens, for example, after installing Cython. Cython provides a command, 'cython', and a (extension) module called 'cython'. If one launches cython using the script launcher, the 'cython' module will not be importable (because "import cython" will import the launcher script). Presumably, this is why '-script' was added to the launcher scripts in previous versions. This is a rather unfortunate situation, and I'd like to solicit comments for a way to avoid this situation. I see a few options: 1. Have the setuptools-generated launcher scripts del sys.path[0] before launching. 2. Accept the implementation and file bugs with the offending projects like Cython to have them either rename their script or rename their internal module. 3. Continue to generate the script names with '-script.py' appended, requiring invocation to always include -script.py on Windows. 4. Continue to generate executables, duplicating the effort of pylauncher, and dealing with the maintenance burden of that functionality. I don't see (2), (3), or (4) as really viable, so my proposal is to move forward with (1) if there aren't any better suggestions. If we move forward with (1), there are a few concerns that come to mind. First, this approach would not apply to package-supplied scripts. What should be done about those (if anything)? Second, this would apply to Unix and Windows scripts (I'd rather not make the distinction if possible) - are there any concerns about removing sys.path[0] in the launch scripts on Unix? Third, is it possible some users are depending on the presence of sys.path[0] or the assumed positions of other items in sys.path that should be protected by this change? Would it make sense to replace sys.path[0] with a non-existent directory instead? My instinct is that these concerns are not of sufficient importance to account for them, and that we should just implement (1) as simply as possible. Your comments and suggestions are appreciated. smime.p7s Description: S/MIME cryptographic signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
