Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Thursday, December 05, 2013 06:37:39 PM Michael Jansen wrote: On Tuesday, December 03, 2013 12:33:22 PM Michael Jansen wrote: Changes to distutils itself are fairly pointless, since the earliest possible date for publication of any such changes is now as part of Python 3.5 in 2015. The sheer impracticality of that approach when plenty of people are still running Python 2.6 is why we're going through the somewhat painful exercise of decoupling the preferred build system from the standard library :) So, at this point, the appropriate place to make such a change is in setuptools: https://bitbucket.org/pypa/setuptools That will allow it to be made in a way that supports Python 2.6+, whereas a distutils change won't really help anyone. A point well made :) . Will do that. I made a proof of concept implementation (not ready to be merged) here No one interested in that? Got no answer from the maintainers too. Mike ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 7 December 2013 10:12, Michael Jansen i...@michael-jansen.biz wrote: I made a proof of concept implementation (not ready to be merged) here No one interested in that? Got no answer from the maintainers too. Personally, I'm against the idea on principle, because the bin/sbin distinction is not a cross-platform concept (Windows does not have it). For the core tools, I'd prefer *not* to include tools that encourage platform-specific behaviour (I've seen too many projects in the past which would work perfectly on Windows except for something like this). As a 3rd-party setuptools extension providing utilities that make writing POSIX-specific packages easier, I see no problem with it, though. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Saturday, December 07, 2013 10:56:16 AM Paul Moore wrote: On 7 December 2013 10:12, Michael Jansen i...@michael-jansen.biz wrote: I made a proof of concept implementation (not ready to be merged) here No one interested in that? Got no answer from the maintainers too. Personally, I'm against the idea on principle, because the bin/sbin distinction is not a cross-platform concept (Windows does not have it). For the core tools, I'd prefer *not* to include tools that encourage platform-specific behaviour (I've seen too many projects in the past which would work perfectly on Windows except for something like this). As a 3rd-party setuptools extension providing utilities that make writing POSIX-specific packages easier, I see no problem with it, though. My next step would be to implement support for fhs in the way that you can specify it like that '$bindir/bin_script', '$sbindir/sbin_script', ['$sbindir', ['sbin/sbin_script']], ['$bindir', ['bin/bin_script2']], ['$whateverwwwisnamedinfhs/myprefix', ['bin/mymod.wsgi']] And then setting those variables in cross platform compatible ways. Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 7 December 2013 11:14, Michael Jansen i...@michael-jansen.biz wrote: And then setting those variables in cross platform compatible ways. Well, OK, but there may not *be* a cross platform compatible answer - we could set sbin to the same as bin on Windows, but www is probably inside the Apache installation (or somewhere arbitrary if the user has configured things to non-default) on Windows, so there's no way Python can know this. The whole point here is that FHS gives you a way of naming things consistently, on a platform where absolue paths are the norm. On Windows, *relative* paths are the norm (often relative to the application installation directory by default) and applications' files are isolated from each other much more strongly. FHS doesn't really make sense in such an environment. But as I say, I don't have a problem with what you're doing as a POSIX solution. I just don't think we want to get into the huge cross-platform filesystem layout issue here. Just accept that it's not Windows-compatible and move on :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
I really don't have all the answers here, but... On 7 December 2013 12:30, Michael Jansen i...@michael-jansen.biz wrote: So you consider anything that has parts that need to be installed into/under a web server as not windows compatible? Not at all. If you're talking about documents, what's to install? Just ask the user to drop them under the docroot. If they are something like an Apache DLL, just ask the user in the installer where Apache is located. Or something like that. There's just no answer that assumes everyone's installation is the same and you don't have to get information from the user. Anything that wants to install pdf/html documentation? Manual Pages? Why install them. Just put them somewhere for the user to read. Add a start menu item if you want them accessible. Anything that needs configuration files which usually end up under /etc Well, that's a leading question. There are no such configuration files on Windows because there's no /etc. But I know what you mean and the answer is put them next to the executable. Or in the registry if you believe in that. And generally give the user a configuration app rather than expect them to manually edit a file. Anything that requires a spool file (/var/spool normally)? I can't think of anything like that, you'd have to give an example. I'm not sure the concept exists on Windows... Of course, all of the above are *applications*, not Python modules, so I wouldn't expect you to distribute them with setuptools/PyPI anyway. But that's another aspect of the issue. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Saturday, December 07, 2013 01:16:27 PM Paul Moore wrote: I really don't have all the answers here, but... On 7 December 2013 12:30, Michael Jansen i...@michael-jansen.biz wrote: So you consider anything that has parts that need to be installed into/under a web server as not windows compatible? Not at all. If you're talking about documents, what's to install? Just ask the user to drop them under the docroot. If they are something like an Apache DLL, just ask the user in the installer where Apache is located. Or something like that. There's just no answer that assumes everyone's installation is the same and you don't have to get information from the user. If we add something like $wwwroot setup.py could notice this is used and then ask the user where to put it. pip/wheel too. Currently it all gets lumped somewhere under data_files and no one has any chance to automate this. On linux the default could be to check whatever is appropriate for the distribution (/srv/www or /var/www) and if those are not there bail out with the message you have to specify the correct path on the command line. That could be the default on windows. Anything that wants to install pdf/html documentation? Manual Pages? Why install them. Just put them somewhere for the user to read. Add a start menu item if you want them accessible. For me installing means putting them somewhere for the user to read in an automated, reliable and reproducable fashion from sources. So building them (with sphinx or whatever) and copying somewhere for the user to read is installing. Anything that needs configuration files which usually end up under /etc Well, that's a leading question. There are no such configuration files on Windows because there's no /etc. But I know what you mean and the answer is put them next to the executable. Or in the registry if you believe in that. And generally give the user a configuration app rather than expect them to manually edit a file. But when enhancing distutils with something like $etcdir we could support that easily with minimal work for the developer. On linux install into /etc, On windows install wherever it is ok to put those stuff (without asking the user) and then provide the script author with an easy way to find out where $etcdir is on the platform so he can find his stuff. Your point of view (i don't want to think about it. Solve it posix only) means people have to develop solutions for windows separately and on top of that each and every time manually (and most won't do the work) or the other way around. A windows package will not work on linux. Anything that requires a spool file (/var/spool normally)? I can't think of anything like that, you'd have to give an example. I'm not sure the concept exists on Windows... Thats just a file where one process pumps data in that another process reads out and handles at his leisure. Of course, all of the above are *applications*, not Python modules, so I wouldn't expect you to distribute them with setuptools/PyPI anyway. But that's another aspect of the issue. Yes it is. And i agree with you that distributing something as cobblerd with pip, pypi is not really a good idea. But setuptools is the ONLY build system that works with python code so i HAVE to use it to build and install cobblerd from sources. So it should gain support to support applications too. Not only libraries and simple scripts. Anyway. You are at least willing to discuss with me. So thanks. I guess i give up on improving setuptools and instead just implement enough to fix cobblerd. After that i guess i will take with me the hope that the next time i have contact with python packaging the situation improved but i had that hope ten years ago too :( . Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 7 December 2013 13:45, Michael Jansen i...@michael-jansen.biz wrote: Anyway. You are at least willing to discuss with me. So thanks. No problem! I hope I wasn't too discouraging - I get the impression that I was :-( My problem is that I don't know the answers on Windows, I only really know enough to know when something feels wrong. In this area, Unix and Windows really are very different, and it shows. Very few people are willing or able to put the effort into developing cross-platform solutions, and those that do are too busy doing it to worry about establishing general principles, etc. I guess i give up on improving setuptools and instead just implement enough to fix cobblerd. After that i guess i will take with me the hope that the next time i have contact with python packaging the situation improved but i had that hope ten years ago too :( . I can understand your situation, but please don't give up! The packaging community definitely needs people willing to help work on these issues. But it's much less about coding solutions, and more about negotiating compromises between people with very different viewpoints. To go back to your original issue (sbin vs bin) - from my point of view as a Windows developer, packages just export commands. Why would I choose to put them in the sbin list rather than the bin list? If that's not clear, then I'll possibly develop packages that don't work right on Unix. Let's thrash out the answer to that question first, and *then* worry about implementing it. Is the relevant point that bin is on PATH but sbin (typically) isn't? If both are on PATH, I don't see why you bother having two directories. So maybe the real idea here is to split entry points into commands that should be on PATH and commands that should be provided, but not on PATH by default. Or is it? My understanding of sbin on Unix is that it's somehow system utilities - but that's just a classification with no practical impact. (What makes formatting a disk a system utility? I used to do it all the time when floppy disks were common.) Or maybe sbin is only on PATH when you're root. How does that translate to typical Windows systems, where there is usually only one user, and he has access to everything? Maybe in that context, sbin should be for utilities that require elevation? Lots of questions, but no obvious answers... As I said, and I'll say again, you don't *have* to answer these questions. But not doing so is not I don't want to think about it, just solve POSIX. Rather it's let's not make everything wait for a perfect solution when POSIX people need something now. But equally, let's not stop working on the cross-platform solution because the POSIX people have something and no longer care - that's *why* the problem never gets solved properly. One other thing I should say here - you mention your project is cobblerd. The d on the end makes me thing it's a daemon. That's inherently POSIX only - on Windows this would be a service (which is a whole other beast). Actually looking at Cobbler on PyPI it says Cobbler is a Linux boot server... so Windows compatibility really isn't crucial in this particular instance, I guess :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
FWIW, tackling FHS compliance is one of the things I have in mind as being suitable for a metadata 2.0 extension. Because Mac OS X and Windows just bundle (almost) everything into the application directory, FHS concepts simply don't map properly, so the current best FHS compliant solution is to use the data scheme path, FHS subdirectories and flag the project as Linux specific in the docs (or just not publish it on PyPI). Alternatively, cross platform Python code sometimes just ignores FHS compliance and bundles everything in with the module code (using pkgutil.get_data to retrieve it at runtime). Getting such packages ready for distro inclusion then ends up being quite a pain :P Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Saturday, December 07, 2013 03:15:15 PM Paul Moore wrote: On 7 December 2013 13:45, Michael Jansen i...@michael-jansen.biz wrote: I can understand your situation, but please don't give up! The packaging community definitely needs people willing to help work on these issues. But it's much less about coding solutions, and more about negotiating compromises between people with very different viewpoints. And thats where i disagree. I have some experience in open source contributions (KDE, cobbler and my own). 1. Usable code always wins. Always. Ideas and Agreements solve nothing. 2. This is a already solved problem. Both autotools and even more so cmake make it possible to develop cross platform. Why python should be differently is something i won't understand. If you want to wait for a compromise and an agreement nothing will ever be implemented. To go back to your original issue (sbin vs bin) - from my point of view as a Windows developer, packages just export commands. Why would I choose to put them in the sbin list rather than the bin list? If that's not clear, then I'll possibly develop packages that don't work right on Unix. Let's thrash out the answer to that question first, and *then* worry about implementing it. Is the relevant point that bin is on PATH but sbin (typically) isn't? If both are on PATH, I don't see why you bother having two directories. So maybe the real idea here is to split entry points into commands that should be on PATH and commands that should be provided, but not on PATH by default. Or is it? My understanding of sbin on Unix is that it's somehow system utilities - but that's just a classification with no practical impact. (What makes formatting a disk a system utility? I used to do it all the time when floppy disks were common.) Or maybe sbin is only on PATH when you're root. How does that translate to typical Windows systems, where there is usually only one user, and he has access to everything? Maybe in that context, sbin should be for utilities that require elevation? Lots of questions, but no obvious answers... A windows developer should not care about the difference. Both end up in the same directory. A unix developer will either not care too (and be reminded by the distro) or has read http://www.pathname.com/fhs/pub/fhs-2.3.html#SBINSYSTEMBINARIES So the not caring unix/windows developer will one day get a bug report/patch from a unix user or distro packager if some script belongs to sbin. As I said, and I'll say again, you don't *have* to answer these questions. But not doing so is not I don't want to think about it, just solve POSIX. Rather it's let's not make everything wait for a perfect solution when POSIX people need something now. But equally, let's not stop working on the cross-platform solution because the POSIX people have something and no longer care - that's *why* the problem never gets solved properly. Thats why i prefer to improve step by step. The sbin/bin distinction can be solved in a platform neutral way so why not do it. One other thing I should say here - you mention your project is cobblerd. The d on the end makes me thing it's a daemon. That's inherently POSIX only - on Windows this would be a service (which is a whole other beast). Actually looking at Cobbler on PyPI it says Cobbler is a Linux boot server... so Windows compatibility really isn't crucial in this particular instance, I guess :-) It most likely will never run on windows but i have seen weirder things (productive clearcase running on windows xp *shudder*). But there is nothing that prevents it from ever running on windows. So if i solved the multi distro support there is nothing that prevents someone to make it running under windows. Why shouldn't you boot and provision your unix virtual machines with a server running on windows? But i guess its unlikely. And its not my project. I only contribute since about three weeks. My homepage says something about drive-by coding and thats what i do. I like the project and noticed it currently only runs on redhat and installed into /usr. There are people using it on sles (its a rather enterprisey thing) which use patches. So i solved all obstacles to get it running out of the box (with correct paths) on opensuse already and thought i could do another attempt to get those improvements that make sense into setuptools. But i think that won't happen. And it was not you. I like to discuss and i always applaud people that help other brainstorm. Anyway. I will solve my problems on top of setuptools for cobblerd only for now. Then i will think about what to do with the results. I need and partially have solution for: - configure some files (patch @@var@@) (implemented) - eg. patch VIRTUALENV into a apache config file for wsgi virtualenv support - patch some distro / platform specific paths in config files. - install scripts in locations != bin (sbin
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 7 December 2013 16:56, Michael Jansen i...@michael-jansen.biz wrote: A windows developer should not care about the difference. Both end up in the same directory. [...] will one day get a bug report/patch from a unix user OK, cool. That makes sense. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
Just a note here: the wxWidgets (and thus wxPython, natch) project has a wxStandardPaths object: http://docs.wxwidgets.org/trunk/classwx_standard_paths.html It provides a cross platform way to get, well, the standard paths an application might need: GetAppDocumentsDir () GetConfigDir () GetDataDir () GetDocumentsDir () GetExecutablePath () GetInstallPrefix () GetLocalDataDir () GetLocalizedResourcesDir () GetPluginsDir () GetResourcesDir () GetTempDir () GetUserConfigDir () GetUserDataDir () GetUserLocalDataDir () . These all do the right thing on the supported platforms -- something may be in the application install dir on Windows, int he app bundle on the MAc, and it /etc on Linux, for instance. granted, wx is an application framework, so it needs this stuff, but while pip is about installing python packages, there really is no clear line between at package with a script or two and an application, and even simple scripts may need a few of these things. I don't see why we couldn't include a set like this, and have platform-specific mappings. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Tuesday, December 03, 2013 12:33:22 PM Michael Jansen wrote: Changes to distutils itself are fairly pointless, since the earliest possible date for publication of any such changes is now as part of Python 3.5 in 2015. The sheer impracticality of that approach when plenty of people are still running Python 2.6 is why we're going through the somewhat painful exercise of decoupling the preferred build system from the standard library :) So, at this point, the appropriate place to make such a change is in setuptools: https://bitbucket.org/pypa/setuptools That will allow it to be made in a way that supports Python 2.6+, whereas a distutils change won't really help anyone. A point well made :) . Will do that. I made a proof of concept implementation (not ready to be merged) here https://bitbucket.org/jansenm/setuptools/commits/all I did fight a bit with mercurial (we had severe philosophical disagreements) so i hope it works. There is a new test that checks that everything works for the distutils syntax. and then one that tests the improvements. I implemented the flexible idea. scripts = [ 'bin/bin_script', 'sbin/sbin_script', ['sbin', ['sbin/sbin_script']], ['bin', ['bin/bin_script2']], ['srv/www/myprefix', ['bin/mymod.wsgi']] The test is run with $ python setup.py test --test-module setuptools.tests.test_install_scripts; and currently prints out the directory it used for testing and does not remove it so its possible to inspect the result. test_distutils_compatibility (...) ... ## /tmp/tmpPMKJNk I btw. noticed that numpy overloads build_scripts itself to handle functions instead of strings. Not sure if my patch will break that. Open: Where to put the scripts in step build_scripts? For now it does this for maximum backwards compatibility. /tmp/tmp5ov6qb/src/build/scripts-2.7 /tmp/tmp5ov6qb/src/build/scripts-2.7/bin_script /tmp/tmp5ov6qb/src/build/scripts-2.7/sbin_script /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7 /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/sbin /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/sbin/sbin_script /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/bin /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/bin/bin_script2 /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv/www /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv/www/myprefix /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv/www/myprefix/mymod.wsgi I had to use the install_data directory to install the scripts into with copy_tree in install_scripts because my preferred solution install_base is not rerooted by distutils (when --root is given to setup.py). There is currently no build-setuptools-scripts parameter for python setup.py build. We would have to add that. Same for build_scripts. it would be build-setuptools-dir there. Documentation is missing too. As i said only a proof of concept for now. Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
It would be fairly simple to propose adding the GNU coding standard directory variables: http://www.gnu.org/prep/standards/html_node/Directory-Variables.html . They could become valid names under the wheel .data/ directory. It probably would. But i am a fan of only adding whats now needed. So i think just adding support for prefix/sbin and prefix/libexec would be enough for the problem i have at hand NOW. I need support for prefix/etc to and the special handling for /usr/etc - /etc too for some other stuff but i would like to do it one thing after the other. Lets first find out if its desired and possible to add something to distuils. It still is the base of all other build systems? Right? After that perhaps adding those to setuptools scripts entry point should be the next step. THEN i start to think about what i need for other stuff :) . You would also want to figure out how to instruct the installer to generate scripts wrappers for these alternative destinations so e.g. the execute bit is set and the #! line points to the correct interpreter. Thats what my patch does. For prefix/sbin for now but support of the libexec directory is easy to add. But quite a bit of copy and paste programming. So another idea i am thinking about is doing it like this: scripts = [ bin/myscript3, # = backwardscompatible ( 'bin/',[ 'bin/myscript1', 'bin/myscript2' ] ), ( 'sbin/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), ( 'lib/myproject/libexec', [ 'libexec/script' ] ) ] So if its a string do it like always and if its a list/tuple do it like data_files does. In an ideal world one could even expand those gnu standard vars in there. Like that: scripts = [ bin/myscript3, # = backwardscompatibility ( '$bindir/',[ 'bin/myscript1', 'bin/myscript2' ] ), ( '$sbindir/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), ( '$libexecdir', [ 'libexec/script' ] ) ] It would be a good idea to consider the needs of Windows and virtualenv users who may need to be able to opt out of the install things anywhere on my system and full FHS compliance features. At least the need for Windows is taken care of by INSTALL_SCHEME in distutils.command.install . The sbin directory is added like that: if sys.version 2.2: dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts' else: dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts' dist_install.INSTALL_SCHEMES['unix_prefix']['sbin_scripts'] = '$base/sbin' dist_install.INSTALL_SCHEMES['unix_home']['sbin_scripts'] = '$base/sbin' dist_install.INSTALL_SCHEMES['unix_user']['sbin_scripts'] = '$userbase/sbin' dist_install.INSTALL_SCHEMES['nt_user']['sbin_scripts'] = '$userbase/Scripts' dist_install.INSTALL_SCHEMES['os2']['sbin_scripts'] = '$base/Scripts' dist_install.INSTALL_SCHEMES['os2_home']['sbin_scripts'] = '$userbase/sbin' As you can see for windows the sbin scripts get installed into Scripts too. Same for os2 unless the --home option is active. And virtualenv does not need any kind of special handling. It has to install the same just into a different prefix. Same should go for pyvenv ? . All those new directories just have to be added there with correct values for all schemes. Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 3 December 2013 08:41, Michael Jansen i...@michael-jansen.biz wrote: So another idea i am thinking about is doing it like this: scripts = [ bin/myscript3, # = backwardscompatible ( 'bin/', [ 'bin/myscript1', 'bin/myscript2' ] ), ( 'sbin/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), ( 'lib/myproject/libexec', [ 'libexec/script' ] ) ] The scripts argument is generally considered obsolete - tools like pip will not generate executable wrappers for scripts specified like this, which means that they will not work properly on Windows. You should really be using (in this case, extending) the setuptools console_scripts entry point. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 3 December 2013 09:38, Michael Jansen i...@michael-jansen.biz wrote: So who will decide if any change to distutils goes through? I have no idea about the python patch process. Any links? Patches can be posted on http://bugs.python.org. But generally, changes to distutils are hard to get through so be prepared for some pushback. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Tuesday, December 03, 2013 08:56:26 AM Paul Moore wrote: On 3 December 2013 08:41, Michael Jansen i...@michael-jansen.biz wrote: So another idea i am thinking about is doing it like this: scripts = [ bin/myscript3, # = backwardscompatible ( 'bin/', [ 'bin/myscript1', 'bin/myscript2' ] ), ( 'sbin/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), ( 'lib/myproject/libexec', [ 'libexec/script' ] ) ] The scripts argument is generally considered obsolete - tools like pip will not generate executable wrappers for scripts specified like this, which means that they will not work properly on Windows. You should really be using (in this case, extending) the setuptools console_scripts entry point. This should be the next step. And it will. The message that scripts from distutils/setuptools is deprecated seems to be a bit unknown out there. http://code.ohloh.net/search?s=setup%20scriptsp=2pp=0fl=Pythonmp=1ml=1me=1md= 1ff=1filterChecked=true So doing both should be the right thing to do. The fact that pip does not support it is imho pips problem. So who will decide if any change to distutils goes through? I have no idea about the python patch process. Any links? Mike -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Tuesday, December 03, 2013 09:37:03 AM Paul Moore wrote: On 3 December 2013 09:38, Michael Jansen i...@michael-jansen.biz wrote: So who will decide if any change to distutils goes through? I have no idea about the python patch process. Any links? Patches can be posted on http://bugs.python.org. But generally, changes to distutils are hard to get through so be prepared for some pushback. I kinda assumed. I can't believe no attempt was made to fix that stuff in the last 10 years or more :) . Found a very old bug. Closed. Perhaps a temporary solution could be to implement both (scripts, console_scripts) for setuptools and THEN try to upstream the parts that make sense for distutils too. Being able to show it works and is backwards compatible could help the case. I guess i will give that a try. -- Michael Jansen http://michael-jansen.biz ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
Am 03.12.2013 09:56, schrieb Paul Moore: The scripts argument is generally considered obsolete - tools like pip will not generate executable wrappers for scripts specified like this, which means that they will not work properly on Windows. The Python documentation does not mention this - neither 2.7 nor 3.x. You should really be using (in this case, extending) the setuptools console_scripts entry point. More and more I get the impression that distributing (writing the setup script and deciding on package formats) Python libraries is currently nearly impossible for the mortal user. distutils, setuptools, wheels... sigh. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 3 December 2013 19:37, Paul Moore p.f.mo...@gmail.com wrote: On 3 December 2013 09:38, Michael Jansen i...@michael-jansen.biz wrote: So who will decide if any change to distutils goes through? I have no idea about the python patch process. Any links? Patches can be posted on http://bugs.python.org. But generally, changes to distutils are hard to get through so be prepared for some pushback. Changes to distutils itself are fairly pointless, since the earliest possible date for publication of any such changes is now as part of Python 3.5 in 2015. The sheer impracticality of that approach when plenty of people are still running Python 2.6 is why we're going through the somewhat painful exercise of decoupling the preferred build system from the standard library :) So, at this point, the appropriate place to make such a change is in setuptools: https://bitbucket.org/pypa/setuptools That will allow it to be made in a way that supports Python 2.6+, whereas a distutils change won't really help anyone. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 3 December 2013 11:27, Thomas Heller thel...@ctypes.org wrote: Am 03.12.2013 09:56, schrieb Paul Moore: The scripts argument is generally considered obsolete - tools like pip will not generate executable wrappers for scripts specified like this, which means that they will not work properly on Windows. The Python documentation does not mention this - neither 2.7 nor 3.x. Fair point. It's more that scripts offers you absolutely no help in writing cross-platform scripts, or wrapping scripts in exes (which, although annoying, is necessary on Windows to avoid subtle and frustrating issues), or otherwise doing anything beyond dumping a few files somewhere that will end up on PATH. The entry point mechanism introduced in setuptools, and extended/enhanced as part of the pip/wheel work, offers a much more robust and flexible solution. You should really be using (in this case, extending) the setuptools console_scripts entry point. More and more I get the impression that distributing (writing the setup script and deciding on package formats) Python libraries is currently nearly impossible for the mortal user. distutils, setuptools, wheels... sigh. I feel your pain. There is work ongoing to write a packaging user guide which documents best practice and sticks to providing a one way to do things approach. But it's very much a work in progress at this point, and things are changing rapidly right now, so it's some way away from being useful yet (don't look at the section on packaging your project unless you are fond of reading sections that say FIXME - or you want to help write those sections :-)) Even as someone directly involved in pip/wheel development, I struggle every time I need to write a setup.py. All I can say is that we're trying to cut through the confusion as fast as we can... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On 3 December 2013 21:42, Paul Moore p.f.mo...@gmail.com wrote: On 3 December 2013 11:27, Thomas Heller thel...@ctypes.org wrote: More and more I get the impression that distributing (writing the setup script and deciding on package formats) Python libraries is currently nearly impossible for the mortal user. distutils, setuptools, wheels... sigh. I feel your pain. There is work ongoing to write a packaging user guide which documents best practice and sticks to providing a one way to do things approach. But it's very much a work in progress at this point, and things are changing rapidly right now, so it's some way away from being useful yet (don't look at the section on packaging your project unless you are fond of reading sections that say FIXME - or you want to help write those sections :-)) Even as someone directly involved in pip/wheel development, I struggle every time I need to write a setup.py. All I can say is that we're trying to cut through the confusion as fast as we can... Aye, we're suffering from the fact that: - the distutils docs haven't been substantially updated in years (remember folks, you need to make sure to build your CPython extensions with Visual C++ 7.1!) - the setuptools docs are improving, but still have large elements of assuming you already know how to use distutils - the distribute and distutils2 docs assumed a future that never eventuated We've also been focusing our efforts on getting the installer side of things sorted out moreso than the build side, and PEP 453 is really the culmination of that effort (mostly implemented now, as there's just a few more issues for pip 1.5 final, along with the docs update to go on the CPython side. See http://bugs.python.org/issue19347 for details, since I can't apply the PEP453 tag on the pip side of things). Metadata 2.0 is then about clearly defining the full data model we want to support for distribution, and I think that will help us better document the build side of things. Getting the packaging user guide up to an at least somewhat usable state is highly desirable though, since that's what I plan to reference from the Python 3.4 docs. Even if it's incomplete, it would be difficult for it to be more misleading than the current distutils docs that are shipping with CPython :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
Am 03.12.2013 09:56, schrieb Paul Moore: The scripts argument is generally considered obsolete - tools like pip will not generate executable wrappers for scripts specified like this, which means that they will not work properly on Windows. Maybe pip (or bdist_wheel ?) should emit a Warning or an error, if it finds a setup-script with scripts argument? Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Install a script to prefix/sbin instead of prefix/bin
Hi
I am currently working on the cobbler (http://cobblerd.org) setup.py and trying
to improve it. It
currently is only capable of installing to /usr for several reasons. Since i
would like to have
virtualenv support when running it i am trying to change that. While doing that
i managed to meet
one of the standard problems with distutils/setuptools - a search returns
results of 2004 or older -
which still seems to be unsolved.
I would be willing to change that but my recent research leaves me a bit
confused with all that
setuptools is not dead, distlibs, tuf and wheels stuff.
So here is the question. If i am willing to do the work is it possible to add
something like
sbin_scripts (or admin_scripts?) to distutils?
I locally solved the problem like that http://pastebin.kde.org/pqrwrud1p (or
attached).
The implementation expands INSTALL_SCHEMA and reuses install_scripts to install
those
sbin_scripts too. That could be debatable. Perhaps a dedicated
build|install_xxx_scripts command.
I want to add that fhs also has a libexec folder (usually
/usr/lib/project/libexec these days)
destined for scripts that are supposed to be only called by other
scripts/programs not manually.
And cobbler for installs python scripts for modwsgi into /srv/www/cobbler/ . So
i guess just adding
sbin_scripts is not really the solution. Perhaps something more flexible is
needed.
Or should i wait for nextgen distlib based swiss knife python build tool and
for now keep my local
additions (there are more).
--
Michael Jansen
http://michael-jansen.biz
from distutils.command import install as dist_install
from distutils.command.build import build as _build
from distutils.command.build_scripts import build_scripts as _build_scripts
from distutils.command.build_scripts import convert_path, newer, first_line_re
from setuptools.command.install import install as _install
from setuptools.command.install_scripts import install_scripts as
_install_scripts
from setuptools import setup as _setup
from setuptools.dist import Distribution as _Distribution
from stat import ST_MODE
import sys
import os
from distutils import log
if sys.version 2.2:
dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts'
else:
dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts'
dist_install.INSTALL_SCHEMES['unix_prefix']['sbin_scripts'] = '$base/sbin'
dist_install.INSTALL_SCHEMES['unix_home']['sbin_scripts'] = '$base/sbin'
dist_install.INSTALL_SCHEMES['unix_user']['sbin_scripts'] = '$userbase/sbin'
dist_install.INSTALL_SCHEMES['nt_user']['sbin_scripts'] = '$userbase/Scripts'
dist_install.INSTALL_SCHEMES['os2']['sbin_scripts'] = '$base/Scripts'
dist_install.INSTALL_SCHEMES['os2_home']['sbin_scripts'] = '$userbase/sbin'
# The keys to an installation scheme; if any new types of files are to be
# installed, be sure to add an entry to every installation scheme above,
# and to SCHEME_KEYS here.
dist_install.SCHEME_KEYS = ('purelib', 'platlib', 'headers', 'scripts', 'data')
+ ('sbin_scripts',)
class install(_install):
Enhance install target aimed for inclusion upstream.
user_options = _install.user_options + [
('install-sbin-scripts=', None, installation directory for sbin
scripts)
]
def initialize_options(self):
_install.initialize_options(self)
self.install_sbin_scripts = None
def finalize_options(self):
_install.finalize_options(self)
if self.root is not None:
self.change_roots('sbin_scripts')
def finalize_unix(self):
_install.finalize_unix(self)
if self.install_base is not None or self.install_platbase is not None:
if self.install_sbin_scripts is None:
raise DistutilsOptionError, \
(install-base or install-platbase supplied, but
installation scheme is incomplete)
return
def finalize_other(self):
_install.finalize_other(self)
# Nothing else to do here
def expand_dirs(self):
_install.expand_dirs(self)
self._expand_attrs(['install_sbin_scripts'])
# :TODO:
# In run() add it to rejectdirs
class install_scripts(_install_scripts):
user_options = _install_scripts.user_options + [
('install-sbin-dir=', 'd', directory to install sbin scripts to)
]
def initialize_options(self):
_install_scripts.initialize_options(self)
self.install_sbin_dir = None
self.build_sbin_dir = None
def finalize_options (self):
_install_scripts.finalize_options(self)
self.set_undefined_options(
'build',
('build_sbin_scripts', 'build_sbin_dir'))
self.set_undefined_options(
'install',
('install_sbin_scripts', 'install_sbin_dir')
)
def run(self):
_install_scripts.run(self)
print self.build_sbin_dir
print self.install_sbin_dir
self.outfiles =
Re: [Distutils] Install a script to prefix/sbin instead of prefix/bin
On Mon, Dec 2, 2013 at 9:14 AM, Michael Jansen i...@michael-jansen.biz wrote: Hi I am currently working on the cobbler (http://cobblerd.org) setup.py and trying to improve it. It currently is only capable of installing to /usr for several reasons. Since i would like to have virtualenv support when running it i am trying to change that. While doing that i managed to meet one of the standard problems with distutils/setuptools - a search returns results of 2004 or older - which still seems to be unsolved. I would be willing to change that but my recent research leaves me a bit confused with all that setuptools is not dead, distlibs, tuf and wheels stuff. So here is the question. If i am willing to do the work is it possible to add something like sbin_scripts (or admin_scripts?) to distutils? I locally solved the problem like that http://pastebin.kde.org/pqrwrud1p (or attached). The implementation expands INSTALL_SCHEMA and reuses install_scripts to install those sbin_scripts too. That could be debatable. Perhaps a dedicated build|install_xxx_scripts command. I want to add that fhs also has a libexec folder (usually /usr/lib/project/libexec these days) destined for scripts that are supposed to be only called by other scripts/programs not manually. And cobbler for installs python scripts for modwsgi into /srv/www/cobbler/ . So i guess just adding sbin_scripts is not really the solution. Perhaps something more flexible is needed. Or should i wait for nextgen distlib based swiss knife python build tool and for now keep my local additions (there are more). -- Michael Jansen http://michael-jansen.biz It would be fairly simple to propose adding the GNU coding standard directory variables: http://www.gnu.org/prep/standards/html_node/Directory-Variables.html . They could become valid names under the wheel .data/ directory. You would also want to figure out how to instruct the installer to generate scripts wrappers for these alternative destinations so e.g. the execute bit is set and the #! line points to the correct interpreter. It would be a good idea to consider the needs of Windows and virtualenv users who may need to be able to opt out of the install things anywhere on my system and full FHS compliance features. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
