Re: [Distutils] Platform specific destinations in wheel files?
On 28 December 2013 06:17, Nick Coghlan ncogh...@gmail.com wrote: The other main thing to look at in terms of current state of the art is npm, for both structure and hooks. Another place to look would be existing usage in distributions on PyPI. I know that Vinay has done some work on extracting build-time data from PyPI (the red-dove metadata repo) - I don't know if he has anything like this available. It's probably too much work to get if nobody has already done it as a side effect of something else, though... Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 28 December 2013 21:36, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: I don't have anything actually written, though all the information is available in the JSON for individual releases under the source/data-files key. I will look at writing a simple scanner which looks at all the unique directories declared by PyPI dists. This should be faster when run locally on the server than a script which fetches the JSON via HTTP. Give me a bit of time, and I'll post what I find. Data driven design? What madness is this? :) Good idea Paul, and thanks for looking into it Vinay. Cheers, Nick. P.S. That reminds me, I should redo the PyPI version compatibility numbers for the latest PEP 440 draft - making pytz relatively easy to convert was the main gain, but I suspect the overall compatibility percentage will improve a little bit regardless. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 27 December 2013 12:34, Marcus Smith qwc...@gmail.com wrote: data: directory for data files. This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the name.data sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new name.app subdirectory in parallel. It's important to point out that the distutils/setuptools implementation of installing setup data_files offers a loophole now to achieve the app notion (at least when installing from sdist). When data_files paths are absolute, it honors them as absolute, allowing you to install anything/anywhere, if you have privileges (otoh when paths are relative, they go into the sysconfig data path as you would expect) as noted, wheel installs have no such mechanism, because there's just *.data and it maps to sysconfig data in the pip wheel installer. Also note that bdist_wheel actually does the wrong thing now with absolute data_files paths by letting them end up relative to site-packages https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_files) but I agree we need something more than data for platform-specific data files. Yeah, I don't think it's an urgent problem, but I believe it's one we will want to solve eventually. If we do another packaging BoF in Montreal (and hopefully we'll have the numbers for it to make sense to do so), that could be a good opportunity to bounce a few ideas around without spamming the list with more incomplete designs like my initial attempt. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On Tue, Dec 24, 2013 at 2:28 PM, Nick Coghlan ncogh...@gmail.com wrote: But that concept doesn't work on all platforms, so we should be careful about isolating it. Encapsulating that assumption is why I think the gnu nesting is justified. There are layout expectations inherent in the autoconf directory design that just don't make sense on Windows, so any package expecting them is going to be a bit quirky if installed on Windows. I'm confused now as to what has been proposed or being discussed, or... I _thought_ that this thread started with a proposal that package authors would do something like specifying a file hierarchy for the stuff they are delivering: /bin/some_scripts /share/some_data /man/some_docs then at installation time, the python distro would decide where to copy all that. But this would beb worthless on non-gnu systems and would require: 1) package writer to write three or four versions 2) wheels to be platform-dependent unnecessarily. So my suggestion was to define the various locations where stuff may need to be installed at a higher level: place_to_put_top_level_scripts place_to_put_documentation place_to_app_data place_to_put_user_configuration_files place_to_put_user_system_configuration_files . Then the python distro would map these to actual paths at install time: gnu systems would map the gnu locations, Windows to Windows-appropriate locations, OS-X to OS-X locations, etc This wold also allow python distros like Anaconda or macports python, or ??? to do their own thing, which may not be putting everything in /usr/local, or ... That may be what you had in mind, but I got confused. -Chris Cheers, Nick. -Chris It would be really easy to try adding these to the dict that wheel installers use to relocate stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 28 December 2013 06:02, Chris Barker chris.bar...@noaa.gov wrote: On Tue, Dec 24, 2013 at 2:28 PM, Nick Coghlan ncogh...@gmail.com wrote: But that concept doesn't work on all platforms, so we should be careful about isolating it. Encapsulating that assumption is why I think the gnu nesting is justified. There are layout expectations inherent in the autoconf directory design that just don't make sense on Windows, so any package expecting them is going to be a bit quirky if installed on Windows. I'm confused now as to what has been proposed or being discussed, or... I _thought_ that this thread started with a proposal that package authors would do something like specifying a file hierarchy for the stuff they are delivering: /bin/some_scripts /share/some_data /man/some_docs then at installation time, the python distro would decide where to copy all that. But this would beb worthless on non-gnu systems and would require: 1) package writer to write three or four versions 2) wheels to be platform-dependent unnecessarily. So my suggestion was to define the various locations where stuff may need to be installed at a higher level: place_to_put_top_level_scripts place_to_put_documentation place_to_app_data place_to_put_user_configuration_files place_to_put_user_system_configuration_files . Then the python distro would map these to actual paths at install time: gnu systems would map the gnu locations, Windows to Windows-appropriate locations, OS-X to OS-X locations, etc This wold also allow python distros like Anaconda or macports python, or ??? to do their own thing, which may not be putting everything in /usr/local, or ... That may be what you had in mind, but I got confused. There's no concrete proposal on the table at this point in time - just an acknowledgement that this is a use case the wheel format doesn't yet handle (sdists can handle it due to the way that setuptools handles the data_files option). However, any concrete proposal will likely use the GNU structure as the basis for kinds of files rather than inventing a new Python-specific naming scheme (or make an extremely compelling case for inventing our own custom scheme), and on Windows and Mac OS X, many of those will likely just map to dumping things in the app directory (or the application data directory in the case of Windows). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 28 December 2013 16:00, Nick Coghlan ncogh...@gmail.com wrote: On 28 December 2013 06:02, Chris Barker chris.bar...@noaa.gov wrote: Then the python distro would map these to actual paths at install time: gnu systems would map the gnu locations, Windows to Windows-appropriate locations, OS-X to OS-X locations, etc This wold also allow python distros like Anaconda or macports python, or ??? to do their own thing, which may not be putting everything in /usr/local, or ... That may be what you had in mind, but I got confused. There's no concrete proposal on the table at this point in time - just an acknowledgement that this is a use case the wheel format doesn't yet handle (sdists can handle it due to the way that setuptools handles the data_files option). However, any concrete proposal will likely use the GNU structure as the basis for kinds of files rather than inventing a new Python-specific naming scheme (or make an extremely compelling case for inventing our own custom scheme), and on Windows and Mac OS X, many of those will likely just map to dumping things in the app directory (or the application data directory in the case of Windows). The other main thing to look at in terms of current state of the art is npm, for both structure and hooks. https://npmjs.org/doc/files/npm-folders.html https://npmjs.org/doc/files/package.json.html https://npmjs.org/doc/misc/npm-scripts.html npm is basically a full-blown platform packaging system (covering config, runtime scripts, etc), but with a heavily POSIX biased bent (even moreso than the Python ecosystem). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
data: directory for data files. This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the name.data sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new name.app subdirectory in parallel. It's important to point out that the distutils/setuptools implementation of installing setup data_files offers a loophole now to achieve the app notion (at least when installing from sdist). When data_files paths are absolute, it honors them as absolute, allowing you to install anything/anywhere, if you have privileges (otoh when paths are relative, they go into the sysconfig data path as you would expect) as noted, wheel installs have no such mechanism, because there's just *.data and it maps to sysconfig data in the pip wheel installer. Also note that bdist_wheel actually does the wrong thing now with absolute data_files paths by letting them end up relative to site-packages https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_files ) but I agree we need something more than data for platform-specific data files. Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On Mon, Dec 23, 2013 at 1:43 PM, Daniel Holth Agreed. My biggest concern with this whole idea is that developers (typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific. exactly. However, with a little thought (possibly hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms. right, but setting the system up to allow a prefix and then hard-specify paths under that makes it impossible to do it cross-platform! Portable by default should be the principle. +1 ... and at least possible (without writing multiple platform-specific versions...)! ANother thought is: who should control where things are put? a) the package developer? b) the python system maintainer? I think clearly the answer is (b). i.e. we'll have reasonable defaults in the python.org builds, but if Anaconda or Canopy, or a BSD or a linux distro, or Raspberry Pi, or Micro. could all define their own paths for standard locations. I think this is much better than the package maintainers fixing that. The GNU autoconf paths are the obvious choice. Certainly a good palce to start. One issue -- that (and the FHS) is built around the idea of a prefix, where you can shift the whole pile to /usr or /usr/local, or /opt, or??? But that concept doesn't work on all platforms, so we should be careful about isolating it. -Chris It would be really easy to try adding these to the dict that wheel installers use to relocate stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 25 Dec 2013 04:14, Chris Barker chris.bar...@noaa.gov wrote: On Mon, Dec 23, 2013 at 1:43 PM, Daniel Holth Agreed. My biggest concern with this whole idea is that developers (typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific. exactly. However, with a little thought (possibly hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms. right, but setting the system up to allow a prefix and then hard-specify paths under that makes it impossible to do it cross-platform! Portable by default should be the principle. +1 ... and at least possible (without writing multiple platform-specific versions...)! ANother thought is: who should control where things are put? a) the package developer? b) the python system maintainer? I think clearly the answer is (b). i.e. we'll have reasonable defaults in the python.org builds, but if Anaconda or Canopy, or a BSD or a linux distro, or Raspberry Pi, or Micro. could all define their own paths for standard locations. I think this is much better than the package maintainers fixing that. The GNU autoconf paths are the obvious choice. Certainly a good palce to start. One issue -- that (and the FHS) is built around the idea of a prefix, where you can shift the whole pile to /usr or /usr/local, or /opt, or??? But that concept doesn't work on all platforms, so we should be careful about isolating it. Encapsulating that assumption is why I think the gnu nesting is justified. There are layout expectations inherent in the autoconf directory design that just don't make sense on Windows, so any package expecting them is going to be a bit quirky if installed on Windows. Cheers, Nick. -Chris It would be really easy to try adding these to the dict that wheel installers use to relocate stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On Sat, Dec 21, 2013 at 2:57 AM, Nick Coghlan ncogh...@gmail.com wrote: compliant daemon like cobblerd as a wheel file - using Python specific formats to define the layout of full applications, not just libraries. I'd generally been resisting the idea of supporting this (since I favour interoperating with system packaging tools where appropriate over attempting to replace them entirely), but in this case I'm starting to think it may be necessary to support these layouts in the next version of the wheel format in order to *also* support automated conversion of upstream projects to policy compliant system packages. hmm I tend to think, like you, that this isn't a problem wheel should solve, but can also see the advantages...for the moment though talking about how it would solve it may help clarify whether it should. adding a new name.app subdirectory in parallel. A wheel that had content in app would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific. is this necessary -- couldn't there be a way to provide the info in a cross-platform way, and have it mapped to the platform specific locations at install-time? All POSIX systems would at least support the fhs subdirectory. For a system install, this would map to /, for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to ~/.local. then you explicitly put in bin, sbin, share, whatever? This seems really klunky to me, and also forces platform dependence, and is fundamentally tied to how posix does things Maybe it's not possible, but I suggest that we could pre-define the locations that might be needed: executables (bin) system_executables (sbin) user_executables (./bin) documentation (doc) system_data_files (share ?) user_data_files (./share ) app_configuration (/etc/appname) user_app_configuration (./.app_name : ./Library ) This could end up being a pretty big list, but I think it could be finite. Then at install-time, the installer maps these to the appropriate place on the system. It's a little different application, but wxWidgets does this pretty successfully with wxStandardPaths. -Chris I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X. It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 23 December 2013 20:53, Chris Barker chris.bar...@noaa.gov wrote: then you explicitly put in bin, sbin, share, whatever? This seems really klunky to me, and also forces platform dependence, and is fundamentally tied to how posix does things Maybe it's not possible, but I suggest that we could pre-define the locations that might be needed: Agreed. My biggest concern with this whole idea is that developers (typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific. However, with a little thought (possibly hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms. This has been a very common issue historically, and I'm very reluctant to add anything that makes it easier to limit usability of packages across platforms. Portable by default should be the principle. But I see there is a need, and I understand that practicality beats purity. Let's just make sure that it's more effort to use the platform-specific approach than not to, so it doesn't become an attractive nuisance. BTW, I like your idea of defining generic locations - but in the past efforts to do this have been doomed to death by bikeshedding and platform wars. So while I agree in principle, I fear it may not happen in practice :-( Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On Mon, Dec 23, 2013 at 4:22 PM, Paul Moore p.f.mo...@gmail.com wrote: On 23 December 2013 20:53, Chris Barker chris.bar...@noaa.gov wrote: then you explicitly put in bin, sbin, share, whatever? This seems really klunky to me, and also forces platform dependence, and is fundamentally tied to how posix does things Maybe it's not possible, but I suggest that we could pre-define the locations that might be needed: Agreed. My biggest concern with this whole idea is that developers (typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific. However, with a little thought (possibly hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms. This has been a very common issue historically, and I'm very reluctant to add anything that makes it easier to limit usability of packages across platforms. Portable by default should be the principle. But I see there is a need, and I understand that practicality beats purity. Let's just make sure that it's more effort to use the platform-specific approach than not to, so it doesn't become an attractive nuisance. Attractive nuisance is the first phrase that comes to mind when I think of the FHS. BTW, I like your idea of defining generic locations - but in the past efforts to do this have been doomed to death by bikeshedding and platform wars. So while I agree in principle, I fear it may not happen in practice :-( Paul The GNU autoconf paths are the obvious choice. It would be really easy to try adding these to the dict that wheel installers use to relocate stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Platform specific destinations in wheel files?
With the Python 3.4 feature freeze behind us, I've started looking at doing a new update of the draft metadata 2.0 docs. Vaguely related to that are the recent discussions about being able to publish an FHS compliant daemon like cobblerd as a wheel file - using Python specific formats to define the layout of full applications, not just libraries. Currently, all of the paths defined in sysconfig schemes aim to be platform neutral. However, this means that some platform specific concepts can't easily be handled by the wheel format - the data scheme can sort of handle it, but it isn't particularly intuitive, and doesn't inherently reflect the platform dependent nature of the layout being defined. I'd generally been resisting the idea of supporting this (since I favour interoperating with system packaging tools where appropriate over attempting to replace them entirely), but in this case I'm starting to think it may be necessary to support these layouts in the next version of the wheel format in order to *also* support automated conversion of upstream projects to policy compliant system packages. As a reminder, here are the currently defined Python specific directories relevant to wheel files: platlib: directory for site-specific, platform-specific files. purelib: directory for site-specific, non-platform-specific files. include: directory for non-platform-specific header files. platinclude: directory for platform-specific header files. And these are the generic directories that aren't necessarily Python specific: scripts: directory for script files. data: directory for data files. This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the name.data sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new name.app subdirectory in parallel. A wheel that had content in app would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific. All POSIX systems would at least support the fhs subdirectory. For a system install, this would map to /, for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to ~/.local. I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X. It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 21 December 2013 10:57, Nick Coghlan ncogh...@gmail.com wrote: And these are the generic directories that aren't necessarily Python specific: scripts: directory for script files. data: directory for data files. It's worth noting that data, although in essence a platform neutral term, is in practice not very well defined. Projects as a result use it inconsistently, and usually based on their expectations from the platforms they are used to. One thing I have seen data being used for a lot is documentation - and not consistently - some projects use data/man, others use data/share/man, and others data/projectname. I'd suggest the following changes in this area: 1. Add a docs directory to hold documentation. We may need to have a debate on how this should be structured, and how to handle backward compatibility, but this is a genuine need that the current scheme does not cover. 2. Add some more explicit wording on how data should (and should not) be used. For instance, having FHS-style structures like share/man/man1 under data doesn't really help with cross-platform compatibility that much - it just provides somewhere to dump the data on Windows while still interoperating with standard tools on a POSIX system. This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the name.data sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new name.app subdirectory in parallel. A wheel that had content in app would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific. It might be worth having the possibility of optional parts of app. To use the docs example again, it would be easy to see projects putting Unix manpages in app, and so unintentinally making the distribution platform specific, when in practice it would be perfectly OK to simply omit the manpages on Windows. Other app files (for example config files in etc) *will* be essential for proper operation, of course. All POSIX systems would at least support the fhs subdirectory. For a system install, this would map to /, for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to ~/.local. I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X. I'm not sure Program Files is that useful - very few apps would install into *both* Program Files and an application area (site-packages). But Application Data might be useful. Another possibility would be the start menu - but that's not precisely a filesystem location, you'd need to put metadata into the Start Menu area of the wheel, and use the Windows API to create the start menu items at install time. It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific. Agreed in principle. But in practice, do you expect the wheel spec to define exactly how each scheme under app is mapped? And do you expect every tool to implement those features? That implies a lot of work getting consensus and a change to the wheel spec every time someone thinks of a new platform-specific area that's worth supporting. Maybe this is something that should be handled by the post-install hook support. The app area could be an area that is reserved in the wheel for platform-specific data as you say, but there is no specified behaviour for tools to implement. Rather: 1. Tools must make the app area accessible in post-install hooks (how is yet to be defined, simplest answer just put it in a temporary location on disk, and tell the hooks the location of that location) 2. Tools should provide an extensibility mechanism for installing system-wide processors for app data (so that we don't need every wheel to ship the FHS-handling post-install hooks) 3. Standard processing for particular app areas (for example, app/fhs) could be defined independently of the wheel spec. One of the motivations I have here is to keep the install behaviour from getting too complex. We need to remember that pip is not the only wheel installer (there's distlib and wheel itself). So I don't want to mandate too much complexity into the install phase (for example, script wrappers aren't handled in wheel install, and I'm not sure what distlib does - so there's implementation differences creeping in right now). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform specific destinations in wheel files?
On 21 December 2013 21:40, Paul Moore p.f.mo...@gmail.com wrote: On 21 December 2013 10:57, Nick Coghlan ncogh...@gmail.com wrote: And these are the generic directories that aren't necessarily Python specific: scripts: directory for script files. data: directory for data files. It's worth noting that data, although in essence a platform neutral term, is in practice not very well defined. Projects as a result use it inconsistently, and usually based on their expectations from the platforms they are used to. One thing I have seen data being used for a lot is documentation - and not consistently - some projects use data/man, others use data/share/man, and others data/projectname. I'd suggest the following changes in this area: 1. Add a docs directory to hold documentation. We may need to have a debate on how this should be structured, and how to handle backward compatibility, but this is a genuine need that the current scheme does not cover. Evolution of sysconfig as we go forward is indeed something for us to think about. It may be a case where we need to actually decouple the wheel spec from the implicit dependency on those definitions. The stdlib related paths don't make sense to use in a wheel file, so what probably makes sense is for us to make just purelib, platlib, include and platinclude depend on sysconfig (since they're installing to Python-specific locations and need the assistance from the standard library), while other paths like scripts and docs would be defined independently of sysconfig. 2. Add some more explicit wording on how data should (and should not) be used. For instance, having FHS-style structures like share/man/man1 under data doesn't really help with cross-platform compatibility that much - it just provides somewhere to dump the data on Windows while still interoperating with standard tools on a POSIX system. I'd be interesting in reading such guidance, because I really have no idea... (I'd never even noticed the option until Daniel pointed it out to me recently) This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the name.data sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new name.app subdirectory in parallel. A wheel that had content in app would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific. It might be worth having the possibility of optional parts of app. To use the docs example again, it would be easy to see projects putting Unix manpages in app, and so unintentinally making the distribution platform specific, when in practice it would be perfectly OK to simply omit the manpages on Windows. Other app files (for example config files in etc) *will* be essential for proper operation, of course. I think a separate docs path would be a better solution for that. There may be a couple of others we decide to support directly (like an admin_scripts path that maps to sbin under FHS and to the same place that scripts maps to on Windows). All POSIX systems would at least support the fhs subdirectory. For a system install, this would map to /, for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to ~/.local. I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X. I'm not sure Program Files is that useful - very few apps would install into *both* Program Files and an application area (site-packages). But Application Data might be useful. Another possibility would be the start menu - but that's not precisely a filesystem location, you'd need to put metadata into the Start Menu area of the wheel, and use the Windows API to create the start menu items at install time. It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific. Agreed in principle. But in practice, do you expect the wheel spec to define exactly how each scheme under app is mapped? And do you expect every tool to implement those features? That implies a lot of work getting consensus and a change to the wheel spec every time someone thinks of a new platform-specific area that's worth supporting. Maybe this is something that should be handled by the post-install hook support. The app area could be an area that is reserved in the wheel for platform-specific data as you say, but there is no specified behaviour for tools to implement. Rather: 1. Tools must make the app area accessible in post-install hooks (how is yet to be defined, simplest answer just put it in a
