Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. We've used eggs for that purpose for almost 10 years at Enthought, and we did not need that many extensions on top of the egg format after all. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). IMO, given that wheels do most of what's needed, it is worth supporting most simple usecases (compiled libraries required by well known extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to quite horrible custom hacks to support those cases. Hope that clarifies the intent, David I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to simply pip install pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda. BTW, while you can't generally install a conda package in/for another python, you can generally install a wheel in a conda pythonThere are a few issues with
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to simply pip install pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda. BTW, while you can't generally install a conda package in/for another python, you can generally install a wheel in a conda pythonThere are a few issues with pip/setuptools trying to resolve dependencies while not knowing about conda packages, but it does mostly work. Not sure that helped the discussion -- but I've been wrestling with this for a while, so thought I'd get my thoughts out there. I've always thought of wheel as solving only the Python-specific problem. Providing relocatable Python-specific packaging without trying to solve the intractable problem of non-Python dependencies. The strategy works best the more you are targeting Python as your platform and not a specific OS or distribution - sometimes it works well, other times not at all. Obviously if you need a specific build of PostgreSQL wheel isn't going to help you. With enough hacks you could make it work but are we ready to pip install kde? I don't think so. Personally I'm happy to let other tools solve the problem of C-level
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. We've used eggs for that purpose for almost 10 years at Enthought, and we did not need that many extensions on top of the egg format after all. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). IMO, given that wheels do most of what's needed, it is worth supporting most simple usecases (compiled libraries required by well known extensions). Right now, such packages (pyzmq, numpy, cryptography, lxml) resort to quite horrible custom hacks to support those cases. Hope that clarifies the intent, David Then it sounds like I should read about the Enthought egg extensions. It's something else than just defining a separate pypi name for just the libxml.so without the python bits? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 1:19 PM, David Cournapeau courn...@gmail.com wrote: This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows Are these inside or outside the python installation? I'm more than a bit wary of a wheel that would install stuff outside of the sandbox of the python install. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. hmm -- true. I guess where it seems to get more complicated is beyond the wheel (or conda, or...) package itself, to the dependency management, installation tools, etc. But perhaps you are suggesting that we can extend wheel to support a bt more stuff, and leave the rest of the system as separate problem? i.e. Canopy can have it's own find, install, manage-dependency tool, but that it can use the wheel format for the packages themselves? I don't see why not -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
I would advise against using or even reading about our egg extensions, as the implementation is full of legacy (we've been doing this many years :) ): http://enstaller.readthedocs.org/en/master/reference/egg_format.html This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows - a few extensions to the actual binary format (adding support for symlinks is the only one I can think of ATM). Everything else is legacy you really don't want to know (see here if you still want to http://enstaller.readthedocs.org/en/master/reference/egg_format.html) David On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth dho...@gmail.com wrote: On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. We've used eggs for that purpose for almost 10 years at Enthought, and we did not need that many extensions on top of the egg format
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 5:25 PM, Chris Barker chris.bar...@noaa.gov wrote: On Mon, Apr 13, 2015 at 1:19 PM, David Cournapeau courn...@gmail.com wrote: This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows Are these inside or outside the python installation? I'm more than a bit wary of a wheel that would install stuff outside of the sandbox of the python install. I would always install things relative to sys.prefix, for exactly the reasons you mention. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's needed. hmm -- true. I guess where it seems to get more complicated is beyond the wheel (or conda, or...) package itself, to the dependency management, installation tools, etc. But perhaps you are suggesting that we can extend wheel to support a bt more stuff, and leave the rest of the system as separate problem? i.e. Canopy can have it's own find, install, manage-dependency tool, but that it can use the wheel format for the packages themselves? Exactly ! David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
Seems like you could extend wheel to do that easily. On Apr 13, 2015 4:19 PM, David Cournapeau courn...@gmail.com wrote: I would advise against using or even reading about our egg extensions, as the implementation is full of legacy (we've been doing this many years :) ): http://enstaller.readthedocs.org/en/master/reference/egg_format.html This is what we use on top of setuptools egg: - ability to add dependencies which are not python packages (I think most of it is already handled in metadata 2.0/PEP 426, but I would have to re-read the PEP carefully). - ability to run post/pre install/remove scripts - support for all the of the autotools directories, with sensible mapping on windows - a few extensions to the actual binary format (adding support for symlinks is the only one I can think of ATM). Everything else is legacy you really don't want to know (see here if you still want to http://enstaller.readthedocs.org/en/master/reference/egg_format.html) David On Mon, Apr 13, 2015 at 3:55 PM, Daniel Holth dho...@gmail.com wrote: On Mon, Apr 13, 2015 at 3:46 PM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker chris.bar...@noaa.gov wrote: NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. The whole reason I started this discussion is to make sure wheel has a standard way to do what is needed for those usecases. conda, rpm, deb, or eggs as used in enthought are all essentially the same: an archive with a bunch of metadata. The real issue is standardising on the exact formats. As you noticed, there is not much missing in the wheel *spec* to get most of what's
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Apr 13, 2015, at 10:17 PM, Robert Collins robe...@robertcollins.net
wrote:
On 14 April 2015 at 09:35, David Cournapeau courn...@gmail.com wrote:
...
One of the earlier things mentioned here - {pre,post}{install,remove}
scripts - raises a red flag for me.
In Debian at least, the underlying system has the ability to run such
turing complete scripts, and they are a rich source of bugs - both
correctness and performance related.
Nowadays nearly all such scripts are machine generated from higher
level representations such as 'this should be the default command' or
'configure X if Y is installed', but because the plumbing is turing
complete, they all need to be executed, which slows down
install/upgrade paths, and any improvement to the tooling requires a
version bump on *all* the packages using it - because effectively the
package is itself a compiled artifact.
I'd really prefer it if we keep wheels 100% declarative, and instead
focus on defining appropriate metadata for the things you need to
accomplish {pre,post}{install,remove} of a package.
A possible way to implement {pre,post}{install,remove} scripts is to
instead turn them into extensions. One example is that Twisted uses
a setup.py hack to regenerate a cache file of all of the registered
plugins. This needs to happen at install time due to permission issues.
Currently you can’t get this speedup when installing something that
uses twisted plugins and installing via Wheel.
So a possible way for this to work is in a PEP 426 world, simply define
a twisted.plugins extension that says, in a declarative way, “hey when
you install this Wheel, if there’s a plugin that understands this extension
installed, let it do something before you actually move the files into
place”. This let’s Wheels themselves still be declarative and moves the
responsibility of implementing these bits into their own PyPI projects
that can be versioned and independently upgraded and such. We’d probably
need some method of marking an extension as “critical” (e.g. bail out and
don’t install this Wheel if you don’t have something that knows how to handle
it) and then non critical extensions just get ignored if we don’t know
how to handle it.
Popular extensions could possibly be added directly to pip at some point
if a lot of people are using them (or even moved from third party extension
to officially supported extension).
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 14 April 2015 at 09:35, David Cournapeau courn...@gmail.com wrote:
...
One of the earlier things mentioned here - {pre,post}{install,remove}
scripts - raises a red flag for me.
In Debian at least, the underlying system has the ability to run such
turing complete scripts, and they are a rich source of bugs - both
correctness and performance related.
Nowadays nearly all such scripts are machine generated from higher
level representations such as 'this should be the default command' or
'configure X if Y is installed', but because the plumbing is turing
complete, they all need to be executed, which slows down
install/upgrade paths, and any improvement to the tooling requires a
version bump on *all* the packages using it - because effectively the
package is itself a compiled artifact.
I'd really prefer it if we keep wheels 100% declarative, and instead
focus on defining appropriate metadata for the things you need to
accomplish {pre,post}{install,remove} of a package.
-Rob
--
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: help needed
Nick Coghlan ncogh...@gmail.com writes: On 11 Apr 2015 12:22, Alexander Walters tritium-l...@sdamon.com wrote: Is the package index really the best place to put this? This is a very social-networking feature for the authoritative repository of just about all the third party module, and it feels like either it could corrupt the 'sanctity' of the repository (in the absolute worst case) If you're concerned that this feature might weaken the comforting illusion that PyPI published software is contributed and maintained by faceless automatons rather than living, breathing human beings, then yes, encouraging folks to think more about where the software they use is coming from would be a large part of the point of adding such a feature. I can't speak for Alexander, but I'm also −1 to have this *on PyPI*. I'm all for such features existing. What is at issue is whether PyPI is the place to put them. We have been gradually improving the function of PyPI as an authoritative *index* of packages; that's possible because it is a repository of uncontroversial facts, not opinions (i.e. “what is the packaging metadata of this distribution”, “where is its documentation”, “where is its VCS”, etc.). I am not saying the PSF shouldn't do this, but is pypi REALLY the best part of python.org to put it? I personally believe so, yes - sustaining software over the long term is expensive in people's time, but it's often something we take for granted. The specific example Guido brought up in his keynote was the challenge of communicating a project's openness to Python 3 porting assistance. The people doing the work of maintaining PyPI have said many times in recent years that there just isn't enough person-power to add a whole bunch of features that have been requested. Why would we think moderating a social-networking rating, opinion, discussion, or other non-factual database is something reasonable to ask of the PyPI maintainers? Conversely, if we are under the impression that adding ratings, feedback, reviews, discussion, and other features to PyPI is *not* going to be a massive increase in workload for the maintainers, I think that's a foolish delusion which will be quite costly to the reputation PyPI has recently gained through hard effort to clarify its role. By all means, set up a well-maintained social ecosystem around Python packages. But not on PyPI itself: The Python Package Index is feasible in part because it has a clear and simple job, though, and that's not it. -- \“If you can't hear me sometimes, it's because I'm in | `\ parentheses.” —Steven Wright | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: help needed
On Apr 13, 2015, at 8:57 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: Nick Coghlan ncogh...@gmail.com writes: On 11 Apr 2015 12:22, Alexander Walters tritium-l...@sdamon.com wrote: Is the package index really the best place to put this? This is a very social-networking feature for the authoritative repository of just about all the third party module, and it feels like either it could corrupt the 'sanctity' of the repository (in the absolute worst case) If you're concerned that this feature might weaken the comforting illusion that PyPI published software is contributed and maintained by faceless automatons rather than living, breathing human beings, then yes, encouraging folks to think more about where the software they use is coming from would be a large part of the point of adding such a feature. I can't speak for Alexander, but I'm also −1 to have this *on PyPI*. I'm all for such features existing. What is at issue is whether PyPI is the place to put them. We have been gradually improving the function of PyPI as an authoritative *index* of packages; that's possible because it is a repository of uncontroversial facts, not opinions (i.e. “what is the packaging metadata of this distribution”, “where is its documentation”, “where is its VCS”, etc.). I am not saying the PSF shouldn't do this, but is pypi REALLY the best part of python.org to put it? I personally believe so, yes - sustaining software over the long term is expensive in people's time, but it's often something we take for granted. The specific example Guido brought up in his keynote was the challenge of communicating a project's openness to Python 3 porting assistance. The people doing the work of maintaining PyPI have said many times in recent years that there just isn't enough person-power to add a whole bunch of features that have been requested. Why would we think moderating a social-networking rating, opinion, discussion, or other non-factual database is something reasonable to ask of the PyPI maintainers? Conversely, if we are under the impression that adding ratings, feedback, reviews, discussion, and other features to PyPI is *not* going to be a massive increase in workload for the maintainers, I think that's a foolish delusion which will be quite costly to the reputation PyPI has recently gained through hard effort to clarify its role. By all means, set up a well-maintained social ecosystem around Python packages. But not on PyPI itself: The Python Package Index is feasible in part because it has a clear and simple job, though, and that's not it. -- \“If you can't hear me sometimes, it's because I'm in | `\ parentheses.” —Steven Wright | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig I don’t see any problem with the general idea of adding features to PyPI to enable package maintainers to find more help maintaining specific parts of their projects. I do have a problem with expecting the PyPI administrators to fill out or otherwise populate this information. Saying “Here’s a place you can donate to me” is still a fact, it’s just a more social fact than what we currently enable. I’m kind of down on the idea of linking to CVs or linkedin as part of the project metadata because that’s not project specific and is really more maintainer specific. I think that particular feature would be better suited to some sort of global “Python profile” that could then be linked to from PyPI instead of trying to bake it into PyPI itself. However things like “Looking for New Maintainers / Orphan a Project”, or some call to actions on “here are some issues that need fixed” or other things doesn’t seem unreasonable to me. Particularly the ability to orphan a project or look for new maintainers seems like a useful thing to me that really can’t live anywhere other than PyPI reasonably. The other items can live elsewhere if we wanted them to since they would be easy to add to the long_description of a project which would get added to the PyPI page but that has some drawbacks. For things like crowdfunding campaigns the long_description is set when you upload a release, however it’d be useful to have the campaigns update as the campaign progresses (or even ultimately be removed once the campaign has finished). I think an important part of this idea here is that this doesn’t enable anything that authors can’t already do, it just presents the information in a way that is easier for other tooling to take advantage of it as well as allow us to make more informed decisions about how to show it and when to show it without requiring authors to update the long_description of their projects. I think it will also be a strong signal that it’s OK
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 10:17 PM, Robert Collins robe...@robertcollins.net
wrote:
On 14 April 2015 at 09:35, David Cournapeau courn...@gmail.com wrote:
...
One of the earlier things mentioned here - {pre,post}{install,remove}
scripts - raises a red flag for me.
That's indeed a good a priori. I myself removed a lot of those scripts
because of the fragility. Anything that needs to run on a end-user machine
can fail, and writing idempotent scripts is hard.
Unfortunately, pure declarative does not really cut it if you want cross
platform support. Sure, you may be able to deal with menu entries,
environment variables, etc... in a cross-platform manner with a significant
effort, but what about COM registration ? pywin32 is one of the most used
package in the python ecosystem, and its post install script is not
trivial. Another difficult case if when a package needs some specific
configuration to run at all, and that configuration requires values known
at install time only (e.g. sys.prefix, as in the iris package).
I'd really prefer it if we keep wheels 100% declarative, and instead
focus on defining appropriate metadata for the things you need to
accomplish {pre,post}{install,remove} of a package.
What about a way for wheels to specify whether their
post,pre/install,remove scripts are declarative or not, with support for
the most common tasks, with an escape, but opt-in mechanism ? This way it
could be a matter of policy to refuse packages that require non-declarative
scripts.
David
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Make PEP 426 less boring
Hi,
somehow I feel bored if I read PEP 426.
https://www.python.org/dev/peps/pep-0426/
One concrete improvement would be to remove this paragraph:
{{{
The design draws on the Python community's 15 years of experience with distutils based software distribution, and
incorporates ideas and concepts from other distribution systems, including Python's setuptools, pip and other projects,
Ruby's gems, Perl's CPAN, Node.js's npm, PHP's composer and Linux packaging systems such as RPM and APT.
}}}
Because something like this was already saied some lines before
{{{
Metadata 2.0 represents a major upgrade to the Python packaging ecosystem, and attempts to incorporate experience
gained over the 15 years(!) since distutils was first added to the standard library. Some of that is just incorporating
existing practices from setuptools/pip/etc, some of it is copying from other distribution systems (like Linux distros or
other development language communities) and some of it is attempting to solve problems which haven't yet been well
solved by anyone (like supporting clean conversion of Python source packages to distro policy compliant source packages
for at least Debian and Fedora, and perhaps other platform specific distribution systems).
}}}
**And** I would move the historic background (the second of the above quotes)
at the end.
Meta: are you interested in feedback like this?
Regards,
Thomas Güttler
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft don...@stufft.io wrote: On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote: Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento ( https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. To my knowledge, (1) was purposely punted until a later revision of Wheel just to make it easier to land the “basic” wheel. Great. Was there any proposal made to support it at all ? Or should I just work from scratch there ? I think (2) is a reasonable thing as long as we can map it sanely on all platforms. Yes. We support all platforms at Enthought, and Windows is important for us ! I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python More generally, we could be modifying python in a way that is not forward compatible with upstream python: a binary that works on our python may not work on the python from python.org (though the opposite is true). It would be nice if one could make sure pip will not try to install those eggs when installed on top of a python that does not advertise itself as compatible. David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Beyond wheels 1.0: helping downstream, FHS and more
Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento ( https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. thanks, David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote: Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. To my knowledge, (1) was purposely punted until a later revision of Wheel just to make it easier to land the “basic” wheel. I think (2) is a reasonable thing as long as we can map it sanely on all platforms. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip/warehouse feature idea: help needed
+1 overall to Nick' suggestions. -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
NOTE: I don't work for any of the companies involved -- just a somewhat frustrated user... And someone that has been trying for years to make things easier for OS-X users. I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python But why not? -- at least for Anaconda, it's because those libraries likely have non-python dependencies, which are expected to be installed in a particular way. And really, this is not particular to Anaconda/Canopy at all. Python itself has no answer for this issue, and eggs and wheels don't help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to build a binary wheel with non-python dependencies (let's say something like libjpeg, for instance), you need to either: - assume that libjpeg is installed in a standard place -- really no solution at all (at least outside of linux) - statically link it - ship the dynamic lib with the package For the most part, the accepted solution for OS-X has been to statically link, but: - it's a pain to do. The gnu toolchain really likes to use dynamic linking, and building a static lib that will run on a maybe-older-than-the-build-system machine is pretty tricky. - now we end up with multiple copies of the same lib in the python install. There are a handful of libs that are used a LOT. Maybe there is no real downside -- disk space and memory are cheap these days, but it sure feels ugly. And I have yet to feel comfortable with having multiple versions of the same lib linked into one python instance -- I can't say I've seen a problem, but it makes me nervous. On Windows, the choices are the same, except that: It is so much harder to build many of the standard open source libs that package authors are more likely to do it for folks, and you do get the occasional dll hell issues. I had a plan to make some binary wheels for OS-X that were not really python packages, but actually just bundled up libs, so that other wheels could depend on them. OS-X does allow linking to relative paths, so this should have been doable, but I never got anyone else to agree this was a good idea, and I never found the roundtoits anyway. And it doesn't really fit into the PyPi, pip, wheel, etc. philosphy to have dependencies that are platform dependent and even worse, build-dependent. Meanwhile, conda was chugging along and getting a lot of momentum in the Scientific community. And the core thing here is that conda was designed from the ground up to support essentially anything, This means is supports python packages that depend on non-python packages, but also supports packages that have nothing to do with python (Perl, command line tools, what have you...) So I have been focusing on conda lately. Which brings me back to the question: should the python tools (i.e. wheel) be extended to support more use-cases, specifically non-python dependencies? Or do we just figure that that's a problem better solved by projects with a larger scope (i.e. rpm, deb, conda, canopy). I'm on the fence here. I mostly care about Python, and I think we're pretty darn close with allowing wheel to support the non-python dependencies, which would allow us all to simply pip install pretty much anything -- that would be cool. But maybe it's a bit of a slippery slope, and if we go there, we'll end up re-writing conda. BTW, while you can't generally install a conda package in/for another python, you can generally install a wheel in a conda pythonThere are a few issues with pip/setuptools trying to resolve dependencies while not knowing about conda packages, but it does mostly work. Not sure that helped the discussion -- but I've been wrestling with this for a while, so thought I'd get my thoughts out there. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
#1 is pretty straightforward. An entry-point format Python
pre/post/etc. script may do.
I have some ideas for the FHS, though I fear it's full of bikesheds:
1. Allow all GNU directory variables as .data/* subdirectories
(https://www.gnu.org/prep/standards/html_node/Directory-Variables.html).
The distutils names will continue to be allowed.
packagename-1.0.data/mandir/...
2. Make data_files useful again. Interpolate path variables into
distutils data_files using $template syntax. (Only allow at
beginning?)
data_files=[('$mandir/xyz', ['manfile', 'other_man_file'])
In addition to $bindir, $mandir, etc. it will be important to allow
the package name and version to be interpolated into the install
directories.
Inside the wheel archive, you will get
packagename-1.0.data/mandir/manfile and
packagename-1.0.data/mandir/other_man_file
3. Write the install paths (the mapping from $bindir, $mandir, $prefix
etc. to the actual paths used) to one or more of a .py, .json, or
.dist-info/* based on new metadata in WHEEL:
install-paths-to: wheel/_paths.py
It is critical that this be allowed to work without requiring the end
user to look for it with pkg_resources or its pals. It's also good to
only write it if the installed package actually needs to locate its
file categories after it has been installed.
This will also be written inside the wheel itself with relative paths
to the .data/ directory.
4. Allow configurable custom paths. The GNU paths could be
configured relative to the distutils paths as a default. We might let
the user add additional paths with a configuration dict.
paths = {
foo : $bar/${quux},
bar : ${baz}/more/stuff,
baz : ${quux}/again,
quux: larry
}
5. On Windows, no one will really care where most of these files go,
but they probably won't mind if they are installed into separate
directories. Come up with sensible locations for the most important
categories.
On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft don...@stufft.io wrote:
On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote:
Hi there,
During pycon, Nick mentioned there was interest in updating the wheel format
to support downstream distributions. Nick mentioned Linux distributions, but
I would like to express interest for other kind of downstream distributors
like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work
for Enthought).
Right now, wheels have the following limitations for us:
1. lack of post/pre install/removing
2. more fine-grained installation scheme
3. lack of clarify on which tags vendors should use for custom wheels: some
packages we provide would not be installable on normal python, and it
would be nice to have a scheme to avoid confusion there as well.
At least 1. and 2. are of interest not just for us.
Regarding 2., it looks like anything in the wheel_name.data/data directory
will be placed as is in sys.prefix by pip. This is how distutils scheme is
defined ATM, but I am not sure whether that's by design or accident ?
I would suggest to use something close to autotools, with some tweaks to
work well on windows. I implemented something like this in my project bento
(https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
but we could of course tweak that.
For 1., I believe it was a conscious decision not to include them in wheel
1.0 ? Would it make sense to start a discussion to add it to wheel ?
I will be at the pycon sprints until wednesday evening, so that we can flesh
some concrete proposal first, if there is enough interest.
As a background: at Enthought, we have been using eggs to distribute
binaries of python packages and other packages (e.g. C libraries, compiled
binaries, etc...) for a very long time. We had our own extensions to the egg
format to support this, but I want to get out of eggs so as to make our own
software more compatible with where the community is going. I would also
like to avoid making ad-hoc extensions to wheels for our own purposes.
To my knowledge, (1) was purposely punted until a later revision of Wheel
just to make it easier to land the “basic” wheel.
I think (2) is a reasonable thing as long as we can map it sanely on all
platforms.
I’m not sure what (3) means exactly. What is a “normal” Python, do you
modify Python in a way that breaks the ABI but which isn’t reflected in the
standard ABI tag?
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On 13 April 2015 at 16:02, Daniel Holth dho...@gmail.com wrote: #1 is pretty straightforward. An entry-point format Python pre/post/etc. script may do. There's metadata 2.0 information for this. It would be sensible to follow that definition where it applies, but otherwise yes, this shouldn't be hard. Some thoughts, though: 1. Some thought should be put into how we ensure that pre/post install/remove scripts are cross-platform. It would be a shame if a wheel was unusable on Windows for no reason other than that the postinstall script was written as a bash script. Or on Unix because the postinstall script tried to write Windows start menu items. 2. It's worth considering appropriate use of such scripts. The Windows start menu example is relevant here - I can easily imagine users requesting something like that for a wheel they want to install into the system Python, but it's completely inappropriate for installing into a virtualenv. To an extent, there's nothing we can (or maybe even should) do about this - projects that include inappropriate install scripts will get issues raised or will lose users, and the problem is self-correcting to an extent, but it's probably worth including in the implementation, some work to add appropriate documentation to the packaging user guide about best practices for pre/post-install/remove scripts (hmm, a glossary entry with a good name for these beasts would also be helpful :-)) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more
On Mon, Apr 13, 2015 at 10:54 AM, David Cournapeau courn...@gmail.com wrote: On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft don...@stufft.io wrote: On Apr 13, 2015, at 10:39 AM, David Cournapeau courn...@gmail.com wrote: Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on normal python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the wheel_name.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. To my knowledge, (1) was purposely punted until a later revision of Wheel just to make it easier to land the “basic” wheel. Great. Was there any proposal made to support it at all ? Or should I just work from scratch there ? I think (2) is a reasonable thing as long as we can map it sanely on all platforms. Yes. We support all platforms at Enthought, and Windows is important for us ! I’m not sure what (3) means exactly. What is a “normal” Python, do you modify Python in a way that breaks the ABI but which isn’t reflected in the standard ABI tag? It could be multiple things. The most obvious one is that generally. cross-platforms python distributions will try to be relocatable (i.e. the whole installation can be moved and still work). This means they require python itself to be built a special way. Strictly speaking, it is not an ABI issue, but the result is the same though: you can't use libraries from anaconda or canopy on top of a normal python More generally, we could be modifying python in a way that is not forward compatible with upstream python: a binary that works on our python may not work on the python from python.org (though the opposite is true). It would be nice if one could make sure pip will not try to install those eggs when installed on top of a python that does not advertise itself as compatible We need a hook to alter pip's list of compatible tags (pip.pep425tags), and to alter the default tags used by bdist_wheel when creating wheels. One sensible proposal for special wheels is to just use a truncated hash of the platform description (a random hex string) in place of the wheel platform tag. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
