Re: [Distutils] Proposal: using /etc/os-release in the platform tag definition for wheel files
On 29 November 2014 at 01:51, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 29 Nov 2014 01:27:44 +1000 Nick Coghlan ncogh...@gmail.com wrote: Is this not going to be a slippery slope? Only if folks publish Linux binaries themselves, and that's still a bad idea (for the same reason publishing distro binaries is already a rare thing for people to do). Well, let's not make this a matter of ideology. Everyone knows it's a bad idea to publish binaries, yet it's often better than nothing, especially if the software is tedious to compile. It's not a matter of ideology, but a matter of practicality. Debian stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow enough (and have strong enough ABI compatibility guarantees) to be practical for ISVs to target with prebuilt binaries. Beyond that, the rate of development and breadth of target environments in the Linux world tends to make providing prebuilt binaries in *any* format difficult (unless you statically link everything you need, which works fine until you decide you want a particular package to depend on binaries provided by another one by linking to them dynamically). Regardless of target environment, though, being able to prebuild wheel files in a way that clearly indicates the platform restricted nature of the end result is a useful tool for system integrators. That benefit applies regardless of whether the builds are happening in the context of Linux distro development, or in the context of maintaining a particular set of infrastructure services. How many binary packages will package authors have to provide to cover people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied by three or four Python versions is already a lot of binaries to build... I'd still advise against folks posting Linux wheels on PyPI, just as they tend not to post RPM or deb files. This is so we can provide wheels at the distro level (or build them internally) without creating vast amounts of confusion. So do we (software authors) have to wait for that mythical we who are going to build binaries in time for our packages? If the compatibility tagging issue can be resolved, I don't think there should be any restrictions against uploading Linux wheels that avoid the generic Linux platform tags. I just doubt it will make sense for most developers to worry about it, just as most of them don't worry about providing RPMs or Debian packages. Case in point: can I ask you (the mythical we) to build packages for all major distros (including supported LTS releases), and the four most recent Python versions, of the following piece of software: https://github.com/numba/llvmlite No, that would be a service provided by the as yet hypothetical PyPI build farm. If/when that happens, it will need to have a way of tagging Linux wheels appropriately, though. Nearer term (and what prompted me to start this thread), the Fedora Environments Stacks working group is investigating providing prebuilt wheel files for the Fedora ecosystem, and potentially for EPEL as well (see https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement for the broader context of that effort). For other ecosystems, you'll have to ask participants in those ecosystems. This isn't really about that - it's about having a way to tackle it at the distro level, without introducing significant potential for confusion on end user systems I'm not sure I understand: distros provide their own packages, they don't (shouldn't) blindly pull binary wheels from PyPI. Why would they depend on the wheel tagging format? We don't plan to blindly pull anything from PyPI - we're looking at the feasibility of publishing already reviewed software in ecosystem native formats (with the two pilot projects focusing on Java JAR files and Python wheel files). When I last mentioned that idea here, Marcus pointed out that doing that with the generic linux_x86_64 compatibility tag on the wheel filenames would be problematic, as there'd be nothing preventing anyone from pulling them down onto inappropriate systems, with no obvious trace of the Fedora or EPEL specific assumptions that went into building them. While that's a valid concern, I also don't want to go invent our own custom compatibility tagging convention just for Fedora EPEL, but rather work within the limits of what upstream Python packaging natively supports. At the moment there is no such tagging system, which is why I'm interested in pursuing improvements to the definition of the platform tag. However, after thinking further about the situation with EPEL (where we'd likely want a single set of wheel files to cover not only an entire major release series, but also downstream RHEL derivatives like CentOS), the possibility of sharing wheel files between the Fedora Software Collections Python builds, and the point Matthias raised about the limitations of the current platform tag when it comes to
Re: [Distutils] Proposal: using /etc/os-release in the platform tag definition for wheel files
On Sun, 30 Nov 2014 01:47:16 +1000 Nick Coghlan ncogh...@gmail.com wrote: On 29 November 2014 at 01:51, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 29 Nov 2014 01:27:44 +1000 Nick Coghlan ncogh...@gmail.com wrote: Is this not going to be a slippery slope? Only if folks publish Linux binaries themselves, and that's still a bad idea (for the same reason publishing distro binaries is already a rare thing for people to do). Well, let's not make this a matter of ideology. Everyone knows it's a bad idea to publish binaries, yet it's often better than nothing, especially if the software is tedious to compile. It's not a matter of ideology, but a matter of practicality. Debian stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow enough (and have strong enough ABI compatibility guarantees) to be practical for ISVs to target with prebuilt binaries. It seems we disagree on the notion of practicality :-) For me practicality means being able to build a single binary package for all recent Linux distros in a best effort approach. Building a different package for each distro version is far from practical for any reasonable-sized project (i.e. not something sponsored by a 1000+-employee entity, with a dedicated build team and infrastructure). Case in point: can I ask you (the mythical we) to build packages for all major distros (including supported LTS releases), and the four most recent Python versions, of the following piece of software: https://github.com/numba/llvmlite No, that would be a service provided by the as yet hypothetical PyPI build farm. If/when that happens, it will need to have a way of tagging Linux wheels appropriately, though. If/when that happens is not reassuring, especially in the light of how many pie-in-the-sky improvements in the packaging ecosystems have turned out :-/ (at Continuum we have started offering such a service, but it's generic Linux: http://docs.binstar.org/build-config.html#BuildMatrix) Nearer term (and what prompted me to start this thread), the Fedora Environments Stacks working group is investigating providing prebuilt wheel files for the Fedora ecosystem, and potentially for EPEL as well (see https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement for the broader context of that effort). For other ecosystems, you'll have to ask participants in those ecosystems. That's asking software authors to complicate and slow down their development process a lot. Also, there's no guarantee that Fedora or Ubuntu or whatever would actually *accept* to help us, right? I'm not sure I understand: distros provide their own packages, they don't (shouldn't) blindly pull binary wheels from PyPI. Why would they depend on the wheel tagging format? We don't plan to blindly pull anything from PyPI - we're looking at the feasibility of publishing already reviewed software in ecosystem native formats (with the two pilot projects focusing on Java JAR files and Python wheel files). When I last mentioned that idea here, Marcus pointed out that doing that with the generic linux_x86_64 compatibility tag on the wheel filenames would be problematic, as there'd be nothing preventing anyone from pulling them down onto inappropriate systems, with no obvious trace of the Fedora or EPEL specific assumptions that went into building them. Uh, there's a lot of hidden knowledge required to understand those two paragraphs that I don't master. I don't know what inappropriate systems are, what are reviewed software, etc. ;-) Also I don't understand why you're not recompiling as you would normally do. While that's a valid concern, I also don't want to go invent our own custom compatibility tagging convention just for Fedora EPEL, but rather work within the limits of what upstream Python packaging natively supports. Well, *allowing* distro tags in the platform tag is certainly ok. What I'm afraid of is if that's made mandatory. Consider if the following could be included in the pyvenv.cfg file in a virtual environment: [compatibility] python=cp27,cp2,py2 abi=cp27mu platform=linux_x86_64_epel_6 Or for a Python 3 virtual environment: [compatibility] python=cp34,cp3,py3 abi=cp34m,abi3 platform=linux_x86_64_epel_6 If present, these pyvenv.cfg settings would override the normal PEP 425 compatibility tag calculations (I'm OK with the idea of *needing* to be in a virtual environment to gain the power to configure these tags). As long as it's on an opt-in basis, it certainly sounds ok. Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proposal: using /etc/os-release in the platform tag definition for wheel files
On 30 November 2014 at 02:10, Antoine Pitrou solip...@pitrou.net wrote: On Sun, 30 Nov 2014 01:47:16 +1000 Nick Coghlan ncogh...@gmail.com wrote: On 29 November 2014 at 01:51, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 29 Nov 2014 01:27:44 +1000 Nick Coghlan ncogh...@gmail.com wrote: Is this not going to be a slippery slope? Only if folks publish Linux binaries themselves, and that's still a bad idea (for the same reason publishing distro binaries is already a rare thing for people to do). Well, let's not make this a matter of ideology. Everyone knows it's a bad idea to publish binaries, yet it's often better than nothing, especially if the software is tedious to compile. It's not a matter of ideology, but a matter of practicality. Debian stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow enough (and have strong enough ABI compatibility guarantees) to be practical for ISVs to target with prebuilt binaries. It seems we disagree on the notion of practicality :-) Note I said ISVs there - folks that actually make money from targetting those platforms. I don't think providing integrated Linux binaries is currently practical at all for open source projects (note that even CPython just provides a tarball upstream, with the builds handled directly by the distros). For me practicality means being able to build a single binary package for all recent Linux distros in a best effort approach. That isn't realistically possible for anything that isn't completely statically linked, as distro level choices of build options vary too much. We could technically allow statically linked binaries on PyPI by dropping the restriction against generic Linux wheel files, but I'm wary of that in the absence of an automated server side scan for dynamically linked binaries. Many users (quite reasonably, if they're primarily Python developers) have problems working through build failures when attempting to install non-Python extensions from source. Such build failures are usually models of clarity compared to diagnosing dynamic linking failures. Building a different package for each distro version is far from practical for any reasonable-sized project (i.e. not something sponsored by a 1000+-employee entity, with a dedicated build team and infrastructure). This is why I'm saying people *shouldn't* try to provide prebuilt Linux binaries in the general case: it's simply not practical. Hence the rise of higher level, more self-contained formats like Docker, as well as the fact that I'm not even trying to solve the any Linux problem myself, but going after the vastly simpler problem of targeting the ecosystem I need to target (but hopefully in a sufficiently general purpose way that other ecosystems can adopt the same model if they so choose). Case in point: can I ask you (the mythical we) to build packages for all major distros (including supported LTS releases), and the four most recent Python versions, of the following piece of software: https://github.com/numba/llvmlite No, that would be a service provided by the as yet hypothetical PyPI build farm. If/when that happens, it will need to have a way of tagging Linux wheels appropriately, though. If/when that happens is not reassuring, especially in the light of how many pie-in-the-sky improvements in the packaging ecosystems have turned out :-/ Nobody is currently offering to provide paid staff to work on it, so it won't happen until it gets to the top of volunteers' personal priority lists. (at Continuum we have started offering such a service, but it's generic Linux: http://docs.binstar.org/build-config.html#BuildMatrix) Yes, Continuum avoided the distro ABI compatibility problem by defining its own ABI. It's exactly the same model I'm proposing - in the general case, you can't take Continuum built packages, and use them with Fedora/Debian/etc built dynamically linked dependencies. Same with my suggestion - it's about labelling the compatibility requirements, rather than trying to avoid them by statically linking everything. Nearer term (and what prompted me to start this thread), the Fedora Environments Stacks working group is investigating providing prebuilt wheel files for the Fedora ecosystem, and potentially for EPEL as well (see https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement for the broader context of that effort). For other ecosystems, you'll have to ask participants in those ecosystems. That's asking software authors to complicate and slow down their development process a lot. If the distro level service doesn't benefit them, there's no reason for software authors to use it. This is about adding a new option for consuming software, not taking anything away. In my case, I won't deploy something until it's been through at least a basic licensing review, and if we're doing that work anyway, I may as well see if we can find a way to do it
Re: [Distutils] Proposal: using /etc/os-release in the platform tag definition for wheel files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/29/2014 11:10 AM, Antoine Pitrou wrote: For me practicality means being able to build a single binary package for all recent Linux distros in a best effort approach. I can't imagine finding any such binary useful (assuming a none-pure-Python project): the chance that it *might* blow up makes it easier just to create my own from the sdist. Such binaries would tend to bitrot, even if they were OK for the supported set of distros at the time they were made. How can such wheels be feasible, when cross-distro RPMs / .debs are clearly not (in the general sense)? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlR6BMUACgkQ+gerLs4ltQ70sgCeJ8BmRVDOhEzJ4I70eGWQCLn4 l0cAoJwoHcC26M/t5mhLXgJl4IrugDcV =G5zd -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proposal: using /etc/os-release in the platform tag definition for wheel files
On Sun, 30 Nov 2014 03:02:57 +1000 Nick Coghlan ncogh...@gmail.com wrote: Many users (quite reasonably, if they're primarily Python developers) have problems working through build failures when attempting to install non-Python extensions from source. Such build failures are usually models of clarity compared to diagnosing dynamic linking failures. However, installing a binary doesn't imply a potential longish building step, or the installation of many build dependencies. LLVM can take 20 minutes to compile on a modern quad-core x86. I've been told it takes several hours on a Cortex A8 platform... By comparison, the failure of loading a precompiled dynamic library is instantaneous. And I don't think build failures are understandable by many users. You need to be a seasoned C developer for that. (at Continuum we have started offering such a service, but it's generic Linux: http://docs.binstar.org/build-config.html#BuildMatrix) Yes, Continuum avoided the distro ABI compatibility problem by defining its own ABI. Not exactly. Some ABI problems - for example the glibc-related ones - are still here. Conda and binstar-build are still a best effort (on the GNU/Linux side, that is), not an ideal solution. Well, *allowing* distro tags in the platform tag is certainly ok. What I'm afraid of is if that's made mandatory. OK, that makes more sense. Yes, I agree we need to keep the ability to say this is a prebuilt, self-contained, binary wheel that should run on any Linux system because it doesn't link to any system binaries. Chalk it up as yet another reason that the specific proposal I started the thread with wouldn't actually work :) Great! Regards Antoine. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2
On Nov 13, 2014, at 9:21 PM, Donald Stufft don...@stufft.io wrote: Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256). Ping? Are we OK to make this change? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
