On 30 November 2014 at 02:10, Antoine Pitrou <solip...@pitrou.net> wrote: > On Sun, 30 Nov 2014 01:47:16 +1000 > Nick Coghlan <ncogh...@gmail.com> wrote: >> On 29 November 2014 at 01:51, Antoine Pitrou <solip...@pitrou.net> wrote: >> > On Sat, 29 Nov 2014 01:27:44 +1000 >> > Nick Coghlan <ncogh...@gmail.com> wrote: >> >> > >> >> > Is this not going to be a slippery slope? >> >> >> >> Only if folks publish Linux binaries themselves, and that's still a >> >> bad idea (for the same reason publishing distro binaries is already a >> >> rare thing for people to do). >> > >> > Well, let's not make this a matter of ideology. Everyone knows it's a >> > bad idea to publish binaries, yet it's often better than nothing, >> > especially if the software is tedious to compile. >> >> It's not a matter of ideology, but a matter of practicality. Debian >> stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow >> enough (and have strong enough ABI compatibility guarantees) to be >> practical for ISVs to target with prebuilt binaries. > > It seems we disagree on the notion of "practicality" :-)
Note I said "ISVs" there - folks that actually make money from targetting those platforms. I don't think providing integrated Linux binaries is currently practical at all for open source projects (note that even CPython just provides a tarball upstream, with the builds handled directly by the distros). > For me practicality means being able to build a single binary package > for all recent Linux distros in a best effort approach. That isn't realistically possible for anything that isn't completely statically linked, as distro level choices of build options vary too much. We could technically allow statically linked binaries on PyPI by dropping the restriction against generic Linux wheel files, but I'm wary of that in the absence of an automated server side scan for dynamically linked binaries. Many users (quite reasonably, if they're primarily Python developers) have problems working through build failures when attempting to install non-Python extensions from source. Such build failures are usually models of clarity compared to diagnosing dynamic linking failures. > Building a > different package for each distro version is far from practical for any > reasonable-sized project (i.e. not something sponsored by a > 1000+-employee entity, with a dedicated build team and infrastructure). This is why I'm saying people *shouldn't* try to provide prebuilt Linux binaries in the general case: it's simply not practical. Hence the rise of higher level, more self-contained formats like Docker, as well as the fact that I'm not even trying to solve the "any Linux" problem myself, but going after the vastly simpler problem of targeting the ecosystem I need to target (but hopefully in a sufficiently general purpose way that other ecosystems can adopt the same model if they so choose). >> > Case in point: can I ask you (the mythical "we") to build packages for >> > all major distros (including supported LTS releases), and the four most >> > recent Python versions, of the following piece of software: >> > https://github.com/numba/llvmlite >> >> No, that would be a service provided by the as yet hypothetical PyPI >> build farm. If/when that happens, it will need to have a way of >> tagging Linux wheels appropriately, though. > > "If/when that happens" is not reassuring, especially in the light of > how many pie-in-the-sky improvements in the packaging ecosystems have > turned out :-/ Nobody is currently offering to provide paid staff to work on it, so it won't happen until it gets to the top of volunteers' personal priority lists. > (at Continuum we have started offering such a service, but it's > "generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix) Yes, Continuum avoided the distro ABI compatibility problem by defining its own ABI. It's exactly the same model I'm proposing - in the general case, you can't take Continuum built packages, and use them with Fedora/Debian/etc built dynamically linked dependencies. Same with my suggestion - it's about labelling the compatibility requirements, rather than trying to avoid them by statically linking everything. >> Nearer term (and what prompted me to start this thread), the Fedora >> Environments & Stacks working group is investigating providing >> prebuilt wheel files for the Fedora ecosystem, and potentially for >> EPEL as well (see >> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement >> for the broader context of that effort). For other ecosystems, you'll >> have to ask participants in those ecosystems. > > That's asking software authors to complicate and slow down their > development process a lot. If the distro level service doesn't benefit them, there's no reason for software authors to use it. This is about adding a new option for consuming software, not taking anything away. In my case, I won't deploy something until it's been through at least a basic licensing review, and if we're doing that work anyway, I may as well see if we can find a way to do it upstream in Fedora rather than inside the Red Hat firewall. Same for prebuilding binary wheels - sure, we could do that just for our own internal infrastructure, but it makes more sense to me to at least investigate the idea of handling it upstream first. > Also, there's no guarantee that Fedora or > Ubuntu or whatever would actually *accept* to help us, right? For Fedora, myself and Slavek (the Fedora Python maintainer) are the two folks working on this (& we're both voting members of the Environments & Stacks working group). So yes, in Fedora's case, it's definitely a developer experience problem we want to solve properly, and we're already working on it - this is me asking upstream for help resolving a design problem, rather than coming up with a speculative idea to persuade Fedora to adopt later. I realise that's very different from the historical attitude of Linux distributions to upstream packaging ecosystems. Getting to this point of adopting a more user-focused mindset when asking what services a Linux distro should be offering has been a formidable political challenge involving a lot of work from a lot of different people (including the current Fedora Project Leader), so I'm not even going to speculate on what might be involved in attempting to replicate that in the context of a different distro ecosystem :) >> > I'm not sure I understand: distros provide their own packages, they >> > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they >> > depend on the wheel tagging format? >> >> We don't plan to blindly pull anything from PyPI - we're looking at >> the feasibility of publishing already reviewed software in ecosystem >> native formats (with the two pilot projects focusing on Java JAR files >> and Python wheel files). >> >> When I last mentioned that idea here, Marcus pointed out that doing >> that with the generic "linux_x86_64" compatibility tag on the wheel >> filenames would be problematic, as there'd be nothing preventing >> anyone from pulling them down onto inappropriate systems, with no >> obvious trace of the Fedora or EPEL specific assumptions that went >> into building them. > > Uh, there's a lot of hidden knowledge required to understand those > two paragraphs that I don't master. I don't know what "inappropriate > systems" are, what are "reviewed software", etc. ;-) An example of inappropriate systems: trying to install a Fedora wheel on Ubuntu. That often won't work in the general case, but pip will currently allow it (because they share a platform tag, despite not sharing a platform ABI) "Reviewed software" just refers to the Fedora package set. Currently we only make that available for installation as RPMs, the Environments & Stacks WG is looking at making it available in more developer friendly formats, rather than focusing solely on the operations use case. (We're not sure yet how we'll make that sustainable from a build infrastructure perspective, but that's a large part of what the pilot projects are aiming to figure out) > Also I don't understand why you're not recompiling as you would > normally do. We'll still be recompiling. We just want to publish the results for Python packages as virtual-environment compatible wheel files, not just as RPMs. >> While that's a valid concern, I also don't want to go invent our own >> custom compatibility tagging convention just for Fedora & EPEL, but >> rather work within the limits of what upstream Python packaging >> natively supports. > > Well, *allowing* distro tags in the platform tag is certainly ok. What > I'm afraid of is if that's made mandatory. OK, that makes more sense. Yes, I agree we need to keep the ability to say "this is a prebuilt, self-contained, binary wheel that should run on any Linux system because it doesn't link to any system binaries". Chalk it up as yet another reason that the specific proposal I started the thread with wouldn't actually work :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
