On 13 May 2015 at 16:19, Ben Finney <ben+pyt...@benfinney.id.au> wrote: > Chris Barker <chris.bar...@noaa.gov> writes: > >> On Tue, Apr 14, 2015 at 8:41 AM, Nick Coghlan <ncogh...@gmail.com> wrote: >> >> > The point where I draw the line is supporting *dynamic* linking >> > between modules - >> >> I'm confused -- you don't want a system to be able to install ONE version >> of a lib that various python packages can all link to? That's really the >> key use-case for me.... > > Agreed. A key pain point for Python distributions is the lack of support > for installing *one* instrance of a Python library, and other Python > modules able to discover such installed libraries which meet their > declared dependency.
Are we talking about Python libraries accessed via Python APIs, or linking to external dependencies not written in Python (including linking directly to C libraries shipped with a Python library)? It's the latter I consider to be out of scope for a language specific packaging system - Python packaging dependencies are designed to describe inter-component dependencies based on the Python import system, not dependencies based on the operating system provided C/C++ dynamic linking system. If folks are after the latter, than they want a language independent package system, like conda, nix, or the system package manager in a Linux distribution. > For example: > > * Python distribution ‘foo’, for Python implementation “D” on > architecture “X“, declares dependency on “bar >= 1.7”. > > * Installing Python distribution ‘bar’ version 1.8, on a host running > Python “D” for architecture “X”, goes to a single instance for the > ‘bar’ library for that architecture and Python implementation. > > * Invoking the ‘foo’ code on the same host will go looking > (dynamically?) for the dependency ‘bar’ and find version 1.8 already > installed in the one instance on that host. It uses that and all is > well. > > I'm in agreement with Chris that, while the above example may not > currently play out as described, that is a fault to be fixed by > improving Python's packaging and distribution tools so that it *is* a > first-class use case. > > Nick, you seem to be arguing against that. Can you clarify? I'm arguing against supporting direct C level dependencies between packages that rely on dynamic linking to find each other rather than going through the Python import system, as I consider that the point where you cross the line into defining a new platform of your own, rather than providing components that can plug into a range of platforms. (Another way of looking at this: if a tool can manage the Python runtime in addition to Python modules, it's a full-blown arbitrary software distribution platform, not just a Python package manager). Defining cross-platform ABIs (cf. http://bugs.python.org/issue23966) is an unholy mess that will be quite willing to consume vast amounts of time without a great deal to show for it beyond can already be achieved more easily by telling people to just use one of the many existing systems designed specifically to solve that problem (with conda being my default recommendation if you care about Windows, and nix being my recommendation if you only care about *nix systems). Integrator oriented packaging tools and developer oriented packaging tools solve different problems for different groups of people, so I'm firmly of the opinion that trying to solve both sets of problems with a single tool will produce a result that doesn't work as well for *either* use case as separate tools can. Cheers, Nick. P.S. The ABI definition problem is at least somewhat manageable for Windows and Mac OS X desktop/laptop environments (since you can mostly pretend that architectures other than x86_64 don't exist, with perhaps some grudging concessions to the existence of 32-bit mode), but beyond those two, things get very messy, very fast - identifying CPU architectures, CPU operating modes and kernel syscall interfaces correctly is still a hard problem in the Linux distribution space, and they've been working at it a lot longer than we have (and that's *without* getting into things like determining which vectorisation instructions are available). Folks often try to "deal" with this complexity by wishing it away, but the rise of aarch64 and IBM's creation of the OpenPOWER Foundation is making the data centre space interesting again, while in the mobile and embedded spaces it's ARM that is the default, with x86_64 attempting to make inroads. As a result of all that, distributing software that uses dynamically linked dependencies is genuinely difficult, to the point where even operating system vendors struggle to get it right. This is why "statically link all the things" keeps coming back in various guises (whether that's Go's lack of dynamic linking support, or the surge in adoption of Linux containers as a deployment technique), despite the fact that these techniques inevitably bring back the *other* problems that led to the invention of dynamic linking in the first place. The only solution that is known to work reliably for dynamic linking is to have a curated set of packages all built by the same build system, so you know they're using consistent build settings. Linux distributions provide this, as do multi-OS platforms like nix and conda. We *might* be able to provide it for Python someday if PyPI ever gets an integrated build farm, but that's still a big "if" at this point. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig