On Tue, May 7, 2024 at 7:48 AM Juan Nunez-Iglesias <j...@fastmail.com> wrote:
> On Tue, 7 May 2024, at 7:04 AM, Ralf Gommers wrote: > > This problem could have been avoided by proper use of upper bounds. > Scikit-image 0.22 not including a `numpy<2.0` upper bound is a bug in > scikit-image (definitely for ABI reasons, and arguably also for API > reasons). It would really be useful if downstream packages started to take > adding upper bounds correctly more seriously. E.g., SciPy has always done > it right, so the failure mode that this thread is about doesn't exist for > SciPy. That said, this ship has sailed for 2.0 - most packages don't have > upper bounds in some or all of their recent releases. > > > I don't think this is a downstream problem, > This is definitely a bug in scikit-image (see below), that is really not up for discussion here. If you use the NumPy C API, you MUST add a constraint to stay below the next major version. > I think this is a "PyPI has immutable metadata" problem. > Yes, I'm very much aware: https://pypackaging-native.github.io/key-issues/pypi_metadata_handling/#impact-of-immutable-metadata. This just is what it is though, we have to live within the constraints of how PyPI is designed. > I'm a big fan of Henry Schreiner's "Should You Use Upper Bound Version > Constraints" <https://iscinumpy.dev/post/bound-version-constraints/>, > where he argues convincingly that the answer is almost always no. > Emphasis on "almost". Unfortunately, too many maintainers took away the wrong message from that blog post - probably because it was written in frustration at Poetry *defaulting* to adding upper bounds, and hence is a bit too strong on the arguments against. The point was "default to adding no bounds, only add them if you must" (because on average they do more harm than good). Translating that to "never add any upper bounds" is wrong and harmful. To quote Henry himself from another recent discussion where a maintainer incorrectly used this blog post to argue against upper bounds ( https://github.com/sympy/sympy/issues/26273#issuecomment-2028192560): *"You are correct, the post is not meant to say you can't upper cap (except the Python version, which doesn't even work correctly if you upper cap) if you've thought about it and have a reason to. It does increase friction, but if there's a high chance the next version(s) will break you and you are willing to make releases shortly after the dependency does & keep the cap up to date, go ahead and cap."* In this case, it's not just a high chance that things will break, it's guaranteed to break. NumPy does an import check for ABI mismatch, and will always raise an exception if you build against 1.x and run with 2.x (or 2.x and 3.x, etc.). So you must add `<2` when building against numpy 1.x, and `<3` when building against 2.x. Leaving out the upper bound achieves exactly nothing except for making things break for the users of your package. Dropping Py 3.9 will fix the issue for a subset of users, but certainly not > all users. Someone who pip installs scikit-image==0.22 on Py 3.10 will have > a broken install. But importantly, they will be able to fix it in user > space. > Yes, this cannot be fixed on the NumPy side either way. But that's a separate issue. There are three cases we have to think about: 1. Everything unconstrained: `pip install scikit-image` or `pip install scikit-image numpy` 2. Everything constrained: `pip install scikit-image==0.22 numpy==1.26` (+ everything else, typically used through a lock file) 3. A mix of constrained and unconstrained: `pip install scikit-image==0.22` or `pip install scikit-image numpy=2.0` or `pip install scikit-image==0.22 numpy` (2) is typically robust since lock files are usually generated from already-working envs. Unless the list of packages is incomplete, and then it may turn into (3). (3) is up to the authors of each package, there's nothing we can do in numpy to make `pip install scikit-image==0.22` work if that version of scikit-image depends on an unconstrained numpy version. (1) is what this thread, and the rollout plans for NumPy 2.0, are concerned about. If we drop Python 3.9, we make that work more reliably. If we don't, then on Python 3.9 a plain `pip install scikit-image` will be broken. > > At any rate, it's not like NumPy (or SciPy, or scikit-image) don't change > APIs over several minor versions. > We have extensive docs on this, and a documented backwards compatibility policy that we try hard to adhere to: https://numpy.org/neps/nep-0023-backwards-compatibility.html https://numpy.org/devdocs/dev/depending_on_numpy.html#runtime-dependency-version-ranges I'm honestly a little tired of maintainers just ignoring that and misunderstanding the problem. In short, and independent of the Py3.9 issue, I don't think we should > advocate for upper caps in general > No one is doing that "in general". It is specific to NumPy's situation, and cannot be avoided. It's a price to pay of dealing with an ABI. Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
