On Sun, May 12, 2024 at 8:39 PM Gael Varoquaux <
gael.varoqu...@normalesup.org> wrote:
> On Fri, May 10, 2024 at 04:42:49PM +0200, Ralf Gommers wrote:
> > It gets ever-easier to install new Python versions, with
> pyenv/conda/etc. The "my single Python install comes from python.org and
> I'm using the same one because I am afraid to upgrade" is much less of an
> issue than it was 10 years ago. And it's caused mostly by users having
> knowledge gaps. So yes it can be a pain for them, but they'll have to learn
> at some point anyway. Same for "my old HPC cluster uses ..." - it's often
> an even older Python anyway, and you'll have tons of reasons why you don't
> want your cluster-installed stack - learn to use Spack or Conda, and you'll
> be much happier in the long run.
>
> IMHO the view that its a tooling/knowledge gap problem is a bit
> disconnected of users. I meet many users who either
>
> 1. cannot control the Python version they run, or even the base
> environment, because of company culture (decisions at company level on
> these constraints). Maybe upper management in completely misguided here,
> but then upper management must be targeted, and it is not clear at all that
> constraints on packages is the right way to do it, as they typically never
> run any code.
>
> 2. have environments with many dependencies that get in gridlocks of
> dependencies that cannot be mutually satisfied. For my own work it becomes
> increasingly frequent for me to spawn multiple environments that then talk
> to each others (eg via files) to work around these problems.
>
>
> Problem 1 is probably something that user organizations should change, and
> we need to understand why they lock Python versions.
I don't think the problem is what you think it is. It's typically general
IT policies, meaning you cannot do things like download and install random
executables in places like C:\. This is quite common, and it's mostly some
Python installers (python.org ones in particular) falling under a general
policy. It's very rare to have a Python-tooling-specific block - I don't
think I have ever seen something like that. If you can run `pip` and
install to for example locations under your home dir, then you almost
always can also run pyenv/conda/mamba & co to install different Python
versions. And if you cannot run `pip`, there is nothing to discuss w.r.t.
release policy, new package versions don't matter.
It could be a QA issue, and this might reveal both a lack of good practices
> for QA (ie testing) but also the instability of the ecosystems that creates
> a fear of upgrade. We should not be too fast in dismissing these
> organizations as strife with bad practices that could easily be changed, as
> even tech-savy organizations (such as Google, I believe) run into these
> problems.
>
Google (along with many other large tech companies) uses a monorepo. That
means that there is only a single version of any package. This can be a
great strategy once you're large enough; it has costs too, but certainly
shouldn't be qualified as a bad or uninformed strategy. A common support
window like SPEC 0 is actually helpful there.
> Problem 2 is not a problem solvable by users: it comes from the fact that
> dependency version windows are too narrow. Without broad windows on
> dependencies, the more dependencies one has, the more one gets into an
> impossible situation. For this last reason, I strongly advocate that
> packages, in particular core packages, try hard to be as permissible as
> reasonably possible on dependencies.
>
You don't give any details, but the one common place I'm aware of where
narrow support windows are common is when deep learning / CUDA are used.
Many deep learning packages use something like a 3-6 month deprecation
policy, and a 6-12 month support window. And then often depend on 1 or max
2 versions of CUDA and cuDNN. This makes it very challenging to work with
more than one deep learning framework or use them in combination with other
complex packages like Ray. So yes, when you have lots of dependencies like
that, you will have a hard time.
Given you work with deep learning, I'll assume that that's the issue you
are seeing. I don't think we have evidence that a 3 year support window is
too narrow, and we do understand the cost of longer windows. I'll note also
that SPEC 0 says a minimum of 2 years after initial release (so back in
time), but due to backwards compatibility policies the support also goes at
least 1-1.5 years forward in time. So there's a >=3 year window of support
for feature releases of core packages in the scientific Python ecosystem.
Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
