Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On 10.12.23 14:06, Rebecca N. Palmer wrote: I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably soon. Given that pandas 2.x is *not* required for Python 3.12 (but is required for Cython 3.0), should we wait for the Python 3.12 transition to be done first? These are broken by pandas 2.x and have a possible (but untested) fix in their bug - please test and apply it: dask(?) dials influxdb-python* python-altair python-feather-format python-upsetplot seaborn tqdm* (* = this package is currently also broken for a non-pandas reason, probably Python 3.12, that I don't have a fix for) These are broken by pandas 2.x and have no known-to-me fix: augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas Some generic things to try are pandas.util.testing -> pandas.testing, .iteritems() -> .items(), and if one exists, a more recent upstream version. Is this an acceptable amount of breakage or should we continue to wait? Bear in mind that if we wait too long, we may be forced into it by some transition further up the stack (e.g. a future Python or numpy) that breaks pandas 1.x. up to the maintainers. But please wait at least until the current pandas and numpy migrated to testing, e.g. that the autopkg tests of pandas and numpy triggered by python3-defaults pass. Is there a way to see the binNMUs which are still stuck in unstable, and don't migrate? Matthias
Re: cython 3.x (for Python 3.12)
On 10.12.23 21:32, Andrey Rakhmatullin wrote: On Sun, Dec 10, 2023 at 09:30:03PM +0100, Andrey Rakhmatullin wrote: I find that there's also a significant issue with relying on cython3-legacy: it conflicts with cython3, meaning that it will be impossible to simultaneously install packages depending on cython3 and cython3-legacy. Once cython 3.x moves from experimental to unstable to testing and packages start depending on it, this will become a significant issue. I assume that the aim will be for everything to be ported to cython 3.x and for cython3-legacy to be dropped from testing before the trixie freeze? I wonder how many packages actually need a runtime dep on cython. I quickly checked three packages from `reverse-depends cython3` and while python3-pysph probably uses cython to generate code at the run time (not sure), python3-pyzoltan seems to only use it at the build time and python3-epimodels doesn't seem to use it at all. (I may be terribly wrong, I just thought cython is almost always a build time only dep and there are around 30 packages depending on it which was surprising) yes, many setup.py's list that under install_requires. I hope we now identified most of them, and fixed these. At least we were now able to do all the binNMUs. You could package a non-conflicting cython-legacy, however that would require more changes, also how to build it. Matthias
Re: pandas 1.5 -> 2.1?
On 2023-12-10 21:55, Kingsley G. Morse Jr. wrote: Hi Rebecca, Julian and all science minded pythonistas of debian, great and small! 3.) The following one-liner suggests 44 debian packages might be affected by the breaks Rebecca said would be caused by pandas 2.x: $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas ; do apt-cache search "$s" ; done | less I'd say don't worry about pymatgen in any case. It's got a new version to upgrade to, and they make regular releases. Drew
pandas 1.5 -> 2.1?
Hi Rebecca, Julian and all science minded pythonistas of debian, great and small! I like your correspondence about upgrading from version 1.5 of pandas to 2.1. It's open, scientific and explores the ideal of proceeding wisely in a matter of public interest. My humble thoughts are: 1.) Rebecca: *Why* did you write that you'd like to move forward with the pandas 1.5 -> 2.1 transition? What's your reason? 2.) What may be the advantage of migrating to version 3.0 of Cython? 3.) The following one-liner suggests 44 debian packages might be affected by the breaks Rebecca said would be caused by pandas 2.x: $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas ; do apt-cache search "$s" ; done | less 4.) The break that worries me the most is sklearn-pandas, because it seems to me that sklearn is popular and fundamental. Comment welcome, Kingsley On 12/10/2023 20:16, Julian Gilbey wrote: > On Sun, Dec 10, 2023 at 01:06:01PM +, Rebecca N. Palmer wrote: > > I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably > > soon. > > > > Given that pandas 2.x is *not* required for Python 3.12 (but is required for > > Cython 3.0), should we wait for the Python 3.12 transition to be done first? > > Well, I have seen at least one package that has an RC bug for the > Python 3.12 transition that might be because it's still using an old > version of cython3 :( So it's a bit of chicken-and-egg - having Cython > 3.0 might be very helpful. But then there is this list of 28 packages > broken by pandas 2.x. On the other hand, these will need fixing at > some point soon anyway, so I'd be in favour of doing the pandas > transition now, which will allow Cython 3.0 to move into unstable. > > Just my 2 cents' worth... > > Best wishes, > >Julian > -- Time is the fire in which we all burn.
Re: cython 3.x (for Python 3.12)
On Sun, Dec 10, 2023 at 09:30:03PM +0100, Andrey Rakhmatullin wrote: > > I find that there's also a significant issue with relying on > > cython3-legacy: it conflicts with cython3, meaning that it will be > > impossible to simultaneously install packages depending on cython3 and > > cython3-legacy. Once cython 3.x moves from experimental to unstable > > to testing and packages start depending on it, this will become a > > significant issue. I assume that the aim will be for everything to be > > ported to cython 3.x and for cython3-legacy to be dropped from testing > > before the trixie freeze? > I wonder how many packages actually need a runtime dep on cython. I > quickly checked three packages from `reverse-depends cython3` and while > python3-pysph probably uses cython to generate code at the run time (not > sure), python3-pyzoltan seems to only use it at the build time and > python3-epimodels doesn't seem to use it at all. (I may be terribly wrong, I just thought cython is almost always a build time only dep and there are around 30 packages depending on it which was surprising)
Re: cython 3.x (for Python 3.12)
On Sun, Dec 10, 2023 at 08:12:40PM +, Julian Gilbey wrote: > On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > > to experimental. > > [...] > > > > So, that's 71 regressions with cython3. dd-list below. Please help us > > port to cython 3. If this isn't possible, Graham is preparing a > > cython-legacy package, to help the stragglers. But we're expecting that > > this won't have great Python 3.12 support... > > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html > > I find that there's also a significant issue with relying on > cython3-legacy: it conflicts with cython3, meaning that it will be > impossible to simultaneously install packages depending on cython3 and > cython3-legacy. Once cython 3.x moves from experimental to unstable > to testing and packages start depending on it, this will become a > significant issue. I assume that the aim will be for everything to be > ported to cython 3.x and for cython3-legacy to be dropped from testing > before the trixie freeze? I wonder how many packages actually need a runtime dep on cython. I quickly checked three packages from `reverse-depends cython3` and while python3-pysph probably uses cython to generate code at the run time (not sure), python3-pyzoltan seems to only use it at the build time and python3-epimodels doesn't seem to use it at all.
Re: Bug#1043240: transition: pandas 1.5 -> 2.1
On Sun, Dec 10, 2023 at 01:06:01PM +, Rebecca N. Palmer wrote: > I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably > soon. > > Given that pandas 2.x is *not* required for Python 3.12 (but is required for > Cython 3.0), should we wait for the Python 3.12 transition to be done first? Well, I have seen at least one package that has an RC bug for the Python 3.12 transition that might be because it's still using an old version of cython3 :( So it's a bit of chicken-and-egg - having Cython 3.0 might be very helpful. But then there is this list of 28 packages broken by pandas 2.x. On the other hand, these will need fixing at some point soon anyway, so I'd be in favour of doing the pandas transition now, which will allow Cython 3.0 to move into unstable. Just my 2 cents' worth... Best wishes, Julian
Re: cython 3.x (for Python 3.12)
On Sat, Nov 25, 2023 at 04:23:46PM +, Stefano Rivera wrote: > As part of preparing for Python 3.12 in Debian, I've uploaded cython 3 > to experimental. > [...] > > So, that's 71 regressions with cython3. dd-list below. Please help us > port to cython 3. If this isn't possible, Graham is preparing a > cython-legacy package, to help the stragglers. But we're expecting that > this won't have great Python 3.12 support... > https://ftp-master.debian.org/new/cython-legacy_0.29.36-1~exp1.html I find that there's also a significant issue with relying on cython3-legacy: it conflicts with cython3, meaning that it will be impossible to simultaneously install packages depending on cython3 and cython3-legacy. Once cython 3.x moves from experimental to unstable to testing and packages start depending on it, this will become a significant issue. I assume that the aim will be for everything to be ported to cython 3.x and for cython3-legacy to be dropped from testing before the trixie freeze? Best wishes, Julian
Bug#1043240: transition: pandas 1.5 -> 2.1
I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably soon. Given that pandas 2.x is *not* required for Python 3.12 (but is required for Cython 3.0), should we wait for the Python 3.12 transition to be done first? These are broken by pandas 2.x and have a possible (but untested) fix in their bug - please test and apply it: dask(?) dials influxdb-python* python-altair python-feather-format python-upsetplot seaborn tqdm* (* = this package is currently also broken for a non-pandas reason, probably Python 3.12, that I don't have a fix for) These are broken by pandas 2.x and have no known-to-me fix: augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas Some generic things to try are pandas.util.testing -> pandas.testing, .iteritems() -> .items(), and if one exists, a more recent upstream version. Is this an acceptable amount of breakage or should we continue to wait? Bear in mind that if we wait too long, we may be forced into it by some transition further up the stack (e.g. a future Python or numpy) that breaks pandas 1.x. Build logs: https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1/+builds?build_text=&build_state=failed https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1n/+builds?build_text=&build_state=failed (The second is more recent, but includes fewer packages.) Autopkgtest logs: https://qa.debian.org/excuses.php?experimental=1&package=pandas (Because of the Python 3.12 transition, this may currently be wrong about what is a regression and what is not.)
