On Fri, 4 Dec 2020 at 21:11, Uwe L. Korn <m...@uwekorn.com> wrote: > Hello all, > > Today the Karotothek CI turned quite red in > https://github.com/JDASoftwareGroup/kartothek/pull/383 / > https://github.com/JDASoftwareGroup/kartothek/pull/383/checks?check_run_id=1497941813 > as the new NumPy 1.20rc1 was pulled in. It simply broke all pyarrow<->NumPy > interop as now dtypes returned by numpy are actual subclasses not directly > numpy.dtype instances anymore. I reported the issue over at > https://github.com/numpy/numpy/issues/17913. We are running into that as > we build our wheels and conda packages with an older release of NumPy that > has a faulty implementation of PyArray_DescrCheck. > > (a) For upcoming releases, we can either move our minimal supported NumPy > to 1.16.6 or merge the PR over at > https://github.com/apache/arrow/pull/8834 > (b) Existing conda(-forge) packages can get a repodata patch that adds a > numpy<1.20 constraint to them > (c) I'll rebuild the latest but still frequently used pyarrow releases on > conda-forge using numpy 1.16.6 > (d) Old pyarrow wheels (Python<3.8) though won't be easily fixed and > instead will return the confusing "ArrowTypeError: Did not pass numpy.dtype > object" error message. Personally my approach would be here to not do > anything and simply direct users to downgrade NumPy if they run into the > issue. > > In addition to this last item (pip installs), doing a small 2.0.1 bugfix release with this patch would also help a lot I think. It would at least ensure that plain pip installs with latest versions will work (while it doesn't solve it for older pyarrow releases of course, in case people upgrade numpy in an existing environment, or install numpy with pyarrow pinned to an older version).
Does our project governance allow doing a python-only release? (meaning, a release branch where the 2.0.1 tag compared to 2.0.0 tag only includes changes to the python libraries) That would make it less burdensome to resolve part of this situation. > Is anyone objecting to this approach? > > Cheers > Uwe >
