Hello all, Today the Karotothek CI turned quite red in https://github.com/JDASoftwareGroup/kartothek/pull/383 / https://github.com/JDASoftwareGroup/kartothek/pull/383/checks?check_run_id=1497941813 as the new NumPy 1.20rc1 was pulled in. It simply broke all pyarrow<->NumPy interop as now dtypes returned by numpy are actual subclasses not directly numpy.dtype instances anymore. I reported the issue over at https://github.com/numpy/numpy/issues/17913. We are running into that as we build our wheels and conda packages with an older release of NumPy that has a faulty implementation of PyArray_DescrCheck.
(a) For upcoming releases, we can either move our minimal supported NumPy to 1.16.6 or merge the PR over at https://github.com/apache/arrow/pull/8834 (b) Existing conda(-forge) packages can get a repodata patch that adds a numpy<1.20 constraint to them (c) I'll rebuild the latest but still frequently used pyarrow releases on conda-forge using numpy 1.16.6 (d) Old pyarrow wheels (Python<3.8) though won't be easily fixed and instead will return the confusing "ArrowTypeError: Did not pass numpy.dtype object" error message. Personally my approach would be here to not do anything and simply direct users to downgrade NumPy if they run into the issue. Is anyone objecting to this approach? Cheers Uwe
