Re: Incompatability of all existing pyarrow releases with the next NumPy release
I believe we can do a release that is just focused on the Python artifacts, yes. On Mon, Dec 7, 2020 at 6:52 AM Joris Van den Bossche wrote: > > On Fri, 4 Dec 2020 at 21:11, Uwe L. Korn wrote: > > > Hello all, > > > > Today the Karotothek CI turned quite red in > > https://github.com/JDASoftwareGroup/kartothek/pull/383 / > > https://github.com/JDASoftwareGroup/kartothek/pull/383/checks?check_run_id=1497941813 > > as the new NumPy 1.20rc1 was pulled in. It simply broke all pyarrow<->NumPy > > interop as now dtypes returned by numpy are actual subclasses not directly > > numpy.dtype instances anymore. I reported the issue over at > > https://github.com/numpy/numpy/issues/17913. We are running into that as > > we build our wheels and conda packages with an older release of NumPy that > > has a faulty implementation of PyArray_DescrCheck. > > > > (a) For upcoming releases, we can either move our minimal supported NumPy > > to 1.16.6 or merge the PR over at > > https://github.com/apache/arrow/pull/8834 > > (b) Existing conda(-forge) packages can get a repodata patch that adds a > > numpy<1.20 constraint to them > > (c) I'll rebuild the latest but still frequently used pyarrow releases on > > conda-forge using numpy 1.16.6 > > (d) Old pyarrow wheels (Python<3.8) though won't be easily fixed and > > instead will return the confusing "ArrowTypeError: Did not pass numpy.dtype > > object" error message. Personally my approach would be here to not do > > anything and simply direct users to downgrade NumPy if they run into the > > issue. > > > > In addition to this last item (pip installs), doing a small 2.0.1 bugfix > release with this patch would also help a lot I think. It would at least > ensure that plain pip installs with latest versions will work (while it > doesn't solve it for older pyarrow releases of course, in case people > upgrade numpy in an existing environment, or install numpy with pyarrow > pinned to an older version). > > Does our project governance allow doing a python-only release? (meaning, a > release branch where the 2.0.1 tag compared to 2.0.0 tag only includes > changes to the python libraries) That would make it less burdensome to > resolve part of this situation. > > > > Is anyone objecting to this approach? > > > > Cheers > > Uwe > >
Demise of Ursabot CI jobs
hi folks -- I just wanted to confirm that the Ursabot CI jobs that went down a few weeks ago won't be coming back, at least not in the Buildbot form factor. The Buildbot master was hosted on a physical machine which suffered some kind of network configuration problem during a Linux update that I don't have the sysadmin skills to resolve, so that machine has been wiped and with it the Buildbot setup. There are some jobs that were being run on these machines (CUDA, large memory tests) that still need to be run. I highly recommend that we move these to physical machines that report in to our free Buildkite org (https://buildkite.com/apache-arrow), that way things will be less brittle and machines can easily come and go without any involvement from ASF Infra (which is why I've been against self-hosted GitHub Actions). Thanks, Wes
Re: Incompatability of all existing pyarrow releases with the next NumPy release
On Fri, 4 Dec 2020 at 21:11, Uwe L. Korn wrote: > Hello all, > > Today the Karotothek CI turned quite red in > https://github.com/JDASoftwareGroup/kartothek/pull/383 / > https://github.com/JDASoftwareGroup/kartothek/pull/383/checks?check_run_id=1497941813 > as the new NumPy 1.20rc1 was pulled in. It simply broke all pyarrow<->NumPy > interop as now dtypes returned by numpy are actual subclasses not directly > numpy.dtype instances anymore. I reported the issue over at > https://github.com/numpy/numpy/issues/17913. We are running into that as > we build our wheels and conda packages with an older release of NumPy that > has a faulty implementation of PyArray_DescrCheck. > > (a) For upcoming releases, we can either move our minimal supported NumPy > to 1.16.6 or merge the PR over at > https://github.com/apache/arrow/pull/8834 > (b) Existing conda(-forge) packages can get a repodata patch that adds a > numpy<1.20 constraint to them > (c) I'll rebuild the latest but still frequently used pyarrow releases on > conda-forge using numpy 1.16.6 > (d) Old pyarrow wheels (Python<3.8) though won't be easily fixed and > instead will return the confusing "ArrowTypeError: Did not pass numpy.dtype > object" error message. Personally my approach would be here to not do > anything and simply direct users to downgrade NumPy if they run into the > issue. > > In addition to this last item (pip installs), doing a small 2.0.1 bugfix release with this patch would also help a lot I think. It would at least ensure that plain pip installs with latest versions will work (while it doesn't solve it for older pyarrow releases of course, in case people upgrade numpy in an existing environment, or install numpy with pyarrow pinned to an older version). Does our project governance allow doing a python-only release? (meaning, a release branch where the 2.0.1 tag compared to 2.0.0 tag only includes changes to the python libraries) That would make it less burdensome to resolve part of this situation. > Is anyone objecting to this approach? > > Cheers > Uwe >
[NIGHTLY] Arrow Build Report for Job nightly-2020-12-07-0
Arrow Build Report for Job nightly-2020-12-07-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0 Failed Tasks: - test-conda-python-3.7-spark-branch-3.0: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-test-conda-python-3.7-spark-branch-3.0 - test-conda-python-3.8-jpype: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-test-conda-python-3.8-jpype - test-debian-10-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-circle-test-debian-10-cpp - test-ubuntu-18.04-docs: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-test-ubuntu-18.04-docs - wheel-win-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-wheel-win-cp36m - wheel-win-cp37m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-wheel-win-cp37m - wheel-win-cp38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-wheel-win-cp38 Succeeded Tasks: - centos-7-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-travis-centos-7-aarch64 - centos-7-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-centos-7-amd64 - centos-8-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-travis-centos-8-aarch64 - centos-8-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-centos-8-amd64 - conda-clean: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-clean - conda-linux-gcc-py36-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-drone-conda-linux-gcc-py36-aarch64 - conda-linux-gcc-py36-cpu-r36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-linux-gcc-py36-cpu-r36 - conda-linux-gcc-py36-cuda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-linux-gcc-py36-cuda - conda-linux-gcc-py37-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-drone-conda-linux-gcc-py37-aarch64 - conda-linux-gcc-py37-cpu-r40: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-linux-gcc-py37-cpu-r40 - conda-linux-gcc-py37-cuda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-linux-gcc-py37-cuda - conda-linux-gcc-py38-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-drone-conda-linux-gcc-py38-aarch64 - conda-linux-gcc-py38-cpu: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-linux-gcc-py38-cpu - conda-linux-gcc-py38-cuda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-linux-gcc-py38-cuda - conda-osx-clang-py36-r36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-osx-clang-py36-r36 - conda-osx-clang-py37-r40: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-osx-clang-py37-r40 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-osx-clang-py38 - conda-win-vs2017-py36-r36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-win-vs2017-py36-r36 - conda-win-vs2017-py37-r40: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-win-vs2017-py37-r40 - conda-win-vs2017-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-azure-conda-win-vs2017-py38 - debian-buster-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-debian-buster-amd64 - debian-buster-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-travis-debian-buster-arm64 - example-cpp-minimal-build-static-system-dependency: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-example-cpp-minimal-build-static-system-dependency - example-cpp-minimal-build-static: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-github-example-cpp-minimal-build-static - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-travis-gandiva-jar-osx - gandiva-jar-xenial: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-travis-gandiva-jar-xenial - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-12-07-0-trav
