h-vetinari commented on PR #14102: URL: https://github.com/apache/arrow/pull/14102#issuecomment-1245869041
> I think the config on our `tasks.yml` has to be updated to match the new file names, as an example this line: https://github.com/apache/arrow/blob/master/dev/tasks/tasks.yml#L255 that's why the builds failed. That was very much correct thanks! :) > @h-vetinari I've never done it before, could you share, or point me to where I can investigate, the process you followed to update these files? From what I understand this is automated, we should document how to do it for future updates. I am happy to document it if you point me on what needs to be done. Thanks for the PR! Yes of course. This comes from more or less 1:1 from the so-called [feedstock](https://github.com/conda-forge/arrow-cpp-feedstock) on the conda-forge side, where the recipe is maintained, and where all these `.ci_support/*.yaml` files get generated based on the recipe & the conda-forge global [pinning](https://github.com/conda-forge/conda-forge-pinning-feedstock) by our friendly automation bots. Conda-forge is [constantly](https://conda-forge.org/status/#other_migrations) updating builds against new versions (e.g. abseil/grpc/libprotobuf), and once arrow has been successfully built against those packages, the builds are published (by the CI on the feedstock) and the recipe is updated. In this case, there's been a while since the last update, so our infrastructure has changed a bit. Also, the relatively scary-looking changes in `bld-arrow.bat` are mostly just simple alphabetizations (more commit history on the feedstock). I synced the recipe separately from the ci-files, taking care to undo some things that are intentionally different in arrow-CI: <details> ```diff diff --git a/dev/tasks/conda-recipes/arrow-cpp/bld-pyarrow.bat b/dev/tasks/conda-recipes/arrow-cpp/bld-pyarrow.bat index e1073b563..bd20c79ef 100644 --- a/dev/tasks/conda-recipes/arrow-cpp/bld-pyarrow.bat +++ b/dev/tasks/conda-recipes/arrow-cpp/bld-pyarrow.bat @@ -1,16 +1,18 @@ +@echo on pushd "%SRC_DIR%"\python @rem the symlinks for cmake modules don't work here -del cmake_modules\BuildUtils.cmake -del cmake_modules\SetupCxxFlags.cmake -del cmake_modules\CompilerInfo.cmake -del cmake_modules\FindNumPy.cmake -del cmake_modules\FindPythonLibsNew.cmake -copy /Y "%SRC_DIR%\cpp\cmake_modules\BuildUtils.cmake" cmake_modules\ -copy /Y "%SRC_DIR%\cpp\cmake_modules\SetupCxxFlags.cmake" cmake_modules\ -copy /Y "%SRC_DIR%\cpp\cmake_modules\CompilerInfo.cmake" cmake_modules\ -copy /Y "%SRC_DIR%\cpp\cmake_modules\FindNumPy.cmake" cmake_modules\ -copy /Y "%SRC_DIR%\cpp\cmake_modules\FindPythonLibsNew.cmake" cmake_modules\ +@rem NOTE: In contrast to conda-forge, they work here as we clone from git. +@rem del cmake_modules\BuildUtils.cmake +@rem del cmake_modules\SetupCxxFlags.cmake +@rem del cmake_modules\CompilerInfo.cmake +@rem del cmake_modules\FindNumPy.cmake +@rem del cmake_modules\FindPythonLibsNew.cmake +@rem copy /Y "%SRC_DIR%\cpp\cmake_modules\BuildUtils.cmake" cmake_modules\ +@rem copy /Y "%SRC_DIR%\cpp\cmake_modules\SetupCxxFlags.cmake" cmake_modules\ +@rem copy /Y "%SRC_DIR%\cpp\cmake_modules\CompilerInfo.cmake" cmake_modules\ +@rem copy /Y "%SRC_DIR%\cpp\cmake_modules\FindNumPy.cmake" cmake_modules\ +@rem copy /Y "%SRC_DIR%\cpp\cmake_modules\FindPythonLibsNew.cmake" cmake_modules\ SET ARROW_HOME=%LIBRARY_PREFIX% SET SETUPTOOLS_SCM_PRETEND_VERSION=%PKG_VERSION% diff --git a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml index 107b8433a..fd0625296 100644 --- a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml +++ b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml @@ -1,6 +1,7 @@ -{% set version = "9.0.0" %} +# NOTE: In constrast to the conda-forge recipe, ARROW_VERSION is a templated variable here. +{% set version = ARROW_VERSION %} {% set cuda_enabled = cuda_compiler_version != "None" %} -{% set build_ext_version = "3.0.0" %} +{% set build_ext_version = ARROW_VERSION %} {% set build_ext = "cuda" if cuda_enabled else "cpu" %} {% set proc_build_number = "0" %} {% set llvm_version = "14" %} @@ -10,15 +11,10 @@ package: version: {{ version }} source: - url: https://dist.apache.org/repos/dist/release/arrow/arrow-{{ version }}/apache-arrow-{{ version }}.tar.gz - sha256: a9a033f0a3490289998f458680d19579cf07911717ba65afde6cb80070f7a9b5 - patches: - # backport of the following commit for compatibility with VS2019 - # https://github.com/apache/arrow/commit/897c186f475f3dd82c1ab47e5cfb87cb0fed8440 - - patches/0001-ARROW-17433-CI-C-Use-Visual-Studio-2019-on-AppVeyor-.patch + path: ../../../../ build: - number: 6 + number: 0 # for cuda support, building with one version is enough to be compatible with # all later versions, since arrow is only using libcuda, and not libcudart. skip: true # [(win or linux64) and cuda_compiler_version not in ("None", "10.2")] @@ -105,9 +101,7 @@ outputs: - re2 - snappy - thrift-cpp - # https://github.com/apache/arrow/blob/apache-arrow-9.0.0/cpp/cmake_modules/ThirdpartyToolchain.cmake#L2245 - # currently pins to exact xsimd version - - xsimd 8.1.0 + - xsimd - zlib - zstd run: ``` </details> After fixing the paths to the CI-support files in `tasks.yml`, I decided to tackle a todo about the windows CUDA builds, which we also have on the feedstock. The one divergence is that I didn't enable the CUDA builds on aarch64, because these are currently run in emulation (and not cross-compiled), which takes ~6h instead of ~1h. Hope that makes things clearer. 🙃 Though, to be frank, I really only know the conda-forge side of this, so for arrow-specific things, we should definitely ask @xhochy to double-check the work here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
