jonkeane commented on a change in pull request #9579: URL: https://github.com/apache/arrow/pull/9579#discussion_r591973907
########## File path: dev/tasks/r/github.macos-linux.local.yml ########## @@ -0,0 +1,93 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# NOTE: must set "Crossbow" as name to have the badge links working in the +# github comment reports! +name: Crossbow + +on: + push: + branches: + - "*-github-*" + +jobs: + autobrew: + name: "install from local source" + runs-on: {{ '${{ matrix.os }}' }} + strategy: + fail-fast: false + matrix: + os: [macOS-latest, ubuntu-20.04] + + + + steps: + - name: Checkout Arrow + run: | + git clone --no-checkout {{ arrow.remote }} arrow + git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }} + git -C arrow checkout FETCH_HEAD + git -C arrow submodule update --init --recursive + - name: Configure non-autobrew dependencies (macos) + run: | + cd arrow/r + brew install openssl + if: contains(matrix.os, 'macOS') + - name: Configure non-autobrew dependencies (linux) + run: | + cd arrow/r + sudo apt install libcurl4-openssl-dev libssl-dev + if: contains(matrix.os, 'ubuntu') + - uses: r-lib/actions/setup-r@v1 + - name: Install dependencies + run: | + install.packages("remotes") + remotes::install_deps("arrow/r", dependencies = TRUE) + remotes::install_cran(c("rcmdcheck", "sys", "sessioninfo")) + shell: Rscript {0} + - name: Session info + run: | + options(width = 100) + pkgs <- installed.packages()[, "Package"] + sessioninfo::session_info(pkgs, include_base = TRUE) + shell: Rscript {0} + - name: Install + env: + _R_CHECK_CRAN_INCOMING_: false + ARROW_USE_PKG_CONFIG: false + FORCE_TOOLS_LIBS_SCRIPT: true + LIBARROW_MINIMAL: false + TEST_R_WITH_ARROW: TRUE + ARROW_R_DEV: TRUE + run: | + cd arrow/r + # Setting the SDK root is necesary on modern macOSes/SDKs to find standard libraries when building Review comment: This is only necessary when building bzip2 from source, so I suspect you've never built bzip2 from source. But you can experience this if you run: ``` cmake \ -DARROW_DEPENDENCY_SOURCE=BUNDLED \ -DARROW_WITH_BZ2=ON \ .. make install [cmake config output with the crucial bit: -- Building BZip2 from source] Scanning dependencies of target bzip2_ep Scanning dependencies of target jemalloc_ep Scanning dependencies of target xsimd_ep [ 0%] Creating directories for 'bzip2_ep' [ 1%] Creating directories for 'jemalloc_ep' [ 2%] Creating directories for 'xsimd_ep' [ 4%] Performing download step (download, verify and extract) for 'jemalloc_ep' [ 4%] Performing download step (download, verify and extract) for 'bzip2_ep' [ 4%] Performing download step (download, verify and extract) for 'xsimd_ep' -- bzip2_ep download command succeeded. See also /Users/jkeane/repos/arrow/cpp/build/bzip2_ep-prefix/src/bzip2_ep-stamp/bzip2_ep-download-*.log [ 5%] No update step for 'bzip2_ep' [ 5%] No patch step for 'bzip2_ep' -- xsimd_ep download command succeeded. See also /Users/jkeane/repos/arrow/cpp/build/src/xsimd_ep-stamp/xsimd_ep-download-*.log [ 6%] No configure step for 'bzip2_ep' [ 6%] No update step for 'xsimd_ep' [ 7%] Performing build step for 'bzip2_ep' [ 8%] No patch step for 'xsimd_ep' [ 9%] Performing configure step for 'xsimd_ep' CMake Error at /Users/jkeane/repos/arrow/cpp/build/bzip2_ep-prefix/src/bzip2_ep-stamp/bzip2_ep-build-RELEASE.cmake:37 (message): Command failed: 2 '/usr/bin/make' 'libbz2.a' 'CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc' 'CFLAGS= -Qunused-arguments -O3 -DNDEBUG -O3 -DNDEBUG -fPIC' See also /Users/jkeane/repos/arrow/cpp/build/bzip2_ep-prefix/src/bzip2_ep-stamp/bzip2_ep-build-*.log -- stdout output is: If compilation produces errors, or a large number of warnings, please read README.COMPILATION.PROBLEMS -- you might be able to adjust the flags in this Makefile to improve matters. Also in README.COMPILATION.PROBLEMS are some hints that may help if your build produces an executable which is unable to correctly handle so-called 'large files' -- files of size 2GB or more. -- stderr output is: In file included from blocksort.c:22: In file included from huffman.c:22: ./bzlib_private.h:25:10:./bzlib_private.h:25:10 fatal error: 'stdlib.h' file not found : fatal error: 'stdlib.h' file not found #include <stdlib.h> -- jemalloc_ep download command succeeded. See also /Users/jkeane/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-download-*.log ^~~~~~~~~~ #include <stdlib.h> ^~~~~~~~~~ In file included from crctable.c:22: ./bzlib_private.h:25:10: fatal error: 'stdlib.h' file not found #include <stdlib.h> ^~~~~~~~~~ 1 error generated. In file included from compress.c:29: ./bzlib_private.h:25:10: fatal error: 'stdlib.h' file not found In file included from decompress.c:22: ./bzlib_private.h:25:10: fatal error: 'stdlib.h' file not found #include <stdlib.h> ^~~~~~~~~~ In file included from randtable.c:22: ./bzlib_private.h:25:10: fatal error: 'stdlib.h' file not found #include <stdlib.h> ^~~~~~~~~~ 1 error generated. #include <stdlib.h> ^~~~~~~~~~ In file included from bzlib.c:31: ./bzlib_private.h:25:10: fatal error: 'stdlib.h' file not found #include <stdlib.h> ^~~~~~~~~~ 1 error generated. make[3]: *** [huffman.o] Error 1 make[3]: *** Waiting for unfinished jobs.... ``` `ARROW_DEPENDENCY_SOURCE=BUNDLED` which seems to be the operable bit is set [inside of `build_arrow_static.sh`](https://github.com/apache/arrow/blob/master/r/inst/build_arrow_static.sh#L54) which `nixlibs.sh` calls in this configuration. I will add more details to the explanation that this is _only_ necessary for building bzip2 and this particular special case. Alternatively, we could explore changing `ARROW_DEPENDENCY_SOURCE=BUNDLED` in this case so that the build process can find the bzip2 that's on the system already. We already link to the brew-built openssl which isn't 100% bundling. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
