nealrichardson commented on code in PR #55594:
URL: https://github.com/apache/arrow/pull/55594#discussion_r4586551083
##########
ci/scripts/r_windows_build.sh:
##########
@@ -17,76 +17,251 @@
# specific language governing permissions and limitations
# under the License.
+# Build the static libarrow that r/configure.win downloads on Windows.
Review Comment:
AFAICT reading the diff, there is basically nothing shared between the old
version and the new one, so maybe it would be cleaner and more readable to just
have two different scripts.
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -3745,12 +3745,24 @@ function(build_google_cloud_cpp_storage)
TRUE
PARENT_SCOPE)
- # Workaround missing BCRYPT_RSA_ALG_HANDLE macro in older MinGW-w64 headers.
- # google-cloud-cpp v3+ uses it without guards in sign_using_sha256.cc.
+ # Workaround missing BCRYPT_RSA_ALG_HANDLE and BCRYPT_MD5_ALG_HANDLE macros
+ # in older MinGW-w64 headers. google-cloud-cpp v3+ uses them without guards.
Review Comment:
Would turning off google-cloud-cpp in the older Windows builds simplify this?
##########
.github/workflows/r.yml:
##########
@@ -205,14 +205,20 @@ jobs:
fail-fast: false
matrix:
config:
- - { rtools: 40, arch: 'ucrt64' }
+ - { rtools: 45, arch: 'ucrt64' }
Review Comment:
Forgive my ignorance, but if we only build rtools45 here, where does the
rtools40 get built?
##########
ci/scripts/r_windows_build.sh:
##########
@@ -17,76 +17,251 @@
# specific language governing permissions and limitations
# under the License.
+# Build the static libarrow that r/configure.win downloads on Windows.
+#
+# * Rtools40 (MSYS2): makepkg-mingw with ci/scripts/PKGBUILD.
+# * Rtools42+ (MXE, no pacman/makepkg): call CMake directly against the
+# toolchain's static libraries.
+#
+# The two zips differ in contents; r/configure.win handles both.
+
set -ex
: "${ARROW_HOME:=$(pwd)}"
# Make sure it is absolute and exported
ARROW_HOME="$(cd "${ARROW_HOME}" && pwd)"
export ARROW_HOME
-pacman --noconfirm -Syy
+: "${RTOOLS_VERSION:=45}"
-: "${MINGW_ARCH:=mingw32 mingw64 ucrt64}"
-export MINGW_ARCH
+if [ "${RTOOLS_VERSION}" = "40" ]; then
+ pacman --noconfirm -Syy
-cp "${ARROW_HOME}/ci/scripts/PKGBUILD" .
-printenv
-makepkg-mingw --noconfirm --noprogressbar --skippgpcheck --nocheck --syncdeps
--cleanbuild
+ : "${MINGW_ARCH:=mingw32 mingw64 ucrt64}"
+ export MINGW_ARCH
-VERSION="$(grep Version "${ARROW_HOME}/r/DESCRIPTION" | cut -d " " -f 2)"
-DST_DIR="r-libarrow-windows-x86_64-$VERSION"
-
-# Collect the build artifacts and make the shape of zip file that rwinlib
expects
-ls
-mkdir -p build
-mv mingw* build
-cd build
-
-# This may vary by system/CI provider
-MSYS_LIB_DIR="/c/rtools${RTOOLS_VERSION}"
-
-# Untar the builds we made
-find . -maxdepth 1 -type f -name '*.xz' -print0 | xargs -0 -n1 tar -xJf
-mkdir -p "${DST_DIR}"
-# Grab the headers from one, either one is fine
-# (if we're building twice to combine old and new toolchains, this may already
exist)
-if [ ! -d "${DST_DIR}/include" ]; then
- mv "$(echo "${MINGW_ARCH}" | cut -d ' ' -f 1)/include" "${DST_DIR}"
-fi
+ cp "${ARROW_HOME}/ci/scripts/PKGBUILD" .
+ printenv
+ makepkg-mingw --noconfirm --noprogressbar --skippgpcheck --nocheck
--syncdeps --cleanbuild
-# mingw64 -> x64
-# mingw32 -> i386
-# ucrt64 -> x64-ucrt
-
-if [ -d mingw64/lib/ ]; then
- ls "${MSYS_LIB_DIR}/mingw64/lib/"
- # Make the rest of the directory structure
- mkdir -p "${DST_DIR}/lib/x64"
- # Move the 64-bit versions of libarrow into the expected location
- mv mingw64/lib/*.a "${DST_DIR}/lib/x64"
- # These are from https://dl.bintray.com/rtools/mingw{32,64}/
- cp
"${MSYS_LIB_DIR}"/mingw64/lib/lib{snappy,zstd,lz4,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a
"${DST_DIR}/lib/x64"
-fi
+ VERSION="$(grep Version "${ARROW_HOME}/r/DESCRIPTION" | cut -d " " -f 2)"
+ DST_DIR="r-libarrow-windows-x86_64-$VERSION"
+
+ # Collect the build artifacts and make the shape of zip file that rwinlib
expects
+ ls
+ mkdir -p build
+ mv mingw* build
+ cd build
+
+ # This may vary by system/CI provider
+ MSYS_LIB_DIR="/c/rtools${RTOOLS_VERSION}"
+
+ # Untar the builds we made
+ find . -maxdepth 1 -type f -name '*.xz' -print0 | xargs -0 -n1 tar -xJf
+ mkdir -p "${DST_DIR}"
+ # Grab the headers from one, either one is fine
+ # (if we're building twice to combine old and new toolchains, this may
already exist)
+ if [ ! -d "${DST_DIR}/include" ]; then
+ mv "$(echo "${MINGW_ARCH}" | cut -d ' ' -f 1)/include" "${DST_DIR}"
+ fi
+
+ # mingw64 -> x64
+ # mingw32 -> i386
+ # ucrt64 -> x64-ucrt
+
+ if [ -d mingw64/lib/ ]; then
+ ls "${MSYS_LIB_DIR}/mingw64/lib/"
+ # Make the rest of the directory structure
+ mkdir -p "${DST_DIR}/lib/x64"
+ # Move the 64-bit versions of libarrow into the expected location
+ mv mingw64/lib/*.a "${DST_DIR}/lib/x64"
+ # These are from https://dl.bintray.com/rtools/mingw{32,64}/
+ cp
"${MSYS_LIB_DIR}"/mingw64/lib/lib{snappy,zstd,lz4,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a
"${DST_DIR}/lib/x64"
+ fi
+
+ # Same for the 32-bit versions
+ if [ -d mingw32/lib/ ]; then
+ ls "${MSYS_LIB_DIR}/mingw32/lib/"
+ mkdir -p "${DST_DIR}/lib/i386"
+ mv mingw32/lib/*.a "${DST_DIR}/lib/i386"
+ cp
"${MSYS_LIB_DIR}"/mingw32/lib/lib{snappy,zstd,lz4,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a
"${DST_DIR}/lib/i386"
+ fi
+
+ # Do the same also for ucrt64
+ if [ -d ucrt64/lib/ ]; then
+ ls "${MSYS_LIB_DIR}/ucrt64/lib/"
+ mkdir -p "${DST_DIR}/lib/x64-ucrt"
+ mv ucrt64/lib/*.a "${DST_DIR}/lib/x64-ucrt"
+ cp
"${MSYS_LIB_DIR}"/ucrt64/lib/lib{snappy,zstd,lz4,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a
"$DST_DIR"/lib/x64-ucrt
+ fi
+
+ # Create build artifact
+ zip -r "${DST_DIR}.zip" "$DST_DIR"
-# Same for the 32-bit versions
-if [ -d mingw32/lib/ ]; then
- ls "${MSYS_LIB_DIR}/mingw32/lib/"
- mkdir -p "${DST_DIR}/lib/i386"
- mv mingw32/lib/*.a "${DST_DIR}/lib/i386"
- cp
"${MSYS_LIB_DIR}"/mingw32/lib/lib{snappy,zstd,lz4,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a
"${DST_DIR}/lib/i386"
+ # Copy that to a file name/path that does not vary by version number so we
+ # can easily find it in the R package tests on CI
+ cp "${DST_DIR}.zip" ../libarrow.zip
+
+ exit 0
fi
-# Do the same also for ucrt64
-if [ -d ucrt64/lib/ ]; then
- ls "${MSYS_LIB_DIR}/ucrt64/lib/"
- mkdir -p "${DST_DIR}/lib/x64-ucrt"
- mv ucrt64/lib/*.a "${DST_DIR}/lib/x64-ucrt"
- cp
"${MSYS_LIB_DIR}"/ucrt64/lib/lib{snappy,zstd,lz4,bz2,crypto,curl,ss*,utf8proc,re2,nghttp2}.a
"$DST_DIR"/lib/x64-ucrt
+: "${RTOOLS_HOME:=/c/rtools${RTOOLS_VERSION}}"
+TOOLCHAIN="${RTOOLS_HOME}/x86_64-w64-mingw32.static.posix"
+export PATH="${TOOLCHAIN}/bin:${RTOOLS_HOME}/usr/bin:${PATH}"
+
+# CMake is a native Windows program, so give it Windows-style paths
+TOOLCHAIN_WIN="$(cygpath -am "${TOOLCHAIN}")"
+BUILD_DIR="$(pwd)/build"
+CPP_BUILD_DIR="${BUILD_DIR}/cpp"
+INSTALL_DIR="${BUILD_DIR}/install"
+INSTALL_DIR_WIN="$(cygpath -am "${INSTALL_DIR}")"
+
+# Prefer the runner's CMake: the one in older Rtools can be too old for Arrow
+CMAKE="cmake"
+if [ -x "${PROGRAMFILES}/CMake/bin/cmake.exe" ]; then
+ CMAKE="${PROGRAMFILES}/CMake/bin/cmake.exe"
fi
+printenv
+gcc --version
+g++ --version
+"${CMAKE}" --version
+make --version | head -n 1
+ls "${TOOLCHAIN}/lib"
+cat "${TOOLCHAIN}/lib/pkgconfig/libcurl.pc"
+
+# google-cloud-cpp links static cURL. CXXFLAGS, not ARROW_CXXFLAGS, because
+# ARROW_CXXFLAGS aren't passed to ExternalProjects.
+export CXXFLAGS="${CXXFLAGS:-} -DCURL_STATICLIB"
+
+rm -rf "${CPP_BUILD_DIR}"
+mkdir -p "${CPP_BUILD_DIR}" "${INSTALL_DIR}"
+pushd "${CPP_BUILD_DIR}"
+
+# * CMAKE_UNITY_BUILD=OFF: otherwise some compute tests segfault.
+# * Bundled because Rtools45 doesn't ship them: AWS SDK, Thrift,
+# google-cloud-cpp, snappy, utf8proc.
+# * re2 bundled: the Rtools45 re2 would pull abseil onto the R link line.
+# * abseil, protobuf bundled: the Rtools45 versions are too old for the
+# bundled google-cloud-cpp.
+# * Brotli bundled: Rtools45 has 1.2.0, without the 1.2.1 security fix
+# (GH-55165). cURL's Brotli symbols resolve from the bundled copy too.
+# * ARROW_USE_STD_CHRONO=0: the R package supplies the time zone database
+# through tzdb, which only the vendored date library uses (GH-48593).
+"${CMAKE}" \
+ "$(cygpath -am "${ARROW_HOME}/cpp")" \
+ -G "MSYS Makefiles" \
+ -DARROW_ACERO=ON \
+ -DARROW_AZURE=OFF \
+ -DARROW_BUILD_SHARED=OFF \
+ -DARROW_BUILD_STATIC=ON \
+ -DARROW_BUILD_UTILITIES=OFF \
+ -DARROW_COMPUTE=ON \
+ -DARROW_CXXFLAGS="-DARROW_USE_STD_CHRONO=0" \
+ -DARROW_CSV=ON \
+ -DARROW_DATASET=ON \
+ -DARROW_FILESYSTEM=ON \
+ -DARROW_GCS=ON \
+ -DARROW_HDFS=OFF \
+ -DARROW_JEMALLOC=OFF \
+ -DARROW_JSON=ON \
+ -DARROW_LZ4_USE_SHARED=OFF \
+ -DARROW_MIMALLOC=ON \
+ -DARROW_PACKAGE_PREFIX="${TOOLCHAIN_WIN}" \
+ -DARROW_PARQUET=ON \
+ -DARROW_S3=ON \
+ -DARROW_SNAPPY_USE_SHARED=OFF \
+ -DARROW_USE_GLOG=OFF \
+ -DARROW_UTF8PROC_USE_SHARED=OFF \
+ -DARROW_VERBOSE_THIRDPARTY_BUILD=ON \
+ -DARROW_WITH_BROTLI=ON \
+ -DARROW_WITH_BZ2=ON \
+ -DARROW_WITH_LZ4=ON \
+ -DARROW_WITH_RE2=ON \
+ -DARROW_WITH_SNAPPY=ON \
+ -DARROW_WITH_ZLIB=ON \
+ -DARROW_WITH_ZSTD=ON \
+ -DARROW_ZSTD_USE_SHARED=OFF \
+ -DAWSSDK_SOURCE=BUNDLED \
+ -DBrotli_SOURCE=BUNDLED \
+ -DCMAKE_BUILD_TYPE=release \
+ -DCMAKE_DISABLE_PRECOMPILE_HEADERS=ON \
+ -DCMAKE_INSTALL_PREFIX="${INSTALL_DIR_WIN}" \
+ -DCMAKE_PREFIX_PATH="${TOOLCHAIN_WIN}" \
+ -DCMAKE_UNITY_BUILD=OFF \
+ -DCMAKE_VERBOSE_MAKEFILE=ON \
+ -DProtobuf_SOURCE=BUNDLED \
+ -DSnappy_SOURCE=BUNDLED \
+ -DThrift_SOURCE=BUNDLED \
+ -Dabsl_SOURCE=BUNDLED \
+ -Dgoogle_cloud_cpp_storage_SOURCE=BUNDLED \
+ -Dre2_SOURCE=BUNDLED \
+ -Dutf8proc_SOURCE=BUNDLED
+
+make -j"${NPROC:-$(nproc)}"
+make install
+popd
+
+ccache --show-stats || echo "ccache not found"
+
+VERSION="$(grep Version "${ARROW_HOME}/r/DESCRIPTION" | cut -d " " -f 2)"
+DST_DIR="r-libarrow-windows-x86_64-${VERSION}"
+
+# Make the zip layout r/configure.win expects: include/ and lib/x64-ucrt/
+pushd "${BUILD_DIR}"
+rm -rf "${DST_DIR}"
+mkdir -p "${DST_DIR}/lib/x64-ucrt"
+cp -r install/include "${DST_DIR}/include"
+cp install/lib/*.a "${DST_DIR}/lib/x64-ucrt/"
+
+# Rtools libraries in PKG_LIBS in r/configure.win; keep in sync
+for lib in z zstd lz4 bz2 crypto ssl curl; do
+ cp "${TOOLCHAIN}/lib/lib${lib}.a" "${DST_DIR}/lib/x64-ucrt/"
+done
+
+# cURL's link dependencies vary by Rtools version, so read them from
+# libcurl.pc, ship the toolchain ones, and record the flags in libcurl.libs
+# for r/configure.win. Static link order matters, so:
Review Comment:
IDK the answer to this but I wonder whether this logic belongs better in
configure.win
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]