This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new d6d8f0329b GH-50774: [CI][Python] Match Protobuf symbol visibility in
bundled Substrait and ORC (#50792)
d6d8f0329b is described below
commit d6d8f0329b66cc14a17aa8e6cb80d062b3db7fb1
Author: tadeja <[email protected]>
AuthorDate: Thu Aug 6 03:15:05 2026 +0200
GH-50774: [CI][Python] Match Protobuf symbol visibility in bundled
Substrait and ORC (#50792)
### Rationale for this change
Fix #50774. After #50650, the Pyodide build failed while loading PyArrow
with ` ImportError: dynamic module does not define module export function
(PyInit_lib)`.
The ImportError is misleading because the `PyInit_lib` is present. The
actual failure is an earlier abort during static initialization of
`libarrow_python.so`, in Protobuf descriptor registration.
#50650 changed the order of libraries merged into
`libarrow_bundled_dependencies.a`. The archive contains duplicate weak
Protobuf/Abseil symbols emitted by the Protobuf runtime and generated code in
Substrait and in ORC. These symbols had inconsistent visibility:
a) Protobuf was compiled with *hidden* visibility,
b) Substrait and ORC generated Protobuf code used default visibility,
so the order change caused `wasm-ld` to select symbols with different
visibility.
(e.g. `google::protobuf::internal::ZeroFieldsBase::Clear()` changed from
hidden in Protobuf's `empty.pb.cc.o` to default visible in Substrait's
`algebra.pb.cc.o`)
The failure showed up in Emscripten, no failures on other platforms, but
matching visibility globally is desirable to prevent any unintended exports and
symbol interposition.
### What changes are included in this PR?
Compile bundled Substrait and ORC with the same hidden visibility for
symbols as Protobuf so not depending on archive order.
Also add debug for Emscripten CI builds `--profiling-funcs`.
### Are these changes tested?
Yes, `test-conda-python-emscripten` passes
(Local repro, debug and fix verification with `ARCH=amd64 archery docker
run conda-python-emscripten`, for symbols `llvm-nm` and `wasm-ld`)
### Are there any user-facing changes?
No.
* GitHub Issue: #50774
Authored-by: Tadeja Kadunc <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
---
ci/scripts/python_build_emscripten.sh | 5 +++++
cpp/cmake_modules/ThirdpartyToolchain.cmake | 12 +++++++++++-
2 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/ci/scripts/python_build_emscripten.sh
b/ci/scripts/python_build_emscripten.sh
index 857c0e2513..061e7df759 100755
--- a/ci/scripts/python_build_emscripten.sh
+++ b/ci/scripts/python_build_emscripten.sh
@@ -38,6 +38,11 @@ cp -aL "${source_dir}" "${python_build_dir}"
# emcmake so we unset them
unset LDFLAGS CFLAGS CXXFLAGS CPPFLAGS
+# Keep WebAssembly function names only in CI to limit wheel size
+if [ "${GITHUB_ACTIONS:-}" = "true" ]; then
+ export PYARROW_CXXFLAGS="${PYARROW_CXXFLAGS:+${PYARROW_CXXFLAGS}
}--profiling-funcs"
+fi
+
pushd "${python_build_dir}"
pyodide build
popd
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake
b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 8506a07ec8..8888f52f72 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -2447,7 +2447,11 @@ macro(build_substrait)
set(SUBSTRAIT_INCLUDES ${SUBSTRAIT_CPP_DIR} ${PROTOBUF_INCLUDE_DIR})
add_library(substrait STATIC ${SUBSTRAIT_SOURCES})
- set_target_properties(substrait PROPERTIES POSITION_INDEPENDENT_CODE ON)
+ # Match Protobuf's visibility because target contains generated Protobuf code
+ set_target_properties(substrait
+ PROPERTIES POSITION_INDEPENDENT_CODE ON
+ CXX_VISIBILITY_PRESET hidden
+ VISIBILITY_INLINES_HIDDEN ON)
target_compile_options(substrait PRIVATE "${SUBSTRAIT_SUPPRESSED_FLAGS}")
target_include_directories(substrait PUBLIC ${SUBSTRAIT_INCLUDES})
target_link_libraries(substrait PUBLIC ${ARROW_PROTOBUF_LIBPROTOBUF})
@@ -3888,6 +3892,10 @@ function(build_orc)
fetchcontent_makeavailable(orc)
+ # ORC compiles generated Protobuf code into its static library
+ set_target_properties(orc PROPERTIES CXX_VISIBILITY_PRESET hidden
+ VISIBILITY_INLINES_HIDDEN ON)
+
# ORC 2.2.1 unconditionally adds /std:c++17 on MSVC via
# add_compile_options, which overrides CMAKE_CXX_STANDARD and causes
# ABI mismatches with protobuf (GlobalEmptyStringConstexpr vs
@@ -3955,7 +3963,9 @@ function(build_orc)
set(ORC_CMAKE_ARGS
${EP_COMMON_CMAKE_ARGS}
"-DCMAKE_CXX_FLAGS=${ORC_CXX_FLAGS}"
+ -DCMAKE_CXX_VISIBILITY_PRESET=hidden
"-DCMAKE_INSTALL_PREFIX=${ORC_PREFIX}"
+ -DCMAKE_VISIBILITY_INLINES_HIDDEN=ON
-DSTOP_BUILD_ON_WARNING=OFF
-DBUILD_LIBHDFSPP=OFF
-DBUILD_JAVA=OFF
