[
https://issues.apache.org/jira/browse/ARROW-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17582836#comment-17582836
]
chendan edited comment on ARROW-17491 at 8/22/22 9:42 AM:
----------------------------------------------------------
[~yibocai]
I follow your instruction and this error disappeared.
Now the error is in linking:
setup start!
– System processor: x86_64
– System processor: arm
set ARROW_CPU_FLAG
– Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE=\{release,debug,...})
– Build Type: RELEASE
ARROW_ARMV8_ARCH:
ARROW_ARMV8_ARCH
– Build output directory:
/root/build/tmp/arrow/python/build/temp.linux-x86_64-3.6/release
CMake Warning (dev) at
/home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:438
(message):
The package name passed to `find_package_handle_standard_args` (PkgConfig)
does not match the name of the calling package (Arrow). This can lead to
problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
/home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPkgConfig.cmake:99
(find_package_handle_standard_args)
cmake_modules/FindArrow.cmake:39 (include)
cmake_modules/FindArrowPython.cmake:46 (find_package)
CMakeLists.txt:219 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.
– Arrow version: 2.0.0 (HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow)
– Arrow SO and ABI version: 200
– Arrow full SO version: 200.0.0
– Found the Arrow core shared library: ARROW_shared_lib-NOTFOUND
– Found the Arrow core import library:
– Found the Arrow core static library: ARROW_static_lib-NOTFOUND
– Found the Arrow Python by HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow
– Found the Arrow Python shared library: ARROW_PYTHON_shared_lib-NOTFOUND
– Found the Arrow Python import library:
– Found the Arrow Python static library: ARROW_PYTHON_static_lib-NOTFOUND
– Parquet version: 1.5.1 (HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow)
– Found the Parquet shared library: PARQUET_shared_lib-NOTFOUND
– Found the Parquet import library:
– Found the Parquet static library: PARQUET_static_lib-NOTFOUND
– Found ArrowDataset:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow/include
(found version "2.0.0")
– Found the Arrow Dataset by HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow
– Found the Arrow Dataset shared library: ARROW_DATASET_shared_lib-NOTFOUND
– Found the Arrow Dataset import library:
– Found the Arrow Dataset static library: ARROW_DATASET_static_lib-NOTFOUND
– Configuring done
– Generating done
– Build files have been written to:
/root/build/tmp/arrow/python/build/temp.linux-x86_64-3.6
– Finished cmake for pyarrow
– Running cmake --build for pyarrow
cmake --build . --config release – -j4
[ 4%] Compiling Cython CXX source for lib...
[ 9%] Compiling Cython CXX source for _fs...
[ 14%] Compiling Cython CXX source for _compute...
[ 19%] Compiling Cython CXX source for _csv...
[ 19%] Built target _csv_pyx
[ 23%] Compiling Cython CXX source for _json...
[ 23%] Built target _compute_pyx
[ 28%] Compiling Cython CXX source for _dataset...
[ 28%] Built target _fs_pyx
[ 33%] Compiling Cython CXX source for _parquet...
[ 33%] Built target _json_pyx
[ 38%] Building CXX object CMakeFiles/_fs.dir/_fs.cpp.o
[ 38%] Built target _parquet_pyx
[ 42%] Building CXX object CMakeFiles/_compute.dir/_compute.cpp.o
[ 42%] Built target _dataset_pyx
[ 47%] Building CXX object CMakeFiles/_csv.dir/_csv.cpp.o
[ 47%] Built target lib_pyx
[ 52%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o
[ 57%] Linking CXX shared module release/_json.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_json.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_json.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 61%] Linking CXX shared module release/_csv.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_csv.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_csv.dir/all] Error 2
[ 66%] Linking CXX shared module
release/_compute.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_compute.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_compute.dir/all] Error 2
[ 71%] Linking CXX shared module release/_fs.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_fs.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_fs.dir/all] Error 2
gmake: *** [all] Error 2
error: command 'cmake' failed with exit status 2
When I pip install pyarrow, of course the _json.cpython-36m-x86_64-linux-gnu.so
and other so files are x86_64 architecture. So must I compile these shared libs
by my arm cross-compiler? How to compile them?
But it seems that _json.cpython-36m-x86_64-linux-gnu.so is the aim lib I need
to build out but not the linked lib. If I have
_json.cpython-36m-armv8a-linux-gnu.so my job is finished. All I need to do is
to make all the lib files in site-packages/pyarrow to be arm format. Did I
understand "building pyarrow" wrongly?
was (Author: JIRAUSER283005):
[~yibocai]
I follow your instruction and this error disappeared.
Now the error is in linking:
setup start!
-- System processor: x86_64
-- System processor: arm
set ARROW_CPU_FLAG
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE=\{release,debug,...})
-- Build Type: RELEASE
ARROW_ARMV8_ARCH:
ARROW_ARMV8_ARCH
-- Build output directory:
/root/build/tmp/arrow/python/build/temp.linux-x86_64-3.6/release
CMake Warning (dev) at
/home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:438
(message):
The package name passed to `find_package_handle_standard_args` (PkgConfig)
does not match the name of the calling package (Arrow). This can lead to
problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
/home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPkgConfig.cmake:99
(find_package_handle_standard_args)
cmake_modules/FindArrow.cmake:39 (include)
cmake_modules/FindArrowPython.cmake:46 (find_package)
CMakeLists.txt:219 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Arrow version: 2.0.0 (HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow)
-- Arrow SO and ABI version: 200
-- Arrow full SO version: 200.0.0
-- Found the Arrow core shared library: ARROW_shared_lib-NOTFOUND
-- Found the Arrow core import library:
-- Found the Arrow core static library: ARROW_static_lib-NOTFOUND
-- Found the Arrow Python by HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow
-- Found the Arrow Python shared library: ARROW_PYTHON_shared_lib-NOTFOUND
-- Found the Arrow Python import library:
-- Found the Arrow Python static library: ARROW_PYTHON_static_lib-NOTFOUND
-- Parquet version: 1.5.1 (HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow)
-- Found the Parquet shared library: PARQUET_shared_lib-NOTFOUND
-- Found the Parquet import library:
-- Found the Parquet static library: PARQUET_static_lib-NOTFOUND
-- Found ArrowDataset:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow/include
(found version "2.0.0")
-- Found the Arrow Dataset by HOME:
/root/anaconda3/envs/pyarrow-dev/lib/python3.6/site-packages/pyarrow
-- Found the Arrow Dataset shared library: ARROW_DATASET_shared_lib-NOTFOUND
-- Found the Arrow Dataset import library:
-- Found the Arrow Dataset static library: ARROW_DATASET_static_lib-NOTFOUND
-- Configuring done
-- Generating done
-- Build files have been written to:
/root/build/tmp/arrow/python/build/temp.linux-x86_64-3.6
-- Finished cmake for pyarrow
-- Running cmake --build for pyarrow
cmake --build . --config release -- -j4
[ 4%] Compiling Cython CXX source for lib...
[ 9%] Compiling Cython CXX source for _fs...
[ 14%] Compiling Cython CXX source for _compute...
[ 19%] Compiling Cython CXX source for _csv...
[ 19%] Built target _csv_pyx
[ 23%] Compiling Cython CXX source for _json...
[ 23%] Built target _compute_pyx
[ 28%] Compiling Cython CXX source for _dataset...
[ 28%] Built target _fs_pyx
[ 33%] Compiling Cython CXX source for _parquet...
[ 33%] Built target _json_pyx
[ 38%] Building CXX object CMakeFiles/_fs.dir/_fs.cpp.o
[ 38%] Built target _parquet_pyx
[ 42%] Building CXX object CMakeFiles/_compute.dir/_compute.cpp.o
[ 42%] Built target _dataset_pyx
[ 47%] Building CXX object CMakeFiles/_csv.dir/_csv.cpp.o
[ 47%] Built target lib_pyx
[ 52%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o
[ 57%] Linking CXX shared module release/_json.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_json.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_json.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 61%] Linking CXX shared module release/_csv.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_csv.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_csv.dir/all] Error 2
[ 66%] Linking CXX shared module
release/_compute.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_compute.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_compute.dir/all] Error 2
[ 71%] Linking CXX shared module release/_fs.cpython-36m-x86_64-linux-gnu.so
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_shared
/opt/aarch64-kedacom-linux/lib/gcc/aarch64-kedacom-linux-gnu/8.3.0/../../../../aarch64-kedacom-linux-gnu/bin/ld:
cannot find -larrow_python_shared
collect2: error: ld returned 1 exit status
gmake[2]: *** [release/_fs.cpython-36m-x86_64-linux-gnu.so] Error 1
gmake[1]: *** [CMakeFiles/_fs.dir/all] Error 2
gmake: *** [all] Error 2
error: command 'cmake' failed with exit status 2
When I pip install pyarrow, of course the _json.cpython-36m-x86_64-linux-gnu.so
and other so files are x86_64 architecture. So must I compile these shared libs
by my arm cross-compiler? How to compile them?
> Build python lib failed for ARMv8
> ---------------------------------
>
> Key: ARROW-17491
> URL: https://issues.apache.org/jira/browse/ARROW-17491
> Project: Apache Arrow
> Issue Type: Task
> Components: Python
> Affects Versions: 2.0.0
> Reporter: chendan
> Priority: Major
>
> I want to build pyarrow lib for my armv8-a with my own arm v8 cross-compiler.
> Althrough arror=2.0.0 is old I need to build it for pyflink and I confirmed
> that 2.0.0 does support armv8 building.
> I follow the steps described in
> [https://arrow.apache.org/docs/developers/python.html#using-conda|https://arrow.apache.org/docs/developers/python.html#using-conda:]
> Firstly , I have build out arrow c++ libs of arm:
> libarrow.a libarrow_python.a
> libarrow_python.so.200.0.0 libarrow.so.200.0.0
> libarrow_bundled_dependencies.a libarrow_python.so libarrow.so
> libparquet.a
> libarrow_dataset.a libarrow_python.so.200 libarrow.so.200
> The make -4j command failed due to some third party arm lib has not been
> built out for linking. But It did not affect the building out of the upper
> libs.
> Then I perform building pyarrow step:
> pushd arrow/python
> export PYARROW_WITH_PARQUET=1
> export PYARROW_WITH_DATASET=1
> export PYARROW_PARALLEL=4 python setup.py build_ext --inplace
> Then an error occured:
> -- System processor: arm
> set ARROW_CPU_FLAG
> -- Arrow build warning level: PRODUCTION
> Using ld linker
> Configured for RELEASE build (set with cmake
> -DCMAKE_BUILD_TYPE=\{release,debug,...})
> -- Build Type: RELEASE
> ARROW_ARMV8_ARCH:
> ARROW_ARMV8_ARCH
> -- Build output directory:
> /root/build/tmp/arrow/python/build/temp.linux-x86_64-3.6/release
> CMake Warning (dev) at
> /home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:438
> (message):
> The package name passed to `find_package_handle_standard_args` (PkgConfig)
> does not match the name of the calling package (Arrow). This can lead to
> problems in calling code that expects `find_package` result variables
> (e.g., `_FOUND`) to follow a certain pattern.
> Call Stack (most recent call first):
>
> /home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPkgConfig.cmake:99
> (find_package_handle_standard_args)
> cmake_modules/FindArrow.cmake:39 (include)
> cmake_modules/FindArrowPython.cmake:46 (find_package)
> CMakeLists.txt:219 (find_package)
> This warning is for project developers. Use -Wno-dev to suppress it.
> -- Could NOT find Arrow (missing: Arrow_DIR)
> -- Checking for module 'arrow'
> -- No package 'arrow' found
> CMake Error at
> /home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230
> (message):
> Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR
> ARROW_FULL_SO_VERSION ARROW_SO_VERSION)
> Call Stack (most recent call first):
>
> /home/anaconda3/envs/pyarrow-dev/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594
> (_FPHSA_FAILURE_MESSAGE)
> cmake_modules/FindArrow.cmake:418 (find_package_handle_standard_args)
> cmake_modules/FindArrowPython.cmake:46 (find_package)
> CMakeLists.txt:219 (find_package)
> -- Configuring incomplete, errors occurred!
> error: command 'cmake' failed with exit status 1
> What's it? It seems that I need to pip install arrow and pyarrow. After the
> installation, the error still occured. How to solve it?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
