I'm trying to get pyarrow built from source after making some changes to
current master. I've successfully built the C++ libraries, but `python
setup.py build_ext`, and all variations thereof that I can try are failing,
unable to find the arrow libs, even though the ARROW_HOME environment
variable is set properly.
It appears that the way cmake is being spawned is a problem, since I can
copy/paste the cmake command reported by the setup script into the console
and it runs just fine. The build gives similar problems, but again
succeeds when run manually. Obviously, more is going on in the setup
script than just the cmake invocations, so I need to get past this problem
so that the build can complete.
FWIW, I'm running this from a virtualenv.
A transcript of the session:
(arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py build_ext
--inplace
running build_ext
-- Running cmake for pyarrow
cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include
-DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib
-DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include
-DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib
-DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python
-DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on
-DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off
-DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off
-DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off
-DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off
-DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on
-DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release
/home/bubba/work/arrow/python
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory:
/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python
-- Found the Arrow Python by
-- Found the Arrow Python shared library:
-- Found the Arrow Python import library:
-- Found the Arrow Python static library:
-- Found the Arrow Flight by
-- Found the Arrow Flight shared library:
-- Found the Arrow Flight import library:
-- Found the Arrow Flight static library:
CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52
(arrow_find_package):
Unknown CMake command "arrow_find_package".
Call Stack (most recent call first):
CMakeLists.txt:453 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log".
See also
"/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log".
error: command 'cmake' failed with exit status 1
(arrow-dev) bubba@localhost ~/work/arrow/python $ cmake
-DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include
-DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib
-DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include
-DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib
-DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python
-DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on
-DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off
-DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off
-DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off
-DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off
-DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on
-DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release
/home/bubba/work/arrow/python
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CXX_SUPPORTS_SSE4_2
-- Performing Test CXX_SUPPORTS_SSE4_2 - Success
-- Performing Test CXX_SUPPORTS_ALTIVEC
-- Performing Test CXX_SUPPORTS_ALTIVEC - Failed
-- Performing Test CXX_SUPPORTS_ARMCRC
-- Performing Test CXX_SUPPORTS_ARMCRC - Failed
-- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO
-- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory: /home/bubba/work/arrow/python/build/release
-- Found PythonInterp: /home/bubba/.python_envs/arrow-dev/bin/python (found
version "3.6.5")
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib PYTHON_LIBRARY-NOTFOUND
-- Found PythonLibs: PYTHON_LIBRARY-NOTFOUND
-- Found NumPy: version "1.17.3"
/home/bubba/.python_envs/arrow-dev/lib/python3.6/site-packages/numpy/core/include
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib PYTHON_LIBRARY-NOTFOUND
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.5.4")
-- Found Arrow: /home/bubba/work/arrow/dist/include (found version "1.0.0")
-- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist)
-- Arrow SO and ABI version: 100
-- Arrow full SO version: 100.0.0
-- Found the Arrow core shared library:
/home/bubba/work/arrow/dist/lib/libarrow.so
-- Found the Arrow core import library:
/home/bubba/work/arrow/dist/lib/libarrow.so
-- Found the Arrow core static library:
/home/bubba/work/arrow/dist/lib/libarrow.a
-- Found ArrowPython: /home/bubba/work/arrow/dist/include (found version
"1.0.0")
-- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist
-- Found the Arrow Python shared library:
/home/bubba/work/arrow/dist/lib/libarrow_python.so
-- Found the Arrow Python import library:
/home/bubba/work/arrow/dist/lib/libarrow_python.so
-- Found the Arrow Python static library:
/home/bubba/work/arrow/dist/lib/libarrow_python.a
-- Found ArrowFlight: /home/bubba/work/arrow/dist/include (found version
"1.0.0")
-- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist
-- Found the Arrow Flight shared library:
/home/bubba/work/arrow/dist/lib/libarrow_flight.so
-- Found the Arrow Flight import library:
/home/bubba/work/arrow/dist/lib/libarrow_flight.so
-- Found the Arrow Flight static library:
/home/bubba/work/arrow/dist/lib/libarrow_flight.a
-- Found ArrowPythonFlight: /home/bubba/work/arrow/dist/include (found
version "1.0.0")
-- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist
-- Found the Arrow Python Flight shared library:
/home/bubba/work/arrow/dist/lib/libarrow_python_flight.so
-- Found the Arrow Python Flight import library:
/home/bubba/work/arrow/dist/lib/libarrow_python_flight.so
-- Found the Arrow Python Flight static library:
/home/bubba/work/arrow/dist/lib/libarrow_python_flight.a
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bubba/work/arrow/python
(arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py build_ext
--inplace
running build_ext
-- Running cmake for pyarrow
cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include
-DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib
-DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include
-DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib
-DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python
-DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on
-DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off
-DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off
-DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off
-DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off
-DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on
-DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release
/home/bubba/work/arrow/python
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory: /home/bubba/work/arrow/python/build/release
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib PYTHON_LIBRARY-NOTFOUND
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib PYTHON_LIBRARY-NOTFOUND
-- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist)
-- Arrow SO and ABI version: 100
-- Arrow full SO version: 100.0.0
-- Found the Arrow core shared library:
/home/bubba/work/arrow/dist/lib/libarrow.so
-- Found the Arrow core import library:
/home/bubba/work/arrow/dist/lib/libarrow.so
-- Found the Arrow core static library:
/home/bubba/work/arrow/dist/lib/libarrow.a
-- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist
-- Found the Arrow Python shared library:
/home/bubba/work/arrow/dist/lib/libarrow_python.so
-- Found the Arrow Python import library:
/home/bubba/work/arrow/dist/lib/libarrow_python.so
-- Found the Arrow Python static library:
/home/bubba/work/arrow/dist/lib/libarrow_python.a
-- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist
-- Found the Arrow Flight shared library:
/home/bubba/work/arrow/dist/lib/libarrow_flight.so
-- Found the Arrow Flight import library:
/home/bubba/work/arrow/dist/lib/libarrow_flight.so
-- Found the Arrow Flight static library:
/home/bubba/work/arrow/dist/lib/libarrow_flight.a
-- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist
-- Found the Arrow Python Flight shared library:
/home/bubba/work/arrow/dist/lib/libarrow_python_flight.so
-- Found the Arrow Python Flight import library:
/home/bubba/work/arrow/dist/lib/libarrow_python_flight.so
-- Found the Arrow Python Flight static library:
/home/bubba/work/arrow/dist/lib/libarrow_python_flight.a
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bubba/work/arrow/python
-- Finished cmake for pyarrow
-- Running cmake --build for pyarrow
cmake --build . --config release --
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory:
/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python
-- Searching for Python libs in
/home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
-- Looking for python3.6m
-- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python
-- Found the Arrow Python by
-- Found the Arrow Python shared library:
-- Found the Arrow Python import library:
-- Found the Arrow Python static library:
-- Found the Arrow Flight by
-- Found the Arrow Flight shared library:
-- Found the Arrow Flight import library:
-- Found the Arrow Flight static library:
CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52
(arrow_find_package):
Unknown CMake command "arrow_find_package".
Call Stack (most recent call first):
CMakeLists.txt:453 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log".
See also
"/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log".
gmake: *** [Makefile:538: cmake_check_build_system] Error 1
error: command 'cmake' failed with exit status 2
(arrow-dev) bubba@localhost ~/work/arrow/python $ cmake --build . --config
release --
Scanning dependencies of target _flight_pyx
[ 6%] Compiling Cython CXX source for _flight...
[ 6%] Built target _flight_pyx
Scanning dependencies of target _flight
[ 13%] Building CXX object CMakeFiles/_flight.dir/_flight.cpp.o
[ 20%] Linking CXX shared module build/release/_
flight.cpython-36m-x86_64-linux-gnu.so
[ 20%] Built target _flight
Scanning dependencies of target _json_pyx
[ 26%] Compiling Cython CXX source for _json...
[ 26%] Built target _json_pyx
Scanning dependencies of target _json
[ 33%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o
[ 40%] Linking CXX shared module build/release/_
json.cpython-36m-x86_64-linux-gnu.so
[ 40%] Built target _json
Scanning dependencies of target lib_pyx
[ 46%] Compiling Cython CXX source for lib...
[ 46%] Built target lib_pyx
Scanning dependencies of target lib
[ 53%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
/home/bubba/work/arrow/python/lib.cpp: In function ‘PyObject*
__pyx_pf_7pyarrow_3lib_92union(PyObject*, PyObject*, PyObject*, PyObject*)’:
/home/bubba/work/arrow/python/lib.cpp:37358:29: warning: comparison of
integer expressions of different signedness: ‘Py_ssize_t’ {aka ‘long int’}
and ‘std::vector<std::shared_ptr<arrow::Field> >::size_type’ {aka ‘long
unsigned int’} [-Wsign-compare]
__pyx_t_2 = ((__pyx_t_6 != __pyx_v_c_fields.size()) != 0);
~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
[ 60%] Linking CXX shared module build/release/
lib.cpython-36m-x86_64-linux-gnu.so
[ 60%] Built target lib
Scanning dependencies of target _fs_pyx
[ 66%] Compiling Cython CXX source for _fs...
[ 66%] Built target _fs_pyx
Scanning dependencies of target _fs
[ 73%] Building CXX object CMakeFiles/_fs.dir/_fs.cpp.o
[ 80%] Linking CXX shared module build/release/_
fs.cpython-36m-x86_64-linux-gnu.so
[ 80%] Built target _fs
Scanning dependencies of target _csv_pyx
[ 86%] Compiling Cython CXX source for _csv...
[ 86%] Built target _csv_pyx
Scanning dependencies of target _csv
[ 93%] Building CXX object CMakeFiles/_csv.dir/_csv.cpp.o
[100%] Linking CXX shared module build/release/_
csv.cpython-36m-x86_64-linux-gnu.so
[100%] Built target _csv
