I'm not sure exactly what is going on but somehow, FindArrow.cmake [1] doesn't seem to be getting imported when the command is getting run in setup.py, I'm not why this would be different then running the same command by hand.
[1] https://github.com/apache/arrow/blob/b282838d3ffea8285a06addab202014825b21993/cpp/cmake_modules/FindArrow.cmake On Wed, Dec 4, 2019 at 10:50 AM Justin Polchlopek <jpolchlo...@azavea.com> wrote: > I'm trying to get pyarrow built from source after making some changes to > current master. I've successfully built the C++ libraries, but `python > setup.py build_ext`, and all variations thereof that I can try are failing, > unable to find the arrow libs, even though the ARROW_HOME environment > variable is set properly. > > It appears that the way cmake is being spawned is a problem, since I can > copy/paste the cmake command reported by the setup script into the console > and it runs just fine. The build gives similar problems, but again > succeeds when run manually. Obviously, more is going on in the setup > script than just the cmake invocations, so I need to get past this problem > so that the build can complete. > > FWIW, I'm running this from a virtualenv. > > A transcript of the session: > > (arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py > build_ext > --inplace > running build_ext > -- Running cmake for pyarrow > cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release > /home/bubba/work/arrow/python > -- Arrow build warning level: PRODUCTION > Using ld linker > Configured for RELEASE build (set with cmake > -DCMAKE_BUILD_TYPE={release,debug,...}) > -- Build Type: RELEASE > -- Build output directory: > /home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > -- Found the Arrow Python by > -- Found the Arrow Python shared library: > -- Found the Arrow Python import library: > -- Found the Arrow Python static library: > -- Found the Arrow Flight by > -- Found the Arrow Flight shared library: > -- Found the Arrow Flight import library: > -- Found the Arrow Flight static library: > CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52 > (arrow_find_package): > Unknown CMake command "arrow_find_package". > Call Stack (most recent call first): > CMakeLists.txt:453 (find_package) > > > -- Configuring incomplete, errors occurred! > See also > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log". > See also > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log". > error: command 'cmake' failed with exit status 1 > (arrow-dev) bubba@localhost ~/work/arrow/python $ cmake > -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release > /home/bubba/work/arrow/python > -- The C compiler identification is GNU 8.3.0 > -- The CXX compiler identification is GNU 8.3.0 > -- Check for working C compiler: /usr/bin/cc > -- Check for working C compiler: /usr/bin/cc -- works > -- Detecting C compiler ABI info > -- Detecting C compiler ABI info - done > -- Detecting C compile features > -- Detecting C compile features - done > -- Check for working CXX compiler: /usr/bin/c++ > -- Check for working CXX compiler: /usr/bin/c++ -- works > -- Detecting CXX compiler ABI info > -- Detecting CXX compiler ABI info - done > -- Detecting CXX compile features > -- Detecting CXX compile features - done > -- Performing Test CXX_SUPPORTS_SSE4_2 > -- Performing Test CXX_SUPPORTS_SSE4_2 - Success > -- Performing Test CXX_SUPPORTS_ALTIVEC > -- Performing Test CXX_SUPPORTS_ALTIVEC - Failed > -- Performing Test CXX_SUPPORTS_ARMCRC > -- Performing Test CXX_SUPPORTS_ARMCRC - Failed > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed > -- Arrow build warning level: PRODUCTION > Using ld linker > Configured for RELEASE build (set with cmake > -DCMAKE_BUILD_TYPE={release,debug,...}) > -- Build Type: RELEASE > -- Build output directory: /home/bubba/work/arrow/python/build/release > -- Found PythonInterp: /home/bubba/.python_envs/arrow-dev/bin/python (found > version "3.6.5") > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > -- Found PythonLibs: PYTHON_LIBRARY-NOTFOUND > -- Found NumPy: version "1.17.3" > > /home/bubba/.python_envs/arrow-dev/lib/python3.6/site-packages/numpy/core/include > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > -- Found PkgConfig: /usr/bin/pkg-config (found version "1.5.4") > -- Found Arrow: /home/bubba/work/arrow/dist/include (found version "1.0.0") > -- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist) > -- Arrow SO and ABI version: 100 > -- Arrow full SO version: 100.0.0 > -- Found the Arrow core shared library: > /home/bubba/work/arrow/dist/lib/libarrow.so > -- Found the Arrow core import library: > /home/bubba/work/arrow/dist/lib/libarrow.so > -- Found the Arrow core static library: > /home/bubba/work/arrow/dist/lib/libarrow.a > -- Found ArrowPython: /home/bubba/work/arrow/dist/include (found version > "1.0.0") > -- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist > -- Found the Arrow Python shared library: > /home/bubba/work/arrow/dist/lib/libarrow_python.so > -- Found the Arrow Python import library: > /home/bubba/work/arrow/dist/lib/libarrow_python.so > -- Found the Arrow Python static library: > /home/bubba/work/arrow/dist/lib/libarrow_python.a > -- Found ArrowFlight: /home/bubba/work/arrow/dist/include (found version > "1.0.0") > -- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist > -- Found the Arrow Flight shared library: > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > -- Found the Arrow Flight import library: > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > -- Found the Arrow Flight static library: > /home/bubba/work/arrow/dist/lib/libarrow_flight.a > -- Found ArrowPythonFlight: /home/bubba/work/arrow/dist/include (found > version "1.0.0") > -- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist > -- Found the Arrow Python Flight shared library: > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > -- Found the Arrow Python Flight import library: > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > -- Found the Arrow Python Flight static library: > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.a > -- Configuring done > -- Generating done > -- Build files have been written to: /home/bubba/work/arrow/python > (arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py > build_ext > --inplace > running build_ext > -- Running cmake for pyarrow > cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release > /home/bubba/work/arrow/python > -- Arrow build warning level: PRODUCTION > Using ld linker > Configured for RELEASE build (set with cmake > -DCMAKE_BUILD_TYPE={release,debug,...}) > -- Build Type: RELEASE > -- Build output directory: /home/bubba/work/arrow/python/build/release > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > -- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist) > -- Arrow SO and ABI version: 100 > -- Arrow full SO version: 100.0.0 > -- Found the Arrow core shared library: > /home/bubba/work/arrow/dist/lib/libarrow.so > -- Found the Arrow core import library: > /home/bubba/work/arrow/dist/lib/libarrow.so > -- Found the Arrow core static library: > /home/bubba/work/arrow/dist/lib/libarrow.a > -- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist > -- Found the Arrow Python shared library: > /home/bubba/work/arrow/dist/lib/libarrow_python.so > -- Found the Arrow Python import library: > /home/bubba/work/arrow/dist/lib/libarrow_python.so > -- Found the Arrow Python static library: > /home/bubba/work/arrow/dist/lib/libarrow_python.a > -- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist > -- Found the Arrow Flight shared library: > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > -- Found the Arrow Flight import library: > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > -- Found the Arrow Flight static library: > /home/bubba/work/arrow/dist/lib/libarrow_flight.a > -- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist > -- Found the Arrow Python Flight shared library: > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > -- Found the Arrow Python Flight import library: > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > -- Found the Arrow Python Flight static library: > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.a > -- Configuring done > -- Generating done > -- Build files have been written to: /home/bubba/work/arrow/python > -- Finished cmake for pyarrow > -- Running cmake --build for pyarrow > cmake --build . --config release -- > -- Arrow build warning level: PRODUCTION > Using ld linker > Configured for RELEASE build (set with cmake > -DCMAKE_BUILD_TYPE={release,debug,...}) > -- Build Type: RELEASE > -- Build output directory: > /home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > -- Searching for Python libs in > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > -- Looking for python3.6m > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > -- Found the Arrow Python by > -- Found the Arrow Python shared library: > -- Found the Arrow Python import library: > -- Found the Arrow Python static library: > -- Found the Arrow Flight by > -- Found the Arrow Flight shared library: > -- Found the Arrow Flight import library: > -- Found the Arrow Flight static library: > CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52 > (arrow_find_package): > Unknown CMake command "arrow_find_package". > Call Stack (most recent call first): > CMakeLists.txt:453 (find_package) > > > -- Configuring incomplete, errors occurred! > See also > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log". > See also > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log". > gmake: *** [Makefile:538: cmake_check_build_system] Error 1 > error: command 'cmake' failed with exit status 2 > (arrow-dev) bubba@localhost ~/work/arrow/python $ cmake --build . --config > release -- > Scanning dependencies of target _flight_pyx > [ 6%] Compiling Cython CXX source for _flight... > [ 6%] Built target _flight_pyx > Scanning dependencies of target _flight > [ 13%] Building CXX object CMakeFiles/_flight.dir/_flight.cpp.o > [ 20%] Linking CXX shared module build/release/_ > flight.cpython-36m-x86_64-linux-gnu.so > [ 20%] Built target _flight > Scanning dependencies of target _json_pyx > [ 26%] Compiling Cython CXX source for _json... > [ 26%] Built target _json_pyx > Scanning dependencies of target _json > [ 33%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o > [ 40%] Linking CXX shared module build/release/_ > json.cpython-36m-x86_64-linux-gnu.so > [ 40%] Built target _json > Scanning dependencies of target lib_pyx > [ 46%] Compiling Cython CXX source for lib... > [ 46%] Built target lib_pyx > Scanning dependencies of target lib > [ 53%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o > /home/bubba/work/arrow/python/lib.cpp: In function ‘PyObject* > __pyx_pf_7pyarrow_3lib_92union(PyObject*, PyObject*, PyObject*, > PyObject*)’: > /home/bubba/work/arrow/python/lib.cpp:37358:29: warning: comparison of > integer expressions of different signedness: ‘Py_ssize_t’ {aka ‘long int’} > and ‘std::vector<std::shared_ptr<arrow::Field> >::size_type’ {aka ‘long > unsigned int’} [-Wsign-compare] > __pyx_t_2 = ((__pyx_t_6 != __pyx_v_c_fields.size()) != 0); > ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~ > [ 60%] Linking CXX shared module build/release/ > lib.cpython-36m-x86_64-linux-gnu.so > [ 60%] Built target lib > Scanning dependencies of target _fs_pyx > [ 66%] Compiling Cython CXX source for _fs... > [ 66%] Built target _fs_pyx > Scanning dependencies of target _fs > [ 73%] Building CXX object CMakeFiles/_fs.dir/_fs.cpp.o > [ 80%] Linking CXX shared module build/release/_ > fs.cpython-36m-x86_64-linux-gnu.so > [ 80%] Built target _fs > Scanning dependencies of target _csv_pyx > [ 86%] Compiling Cython CXX source for _csv... > [ 86%] Built target _csv_pyx > Scanning dependencies of target _csv > [ 93%] Building CXX object CMakeFiles/_csv.dir/_csv.cpp.o > [100%] Linking CXX shared module build/release/_ > csv.cpython-36m-x86_64-linux-gnu.so > [100%] Built target _csv >
