Hi, Did you set -DARROW_PYTHON_INCLUDE_DIR, -DARROW_PYTHON_LIB_DIR=, ... by the PYARROW_CMAKE_OPTIONS environment variable? Normally, you should not define these variables by hand. They should be defined automatically. You may need to pass -DCMAKE_MODULE_PATH=/home/bubba/work/arrow/dist/lib/cmake/arrow by the PYARROW_CMAKE_OPTIONS environment variable.
Thanks, -- kou In <CAOvxMNwx_jgYov2L=b-e3Ts7d6bzdF29tQ=b8ngdiov0cq4...@mail.gmail.com> "Re: PyArrow build problems" on Thu, 12 Dec 2019 11:44:08 -0500, Justin Polchlopek <jpolchlo...@azavea.com> wrote: > Thanks for the response. I wasn't able to figure out what happened, but I > was able to get a docker environment set up to do the build. There must be > something really peculiar with my environment. (Which is probably not > something I should be surprised of. I suppose I was just hoping this was a > thing that had been seen before.) > > > On Thu, Dec 12, 2019 at 3:27 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> I'm not sure exactly what is going on but somehow, FindArrow.cmake [1] >> doesn't seem to be getting imported when the command is getting run in >> setup.py, I'm not why this would be different then running the same command >> by hand. >> >> [1] >> >> https://github.com/apache/arrow/blob/b282838d3ffea8285a06addab202014825b21993/cpp/cmake_modules/FindArrow.cmake >> >> >> On Wed, Dec 4, 2019 at 10:50 AM Justin Polchlopek <jpolchlo...@azavea.com> >> wrote: >> >> > I'm trying to get pyarrow built from source after making some changes to >> > current master. I've successfully built the C++ libraries, but `python >> > setup.py build_ext`, and all variations thereof that I can try are >> failing, >> > unable to find the arrow libs, even though the ARROW_HOME environment >> > variable is set properly. >> > >> > It appears that the way cmake is being spawned is a problem, since I can >> > copy/paste the cmake command reported by the setup script into the >> console >> > and it runs just fine. The build gives similar problems, but again >> > succeeds when run manually. Obviously, more is going on in the setup >> > script than just the cmake invocations, so I need to get past this >> problem >> > so that the build can complete. >> > >> > FWIW, I'm running this from a virtualenv. >> > >> > A transcript of the session: >> > >> > (arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py >> > build_ext >> > --inplace >> > running build_ext >> > -- Running cmake for pyarrow >> > cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include >> > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib >> > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include >> > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib >> > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python >> > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on >> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off >> > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off >> > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off >> > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off >> > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on >> > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release >> > /home/bubba/work/arrow/python >> > -- Arrow build warning level: PRODUCTION >> > Using ld linker >> > Configured for RELEASE build (set with cmake >> > -DCMAKE_BUILD_TYPE={release,debug,...}) >> > -- Build Type: RELEASE >> > -- Build output directory: >> > /home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python >> > -- Found the Arrow Python by >> > -- Found the Arrow Python shared library: >> > -- Found the Arrow Python import library: >> > -- Found the Arrow Python static library: >> > -- Found the Arrow Flight by >> > -- Found the Arrow Flight shared library: >> > -- Found the Arrow Flight import library: >> > -- Found the Arrow Flight static library: >> > CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52 >> > (arrow_find_package): >> > Unknown CMake command "arrow_find_package". >> > Call Stack (most recent call first): >> > CMakeLists.txt:453 (find_package) >> > >> > >> > -- Configuring incomplete, errors occurred! >> > See also >> > >> > >> "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log". >> > See also >> > >> > >> "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log". >> > error: command 'cmake' failed with exit status 1 >> > (arrow-dev) bubba@localhost ~/work/arrow/python $ cmake >> > -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include >> > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib >> > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include >> > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib >> > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python >> > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on >> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off >> > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off >> > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off >> > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off >> > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on >> > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release >> > /home/bubba/work/arrow/python >> > -- The C compiler identification is GNU 8.3.0 >> > -- The CXX compiler identification is GNU 8.3.0 >> > -- Check for working C compiler: /usr/bin/cc >> > -- Check for working C compiler: /usr/bin/cc -- works >> > -- Detecting C compiler ABI info >> > -- Detecting C compiler ABI info - done >> > -- Detecting C compile features >> > -- Detecting C compile features - done >> > -- Check for working CXX compiler: /usr/bin/c++ >> > -- Check for working CXX compiler: /usr/bin/c++ -- works >> > -- Detecting CXX compiler ABI info >> > -- Detecting CXX compiler ABI info - done >> > -- Detecting CXX compile features >> > -- Detecting CXX compile features - done >> > -- Performing Test CXX_SUPPORTS_SSE4_2 >> > -- Performing Test CXX_SUPPORTS_SSE4_2 - Success >> > -- Performing Test CXX_SUPPORTS_ALTIVEC >> > -- Performing Test CXX_SUPPORTS_ALTIVEC - Failed >> > -- Performing Test CXX_SUPPORTS_ARMCRC >> > -- Performing Test CXX_SUPPORTS_ARMCRC - Failed >> > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO >> > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed >> > -- Arrow build warning level: PRODUCTION >> > Using ld linker >> > Configured for RELEASE build (set with cmake >> > -DCMAKE_BUILD_TYPE={release,debug,...}) >> > -- Build Type: RELEASE >> > -- Build output directory: /home/bubba/work/arrow/python/build/release >> > -- Found PythonInterp: /home/bubba/.python_envs/arrow-dev/bin/python >> (found >> > version "3.6.5") >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib PYTHON_LIBRARY-NOTFOUND >> > -- Found PythonLibs: PYTHON_LIBRARY-NOTFOUND >> > -- Found NumPy: version "1.17.3" >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib/python3.6/site-packages/numpy/core/include >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib PYTHON_LIBRARY-NOTFOUND >> > -- Found PkgConfig: /usr/bin/pkg-config (found version "1.5.4") >> > -- Found Arrow: /home/bubba/work/arrow/dist/include (found version >> "1.0.0") >> > -- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist) >> > -- Arrow SO and ABI version: 100 >> > -- Arrow full SO version: 100.0.0 >> > -- Found the Arrow core shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow.so >> > -- Found the Arrow core import library: >> > /home/bubba/work/arrow/dist/lib/libarrow.so >> > -- Found the Arrow core static library: >> > /home/bubba/work/arrow/dist/lib/libarrow.a >> > -- Found ArrowPython: /home/bubba/work/arrow/dist/include (found version >> > "1.0.0") >> > -- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist >> > -- Found the Arrow Python shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python.so >> > -- Found the Arrow Python import library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python.so >> > -- Found the Arrow Python static library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python.a >> > -- Found ArrowFlight: /home/bubba/work/arrow/dist/include (found version >> > "1.0.0") >> > -- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist >> > -- Found the Arrow Flight shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow_flight.so >> > -- Found the Arrow Flight import library: >> > /home/bubba/work/arrow/dist/lib/libarrow_flight.so >> > -- Found the Arrow Flight static library: >> > /home/bubba/work/arrow/dist/lib/libarrow_flight.a >> > -- Found ArrowPythonFlight: /home/bubba/work/arrow/dist/include (found >> > version "1.0.0") >> > -- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist >> > -- Found the Arrow Python Flight shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so >> > -- Found the Arrow Python Flight import library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so >> > -- Found the Arrow Python Flight static library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.a >> > -- Configuring done >> > -- Generating done >> > -- Build files have been written to: /home/bubba/work/arrow/python >> > (arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py >> > build_ext >> > --inplace >> > running build_ext >> > -- Running cmake for pyarrow >> > cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include >> > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib >> > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include >> > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib >> > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python >> > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on >> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off >> > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off >> > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off >> > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off >> > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on >> > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release >> > /home/bubba/work/arrow/python >> > -- Arrow build warning level: PRODUCTION >> > Using ld linker >> > Configured for RELEASE build (set with cmake >> > -DCMAKE_BUILD_TYPE={release,debug,...}) >> > -- Build Type: RELEASE >> > -- Build output directory: /home/bubba/work/arrow/python/build/release >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib PYTHON_LIBRARY-NOTFOUND >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib PYTHON_LIBRARY-NOTFOUND >> > -- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist) >> > -- Arrow SO and ABI version: 100 >> > -- Arrow full SO version: 100.0.0 >> > -- Found the Arrow core shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow.so >> > -- Found the Arrow core import library: >> > /home/bubba/work/arrow/dist/lib/libarrow.so >> > -- Found the Arrow core static library: >> > /home/bubba/work/arrow/dist/lib/libarrow.a >> > -- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist >> > -- Found the Arrow Python shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python.so >> > -- Found the Arrow Python import library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python.so >> > -- Found the Arrow Python static library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python.a >> > -- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist >> > -- Found the Arrow Flight shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow_flight.so >> > -- Found the Arrow Flight import library: >> > /home/bubba/work/arrow/dist/lib/libarrow_flight.so >> > -- Found the Arrow Flight static library: >> > /home/bubba/work/arrow/dist/lib/libarrow_flight.a >> > -- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist >> > -- Found the Arrow Python Flight shared library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so >> > -- Found the Arrow Python Flight import library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so >> > -- Found the Arrow Python Flight static library: >> > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.a >> > -- Configuring done >> > -- Generating done >> > -- Build files have been written to: /home/bubba/work/arrow/python >> > -- Finished cmake for pyarrow >> > -- Running cmake --build for pyarrow >> > cmake --build . --config release -- >> > -- Arrow build warning level: PRODUCTION >> > Using ld linker >> > Configured for RELEASE build (set with cmake >> > -DCMAKE_BUILD_TYPE={release,debug,...}) >> > -- Build Type: RELEASE >> > -- Build output directory: >> > /home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python >> > -- Searching for Python libs in >> > >> > >> /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu >> > -- Looking for python3.6m >> > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python >> > -- Found the Arrow Python by >> > -- Found the Arrow Python shared library: >> > -- Found the Arrow Python import library: >> > -- Found the Arrow Python static library: >> > -- Found the Arrow Flight by >> > -- Found the Arrow Flight shared library: >> > -- Found the Arrow Flight import library: >> > -- Found the Arrow Flight static library: >> > CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52 >> > (arrow_find_package): >> > Unknown CMake command "arrow_find_package". >> > Call Stack (most recent call first): >> > CMakeLists.txt:453 (find_package) >> > >> > >> > -- Configuring incomplete, errors occurred! >> > See also >> > >> > >> "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log". >> > See also >> > >> > >> "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log". >> > gmake: *** [Makefile:538: cmake_check_build_system] Error 1 >> > error: command 'cmake' failed with exit status 2 >> > (arrow-dev) bubba@localhost ~/work/arrow/python $ cmake --build . >> --config >> > release -- >> > Scanning dependencies of target _flight_pyx >> > [ 6%] Compiling Cython CXX source for _flight... >> > [ 6%] Built target _flight_pyx >> > Scanning dependencies of target _flight >> > [ 13%] Building CXX object CMakeFiles/_flight.dir/_flight.cpp.o >> > [ 20%] Linking CXX shared module build/release/_ >> > flight.cpython-36m-x86_64-linux-gnu.so >> > [ 20%] Built target _flight >> > Scanning dependencies of target _json_pyx >> > [ 26%] Compiling Cython CXX source for _json... >> > [ 26%] Built target _json_pyx >> > Scanning dependencies of target _json >> > [ 33%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o >> > [ 40%] Linking CXX shared module build/release/_ >> > json.cpython-36m-x86_64-linux-gnu.so >> > [ 40%] Built target _json >> > Scanning dependencies of target lib_pyx >> > [ 46%] Compiling Cython CXX source for lib... >> > [ 46%] Built target lib_pyx >> > Scanning dependencies of target lib >> > [ 53%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o >> > /home/bubba/work/arrow/python/lib.cpp: In function ‘PyObject* >> > __pyx_pf_7pyarrow_3lib_92union(PyObject*, PyObject*, PyObject*, >> > PyObject*)’: >> > /home/bubba/work/arrow/python/lib.cpp:37358:29: warning: comparison of >> > integer expressions of different signedness: ‘Py_ssize_t’ {aka ‘long >> int’} >> > and ‘std::vector<std::shared_ptr<arrow::Field> >::size_type’ {aka ‘long >> > unsigned int’} [-Wsign-compare] >> > __pyx_t_2 = ((__pyx_t_6 != __pyx_v_c_fields.size()) != 0); >> > ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~ >> > [ 60%] Linking CXX shared module build/release/ >> > lib.cpython-36m-x86_64-linux-gnu.so >> > [ 60%] Built target lib >> > Scanning dependencies of target _fs_pyx >> > [ 66%] Compiling Cython CXX source for _fs... >> > [ 66%] Built target _fs_pyx >> > Scanning dependencies of target _fs >> > [ 73%] Building CXX object CMakeFiles/_fs.dir/_fs.cpp.o >> > [ 80%] Linking CXX shared module build/release/_ >> > fs.cpython-36m-x86_64-linux-gnu.so >> > [ 80%] Built target _fs >> > Scanning dependencies of target _csv_pyx >> > [ 86%] Compiling Cython CXX source for _csv... >> > [ 86%] Built target _csv_pyx >> > Scanning dependencies of target _csv >> > [ 93%] Building CXX object CMakeFiles/_csv.dir/_csv.cpp.o >> > [100%] Linking CXX shared module build/release/_ >> > csv.cpython-36m-x86_64-linux-gnu.so >> > [100%] Built target _csv >> > >>
