Thanks for the response. I wasn't able to figure out what happened, but I was able to get a docker environment set up to do the build. There must be something really peculiar with my environment. (Which is probably not something I should be surprised of. I suppose I was just hoping this was a thing that had been seen before.)
On Thu, Dec 12, 2019 at 3:27 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > I'm not sure exactly what is going on but somehow, FindArrow.cmake [1] > doesn't seem to be getting imported when the command is getting run in > setup.py, I'm not why this would be different then running the same command > by hand. > > [1] > > https://github.com/apache/arrow/blob/b282838d3ffea8285a06addab202014825b21993/cpp/cmake_modules/FindArrow.cmake > > > On Wed, Dec 4, 2019 at 10:50 AM Justin Polchlopek <jpolchlo...@azavea.com> > wrote: > > > I'm trying to get pyarrow built from source after making some changes to > > current master. I've successfully built the C++ libraries, but `python > > setup.py build_ext`, and all variations thereof that I can try are > failing, > > unable to find the arrow libs, even though the ARROW_HOME environment > > variable is set properly. > > > > It appears that the way cmake is being spawned is a problem, since I can > > copy/paste the cmake command reported by the setup script into the > console > > and it runs just fine. The build gives similar problems, but again > > succeeds when run manually. Obviously, more is going on in the setup > > script than just the cmake invocations, so I need to get past this > problem > > so that the build can complete. > > > > FWIW, I'm running this from a virtualenv. > > > > A transcript of the session: > > > > (arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py > > build_ext > > --inplace > > running build_ext > > -- Running cmake for pyarrow > > cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib > > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib > > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python > > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on > > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off > > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off > > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off > > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off > > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on > > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release > > /home/bubba/work/arrow/python > > -- Arrow build warning level: PRODUCTION > > Using ld linker > > Configured for RELEASE build (set with cmake > > -DCMAKE_BUILD_TYPE={release,debug,...}) > > -- Build Type: RELEASE > > -- Build output directory: > > /home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > > -- Found the Arrow Python by > > -- Found the Arrow Python shared library: > > -- Found the Arrow Python import library: > > -- Found the Arrow Python static library: > > -- Found the Arrow Flight by > > -- Found the Arrow Flight shared library: > > -- Found the Arrow Flight import library: > > -- Found the Arrow Flight static library: > > CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52 > > (arrow_find_package): > > Unknown CMake command "arrow_find_package". > > Call Stack (most recent call first): > > CMakeLists.txt:453 (find_package) > > > > > > -- Configuring incomplete, errors occurred! > > See also > > > > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log". > > See also > > > > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log". > > error: command 'cmake' failed with exit status 1 > > (arrow-dev) bubba@localhost ~/work/arrow/python $ cmake > > -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib > > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib > > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python > > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on > > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off > > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off > > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off > > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off > > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on > > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release > > /home/bubba/work/arrow/python > > -- The C compiler identification is GNU 8.3.0 > > -- The CXX compiler identification is GNU 8.3.0 > > -- Check for working C compiler: /usr/bin/cc > > -- Check for working C compiler: /usr/bin/cc -- works > > -- Detecting C compiler ABI info > > -- Detecting C compiler ABI info - done > > -- Detecting C compile features > > -- Detecting C compile features - done > > -- Check for working CXX compiler: /usr/bin/c++ > > -- Check for working CXX compiler: /usr/bin/c++ -- works > > -- Detecting CXX compiler ABI info > > -- Detecting CXX compiler ABI info - done > > -- Detecting CXX compile features > > -- Detecting CXX compile features - done > > -- Performing Test CXX_SUPPORTS_SSE4_2 > > -- Performing Test CXX_SUPPORTS_SSE4_2 - Success > > -- Performing Test CXX_SUPPORTS_ALTIVEC > > -- Performing Test CXX_SUPPORTS_ALTIVEC - Failed > > -- Performing Test CXX_SUPPORTS_ARMCRC > > -- Performing Test CXX_SUPPORTS_ARMCRC - Failed > > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO > > -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed > > -- Arrow build warning level: PRODUCTION > > Using ld linker > > Configured for RELEASE build (set with cmake > > -DCMAKE_BUILD_TYPE={release,debug,...}) > > -- Build Type: RELEASE > > -- Build output directory: /home/bubba/work/arrow/python/build/release > > -- Found PythonInterp: /home/bubba/.python_envs/arrow-dev/bin/python > (found > > version "3.6.5") > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > > -- Found PythonLibs: PYTHON_LIBRARY-NOTFOUND > > -- Found NumPy: version "1.17.3" > > > > > /home/bubba/.python_envs/arrow-dev/lib/python3.6/site-packages/numpy/core/include > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > > -- Found PkgConfig: /usr/bin/pkg-config (found version "1.5.4") > > -- Found Arrow: /home/bubba/work/arrow/dist/include (found version > "1.0.0") > > -- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist) > > -- Arrow SO and ABI version: 100 > > -- Arrow full SO version: 100.0.0 > > -- Found the Arrow core shared library: > > /home/bubba/work/arrow/dist/lib/libarrow.so > > -- Found the Arrow core import library: > > /home/bubba/work/arrow/dist/lib/libarrow.so > > -- Found the Arrow core static library: > > /home/bubba/work/arrow/dist/lib/libarrow.a > > -- Found ArrowPython: /home/bubba/work/arrow/dist/include (found version > > "1.0.0") > > -- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist > > -- Found the Arrow Python shared library: > > /home/bubba/work/arrow/dist/lib/libarrow_python.so > > -- Found the Arrow Python import library: > > /home/bubba/work/arrow/dist/lib/libarrow_python.so > > -- Found the Arrow Python static library: > > /home/bubba/work/arrow/dist/lib/libarrow_python.a > > -- Found ArrowFlight: /home/bubba/work/arrow/dist/include (found version > > "1.0.0") > > -- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist > > -- Found the Arrow Flight shared library: > > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > > -- Found the Arrow Flight import library: > > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > > -- Found the Arrow Flight static library: > > /home/bubba/work/arrow/dist/lib/libarrow_flight.a > > -- Found ArrowPythonFlight: /home/bubba/work/arrow/dist/include (found > > version "1.0.0") > > -- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist > > -- Found the Arrow Python Flight shared library: > > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > > -- Found the Arrow Python Flight import library: > > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > > -- Found the Arrow Python Flight static library: > > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.a > > -- Configuring done > > -- Generating done > > -- Build files have been written to: /home/bubba/work/arrow/python > > (arrow-dev) bubba@localhost ~/work/arrow/python $ python setup.py > > build_ext > > --inplace > > running build_ext > > -- Running cmake for pyarrow > > cmake -DARROW_PYTHON_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > > -DARROW_PYTHON_LIB_DIR=/home/bubba/work/arrow/dist/lib > > -DARROW_FLIGHT_INCLUDE_DIR=/home/bubba/work/arrow/dist/include > > -DARROW_FLIGHT_LIB_DIR=/home/bubba/work/arrow/dist/lib > > -DPYTHON_EXECUTABLE=/home/bubba/.python_envs/arrow-dev/bin/python > > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on > > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ORC=off > > -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off > > -DPYARROW_BUILD_S3=off -DPYARROW_USE_TENSORFLOW=off > > -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off > > -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on > > -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release > > /home/bubba/work/arrow/python > > -- Arrow build warning level: PRODUCTION > > Using ld linker > > Configured for RELEASE build (set with cmake > > -DCMAKE_BUILD_TYPE={release,debug,...}) > > -- Build Type: RELEASE > > -- Build output directory: /home/bubba/work/arrow/python/build/release > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib PYTHON_LIBRARY-NOTFOUND > > -- Arrow version: 1.0.0 (HOME: /home/bubba/work/arrow/dist) > > -- Arrow SO and ABI version: 100 > > -- Arrow full SO version: 100.0.0 > > -- Found the Arrow core shared library: > > /home/bubba/work/arrow/dist/lib/libarrow.so > > -- Found the Arrow core import library: > > /home/bubba/work/arrow/dist/lib/libarrow.so > > -- Found the Arrow core static library: > > /home/bubba/work/arrow/dist/lib/libarrow.a > > -- Found the Arrow Python by HOME: /home/bubba/work/arrow/dist > > -- Found the Arrow Python shared library: > > /home/bubba/work/arrow/dist/lib/libarrow_python.so > > -- Found the Arrow Python import library: > > /home/bubba/work/arrow/dist/lib/libarrow_python.so > > -- Found the Arrow Python static library: > > /home/bubba/work/arrow/dist/lib/libarrow_python.a > > -- Found the Arrow Flight by HOME: /home/bubba/work/arrow/dist > > -- Found the Arrow Flight shared library: > > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > > -- Found the Arrow Flight import library: > > /home/bubba/work/arrow/dist/lib/libarrow_flight.so > > -- Found the Arrow Flight static library: > > /home/bubba/work/arrow/dist/lib/libarrow_flight.a > > -- Found the Arrow Python Flight by HOME: /home/bubba/work/arrow/dist > > -- Found the Arrow Python Flight shared library: > > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > > -- Found the Arrow Python Flight import library: > > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.so > > -- Found the Arrow Python Flight static library: > > /home/bubba/work/arrow/dist/lib/libarrow_python_flight.a > > -- Configuring done > > -- Generating done > > -- Build files have been written to: /home/bubba/work/arrow/python > > -- Finished cmake for pyarrow > > -- Running cmake --build for pyarrow > > cmake --build . --config release -- > > -- Arrow build warning level: PRODUCTION > > Using ld linker > > Configured for RELEASE build (set with cmake > > -DCMAKE_BUILD_TYPE={release,debug,...}) > > -- Build Type: RELEASE > > -- Build output directory: > > /home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/release > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > > -- Searching for Python libs in > > > > > /home/bubba/.python_envs/arrow-dev/lib64;/home/bubba/.python_envs/arrow-dev/lib;/usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu > > -- Looking for python3.6m > > -- Found Python lib /home/bubba/.python_envs/arrow-dev/bin/python > > -- Found the Arrow Python by > > -- Found the Arrow Python shared library: > > -- Found the Arrow Python import library: > > -- Found the Arrow Python static library: > > -- Found the Arrow Flight by > > -- Found the Arrow Flight shared library: > > -- Found the Arrow Flight import library: > > -- Found the Arrow Flight static library: > > CMake Error at cmake_modules/FindArrowPythonFlight.cmake:52 > > (arrow_find_package): > > Unknown CMake command "arrow_find_package". > > Call Stack (most recent call first): > > CMakeLists.txt:453 (find_package) > > > > > > -- Configuring incomplete, errors occurred! > > See also > > > > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log". > > See also > > > > > "/home/bubba/work/arrow/python/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log". > > gmake: *** [Makefile:538: cmake_check_build_system] Error 1 > > error: command 'cmake' failed with exit status 2 > > (arrow-dev) bubba@localhost ~/work/arrow/python $ cmake --build . > --config > > release -- > > Scanning dependencies of target _flight_pyx > > [ 6%] Compiling Cython CXX source for _flight... > > [ 6%] Built target _flight_pyx > > Scanning dependencies of target _flight > > [ 13%] Building CXX object CMakeFiles/_flight.dir/_flight.cpp.o > > [ 20%] Linking CXX shared module build/release/_ > > flight.cpython-36m-x86_64-linux-gnu.so > > [ 20%] Built target _flight > > Scanning dependencies of target _json_pyx > > [ 26%] Compiling Cython CXX source for _json... > > [ 26%] Built target _json_pyx > > Scanning dependencies of target _json > > [ 33%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o > > [ 40%] Linking CXX shared module build/release/_ > > json.cpython-36m-x86_64-linux-gnu.so > > [ 40%] Built target _json > > Scanning dependencies of target lib_pyx > > [ 46%] Compiling Cython CXX source for lib... > > [ 46%] Built target lib_pyx > > Scanning dependencies of target lib > > [ 53%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o > > /home/bubba/work/arrow/python/lib.cpp: In function ‘PyObject* > > __pyx_pf_7pyarrow_3lib_92union(PyObject*, PyObject*, PyObject*, > > PyObject*)’: > > /home/bubba/work/arrow/python/lib.cpp:37358:29: warning: comparison of > > integer expressions of different signedness: ‘Py_ssize_t’ {aka ‘long > int’} > > and ‘std::vector<std::shared_ptr<arrow::Field> >::size_type’ {aka ‘long > > unsigned int’} [-Wsign-compare] > > __pyx_t_2 = ((__pyx_t_6 != __pyx_v_c_fields.size()) != 0); > > ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~ > > [ 60%] Linking CXX shared module build/release/ > > lib.cpython-36m-x86_64-linux-gnu.so > > [ 60%] Built target lib > > Scanning dependencies of target _fs_pyx > > [ 66%] Compiling Cython CXX source for _fs... > > [ 66%] Built target _fs_pyx > > Scanning dependencies of target _fs > > [ 73%] Building CXX object CMakeFiles/_fs.dir/_fs.cpp.o > > [ 80%] Linking CXX shared module build/release/_ > > fs.cpython-36m-x86_64-linux-gnu.so > > [ 80%] Built target _fs > > Scanning dependencies of target _csv_pyx > > [ 86%] Compiling Cython CXX source for _csv... > > [ 86%] Built target _csv_pyx > > Scanning dependencies of target _csv > > [ 93%] Building CXX object CMakeFiles/_csv.dir/_csv.cpp.o > > [100%] Linking CXX shared module build/release/_ > > csv.cpython-36m-x86_64-linux-gnu.so > > [100%] Built target _csv > > >
