Re: Trouble building on Mac OS Mojave
I was previously using virtualenv. On Sun, 1 Sep 2019 at 09:50, Chris Teoh wrote: > Ok thanks for your help. I will attempt to use the conda approach. > > On Sun, 1 Sep 2019 at 03:27, Wes McKinney wrote: > >> Hi Chris -- to follow up to your question also, we intend for contributors >> to build from source using the CMake configuration directly. We have found >> that conda is the easiest and most reliable way to manage the dependencies >> for C++ and Python contributors across all platforms. If you follow the >> Python contribution guide and something doesn't work please let us know so >> we can work with you to fix. >> >> Wes >> >> On Sat, Aug 31, 2019, 9:18 AM Uwe L. Korn wrote: >> >> > Hello Chris, >> > >> > as a contributor, it is often simpler to use conda to construct a local >> > development environment as outlined in >> > https://arrow.apache.org/docs/developers/python.html#using-conda >> > This is the typical environment most contributors work in. Even when not >> > using conda as a package/environment manager elsewhere, I would >> recommend >> > to use it to setup your Arrow build environment as this is the way most >> > developers do. Thus it will be easier to help you and this is the setup >> we >> > (try to) maintain best. >> > >> > Cheers >> > Uwe >> > >> > On Sat, Aug 31, 2019, at 3:48 PM, Chris Teoh wrote: >> > > Does this approach fit with potentially a contributor's workflow? I >> was >> > > looking into contributing though I'm unsure if I am doing it right. >> > > >> > > On Sat, 31 Aug 2019 at 22:22, Jeroen Ooms >> wrote: >> > > >> > > > On Sat, Aug 31, 2019 at 4:48 AM Chris Teoh >> > wrote: >> > > > > >> > > > > That being said, is there an easier way by using a Docker >> container I >> > > > could >> > > > > use to build this in? >> > > > >> > > > An easy way to install arrow on MacOS is using homebrew. To get a >> > > > precompiled version of the latest release: >> > > > >> > > > brew install apache-arrow >> > > > >> > > > Or to build the master branch from source: >> > > > >> > > >brew install apache-arrow --HEAD >> > > > >> > > > If you want to customize the configuration use "brew edit >> > > > apache-arrow" before building from source. >> > > > >> > > >> > > >> > > -- >> > > Chris >> > > >> > >> > > > -- > Chris > -- Chris
Re: Trouble building on Mac OS Mojave
Ok thanks for your help. I will attempt to use the conda approach. On Sun, 1 Sep 2019 at 03:27, Wes McKinney wrote: > Hi Chris -- to follow up to your question also, we intend for contributors > to build from source using the CMake configuration directly. We have found > that conda is the easiest and most reliable way to manage the dependencies > for C++ and Python contributors across all platforms. If you follow the > Python contribution guide and something doesn't work please let us know so > we can work with you to fix. > > Wes > > On Sat, Aug 31, 2019, 9:18 AM Uwe L. Korn wrote: > > > Hello Chris, > > > > as a contributor, it is often simpler to use conda to construct a local > > development environment as outlined in > > https://arrow.apache.org/docs/developers/python.html#using-conda > > This is the typical environment most contributors work in. Even when not > > using conda as a package/environment manager elsewhere, I would recommend > > to use it to setup your Arrow build environment as this is the way most > > developers do. Thus it will be easier to help you and this is the setup > we > > (try to) maintain best. > > > > Cheers > > Uwe > > > > On Sat, Aug 31, 2019, at 3:48 PM, Chris Teoh wrote: > > > Does this approach fit with potentially a contributor's workflow? I was > > > looking into contributing though I'm unsure if I am doing it right. > > > > > > On Sat, 31 Aug 2019 at 22:22, Jeroen Ooms > wrote: > > > > > > > On Sat, Aug 31, 2019 at 4:48 AM Chris Teoh > > wrote: > > > > > > > > > > That being said, is there an easier way by using a Docker > container I > > > > could > > > > > use to build this in? > > > > > > > > An easy way to install arrow on MacOS is using homebrew. To get a > > > > precompiled version of the latest release: > > > > > > > > brew install apache-arrow > > > > > > > > Or to build the master branch from source: > > > > > > > >brew install apache-arrow --HEAD > > > > > > > > If you want to customize the configuration use "brew edit > > > > apache-arrow" before building from source. > > > > > > > > > > > > > -- > > > Chris > > > > > > -- Chris
Re: Trouble building on Mac OS Mojave
Hi Chris -- to follow up to your question also, we intend for contributors to build from source using the CMake configuration directly. We have found that conda is the easiest and most reliable way to manage the dependencies for C++ and Python contributors across all platforms. If you follow the Python contribution guide and something doesn't work please let us know so we can work with you to fix. Wes On Sat, Aug 31, 2019, 9:18 AM Uwe L. Korn wrote: > Hello Chris, > > as a contributor, it is often simpler to use conda to construct a local > development environment as outlined in > https://arrow.apache.org/docs/developers/python.html#using-conda > This is the typical environment most contributors work in. Even when not > using conda as a package/environment manager elsewhere, I would recommend > to use it to setup your Arrow build environment as this is the way most > developers do. Thus it will be easier to help you and this is the setup we > (try to) maintain best. > > Cheers > Uwe > > On Sat, Aug 31, 2019, at 3:48 PM, Chris Teoh wrote: > > Does this approach fit with potentially a contributor's workflow? I was > > looking into contributing though I'm unsure if I am doing it right. > > > > On Sat, 31 Aug 2019 at 22:22, Jeroen Ooms wrote: > > > > > On Sat, Aug 31, 2019 at 4:48 AM Chris Teoh > wrote: > > > > > > > > That being said, is there an easier way by using a Docker container I > > > could > > > > use to build this in? > > > > > > An easy way to install arrow on MacOS is using homebrew. To get a > > > precompiled version of the latest release: > > > > > > brew install apache-arrow > > > > > > Or to build the master branch from source: > > > > > >brew install apache-arrow --HEAD > > > > > > If you want to customize the configuration use "brew edit > > > apache-arrow" before building from source. > > > > > > > > > -- > > Chris > > >
Re: Trouble building on Mac OS Mojave
Hello Chris, as a contributor, it is often simpler to use conda to construct a local development environment as outlined in https://arrow.apache.org/docs/developers/python.html#using-conda This is the typical environment most contributors work in. Even when not using conda as a package/environment manager elsewhere, I would recommend to use it to setup your Arrow build environment as this is the way most developers do. Thus it will be easier to help you and this is the setup we (try to) maintain best. Cheers Uwe On Sat, Aug 31, 2019, at 3:48 PM, Chris Teoh wrote: > Does this approach fit with potentially a contributor's workflow? I was > looking into contributing though I'm unsure if I am doing it right. > > On Sat, 31 Aug 2019 at 22:22, Jeroen Ooms wrote: > > > On Sat, Aug 31, 2019 at 4:48 AM Chris Teoh wrote: > > > > > > That being said, is there an easier way by using a Docker container I > > could > > > use to build this in? > > > > An easy way to install arrow on MacOS is using homebrew. To get a > > precompiled version of the latest release: > > > > brew install apache-arrow > > > > Or to build the master branch from source: > > > >brew install apache-arrow --HEAD > > > > If you want to customize the configuration use "brew edit > > apache-arrow" before building from source. > > > > > -- > Chris >
Re: Trouble building on Mac OS Mojave
Does this approach fit with potentially a contributor's workflow? I was looking into contributing though I'm unsure if I am doing it right. On Sat, 31 Aug 2019 at 22:22, Jeroen Ooms wrote: > On Sat, Aug 31, 2019 at 4:48 AM Chris Teoh wrote: > > > > That being said, is there an easier way by using a Docker container I > could > > use to build this in? > > An easy way to install arrow on MacOS is using homebrew. To get a > precompiled version of the latest release: > > brew install apache-arrow > > Or to build the master branch from source: > >brew install apache-arrow --HEAD > > If you want to customize the configuration use "brew edit > apache-arrow" before building from source. > -- Chris
Re: Trouble building on Mac OS Mojave
On Sat, Aug 31, 2019 at 4:48 AM Chris Teoh wrote: > > That being said, is there an easier way by using a Docker container I could > use to build this in? An easy way to install arrow on MacOS is using homebrew. To get a precompiled version of the latest release: brew install apache-arrow Or to build the master branch from source: brew install apache-arrow --HEAD If you want to customize the configuration use "brew edit apache-arrow" before building from source.
Re: Trouble building on Mac OS Mojave
That being said, is there an easier way by using a Docker container I could
use to build this in?
On Sat, 31 Aug 2019 at 12:44, Chris Teoh wrote:
> Hey there,
>
> Brand new to Arrow here.
>
> Trying to build it following the instructions and I get errors with the
> ORC module building cpp
>
> In file included from
> /Users/test/GitHub/arrow/cpp/build/orc_ep-prefix/src/orc_ep/c++/src/wrap/orc-proto-wrapper.cc:44:
>
> /Users/test/GitHub/arrow/cpp/build/orc_ep-prefix/src/orc_ep-build/c++/src/orc_proto.pb.cc:960:145:
> error: possible misuse of comma operator here [-Werror,-Wcomma]
> static bool dynamic_init_dummy_orc_5fproto_2eproto = (
>
> ::PROTOBUF_NAMESPACE_ID::internal::AddDescriptors(&descriptor_table_orc_5fproto_2eproto),
> true);
>
> ^
>
> /Users/test/GitHub/arrow/cpp/build/orc_ep-prefix/src/orc_ep-build/c++/src/orc_proto.pb.cc:960:57:
> note: cast expression to void to silence warning
> static bool dynamic_init_dummy_orc_5fproto_2eproto = (
>
> ::PROTOBUF_NAMESPACE_ID::internal::AddDescriptors(&descriptor_table_orc_5fproto_2eproto),
> true);
>
> I can disable the ORC module and that part builds fine, build command is:-
>
> pushd arrow/cpp/build
>
>
>
> cmake -DPYTHON_EXECUTABLE=$VIRTUAL_ENV/bin/python
> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
>
> -DCMAKE_INSTALL_LIBDIR=lib \
>
> -DARROW_FLIGHT=ON \
>
> -DARROW_GANDIVA=ON \
>
> -DARROW_ORC=OFF \
>
> -DARROW_PARQUET=ON \
>
> -DARROW_PYTHON=ON \
>
> -DARROW_PLASMA=ON \
>
> -DARROW_BUILD_TESTS=ON \
>
> ..
>
> make -j4
>
> make install
>
> popd
>
> then I try to build the python module:-
>
> pushd arrow/python
> export PYARROW_WITH_FLIGHT=1
> export PYARROW_WITH_GANDIVA=1
> export PYARROW_WITH_ORC=0
> export PYARROW_WITH_PARQUET=1
> python setup.py build_ext --inplace
> popd
>
> and get:-
> running build_ext
> creating build
> creating build/temp.macosx-10.14-intel-2.7
> -- Running cmake for pyarrow
> cmake -DPYTHON_EXECUTABLE=/Users/test/GitHub/pyarrow/bin/python
> -DPYARROW_BUILD_FLIGHT=on -DPYARROW_BUILD_PARQUET=on
> -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_BUILD_GANDIVA=on
> -DCMAKE_BUILD_TYPE=release /Users/test/GitHub/arrow/python
> -- The C compiler identification is AppleClang 10.0.1.10010046
> -- The CXX compiler identification is AppleClang 10.0.1.10010046
> -- Check for working C compiler:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
> -- Check for working C compiler:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
> -- works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Detecting C compile features
> -- Detecting C compile features - done
> -- Check for working CXX compiler:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
> -- Check for working CXX compiler:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
> -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> -- Compiler command: env LANG=C
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
> -v
> -- Compiler version: Apple LLVM version 10.0.1 (clang-1001.0.46.4)
> Target: x86_64-apple-darwin18.6.0
> Thread model: posix
> InstalledDir:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>
> -- Compiler id: Clang
> Selected compiler clang 4.1.0svn
> -- Performing Test CXX_SUPPORTS_SSE4_2
> -- Performing Test CXX_SUPPORTS_SSE4_2 - Success
> -- Performing Test CXX_SUPPORTS_ALTIVEC
> -- Performing Test CXX_SUPPORTS_ALTIVEC - Success
> -- Performing Test CXX_SUPPORTS_ARMCRC
> -- Performing Test CXX_SUPPORTS_ARMCRC - Failed
> -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO
> -- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed
> -- Arrow build warning level: PRODUCTION
> Configured for RELEASE build (set with cmake
> -DCMAKE_BUILD_TYPE={release,debug,...})
> -- Build Type: RELEASE
> -- Build output directory:
> /Users/test/GitHub/arrow/python/build/temp.macosx-10.14-intel-2.7/release
> -- Found PythonInterp: /Users/test/GitHub/pyarrow/bin/python (found
> version "2.7.10")
> -- Found PythonLibs:
> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
> -- Found NumPy: version "1.16.5"
> /Users/test/GitHub/pyarrow/lib/python2.7/site-packages/numpy/core/include
> -- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
> -- Found the Arrow core library: /Users/test/GitHub/dist/lib/libarrow.dylib
> -- Found the Arrow Python library:
> /Users/test/GitHub/dist/lib/libarrow_python.dylib
> -- Added shared library dependency arrow_shared:
> /Users/test/GitHub/dist/lib/libarrow.dylib
> -- Added shared library dependency arrow_python
Trouble building on Mac OS Mojave
Hey there,
Brand new to Arrow here.
Trying to build it following the instructions and I get errors with the ORC
module building cpp
In file included from
/Users/test/GitHub/arrow/cpp/build/orc_ep-prefix/src/orc_ep/c++/src/wrap/orc-proto-wrapper.cc:44:
/Users/test/GitHub/arrow/cpp/build/orc_ep-prefix/src/orc_ep-build/c++/src/orc_proto.pb.cc:960:145:
error: possible misuse of comma operator here [-Werror,-Wcomma]
static bool dynamic_init_dummy_orc_5fproto_2eproto = (
::PROTOBUF_NAMESPACE_ID::internal::AddDescriptors(&descriptor_table_orc_5fproto_2eproto),
true);
^
/Users/test/GitHub/arrow/cpp/build/orc_ep-prefix/src/orc_ep-build/c++/src/orc_proto.pb.cc:960:57:
note: cast expression to void to silence warning
static bool dynamic_init_dummy_orc_5fproto_2eproto = (
::PROTOBUF_NAMESPACE_ID::internal::AddDescriptors(&descriptor_table_orc_5fproto_2eproto),
true);
I can disable the ORC module and that part builds fine, build command is:-
pushd arrow/cpp/build
cmake -DPYTHON_EXECUTABLE=$VIRTUAL_ENV/bin/python
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DCMAKE_INSTALL_LIBDIR=lib \
-DARROW_FLIGHT=ON \
-DARROW_GANDIVA=ON \
-DARROW_ORC=OFF \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=ON \
..
make -j4
make install
popd
then I try to build the python module:-
pushd arrow/python
export PYARROW_WITH_FLIGHT=1
export PYARROW_WITH_GANDIVA=1
export PYARROW_WITH_ORC=0
export PYARROW_WITH_PARQUET=1
python setup.py build_ext --inplace
popd
and get:-
running build_ext
creating build
creating build/temp.macosx-10.14-intel-2.7
-- Running cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/Users/test/GitHub/pyarrow/bin/python
-DPYARROW_BUILD_FLIGHT=on -DPYARROW_BUILD_PARQUET=on
-DPYARROW_BOOST_USE_SHARED=on -DPYARROW_BUILD_GANDIVA=on
-DCMAKE_BUILD_TYPE=release /Users/test/GitHub/arrow/python
-- The C compiler identification is AppleClang 10.0.1.10010046
-- The CXX compiler identification is AppleClang 10.0.1.10010046
-- Check for working C compiler:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Compiler command: env LANG=C
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-v
-- Compiler version: Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
-- Compiler id: Clang
Selected compiler clang 4.1.0svn
-- Performing Test CXX_SUPPORTS_SSE4_2
-- Performing Test CXX_SUPPORTS_SSE4_2 - Success
-- Performing Test CXX_SUPPORTS_ALTIVEC
-- Performing Test CXX_SUPPORTS_ALTIVEC - Success
-- Performing Test CXX_SUPPORTS_ARMCRC
-- Performing Test CXX_SUPPORTS_ARMCRC - Failed
-- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO
-- Performing Test CXX_SUPPORTS_ARMV8_CRC_CRYPTO - Failed
-- Arrow build warning level: PRODUCTION
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory:
/Users/test/GitHub/arrow/python/build/temp.macosx-10.14-intel-2.7/release
-- Found PythonInterp: /Users/test/GitHub/pyarrow/bin/python (found version
"2.7.10")
-- Found PythonLibs:
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
-- Found NumPy: version "1.16.5"
/Users/test/GitHub/pyarrow/lib/python2.7/site-packages/numpy/core/include
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
-- Found the Arrow core library: /Users/test/GitHub/dist/lib/libarrow.dylib
-- Found the Arrow Python library:
/Users/test/GitHub/dist/lib/libarrow_python.dylib
-- Added shared library dependency arrow_shared:
/Users/test/GitHub/dist/lib/libarrow.dylib
-- Added shared library dependency arrow_python_shared:
/Users/test/GitHub/dist/lib/libarrow_python.dylib
-- Checking for module 'parquet'
-- No package 'parquet' found
-- Could not find the parquet library. Looked in system search paths.
CMake Error at CMakeLists.txt:417 (message):
Unable to locate Parquet libraries
-- Configuring incomplete, errors occurred!
My "dist" folder is as follows:-
dist
dist/bin
dist/bin/plasma-store-serv
