One odd thing I noticed is that on Mac, calling ` conda install -c conda-forge pyarrow ` defaults to installing pyarrow 3.0.0. However, on Ubuntu it defaults to installing pyarrow 0.11.1, which is why above I've been specifying '==3.0.0` on all of the conda installs. I don't know why conda on Ubuntu is attempting to install pyarrow 0.11.1 rather than latest.
On Tue, Apr 13, 2021 at 1:45 PM, Xander Dunn < [email protected] > wrote: > > ``` > > $ conda create -n fresh_env python=3.7 > > $ conda activate fresh_env > > $ conda install -c conda-forge pyarrow==3.0.0 > > $ python > > >>> import pyarrow as pa > > >>> pa.__version__ > > '3.0.0' > > >>> pa.__file__ > > '/ home/ xander/ anaconda3/ envs/ fresh_env/ lib/ python3. 7/ site-packages/ > pyarrow/ __init__. py ( > http://home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/__init__.py > ) ' > > >>> import pyarrow.plasma > > >>> import pyarrow.gandiva > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > ModuleNotFoundError: No module named 'pyarrow.gandiva' > > ``` > > Same result in a fresh conda environment. The library is also still > missing from the installed location: > > ``` > > $ l > /home/xander/anaconda3/envs/fresh_env/lib/python3.7/site-packages/pyarrow/ > | grep gandiva > > -rw-rw-r-- 3 xander xander 17K Jan 18 14:00 gandiva.pyx > > ``` > > > > And a correction to what I wrote earlier, I'm on Ubuntu 18.04: > > ``` > > $ lsb_release -a > > No LSB modules are available. > > Distributor ID: Ubuntu > > Description: Ubuntu 18.04.5 LTS > > Release: 18.04 > > Codename: bionic > > $ conda --version > > conda 4.10.0 > > ``` > > > > I do have several arrow packages installed via apt, but I don't think > that's interfering: > > ``` > > apache-arrow-archive-keyring/bionic,now 3.0.0-1 all [installed] > > gir1.2-arrow-1.0/bionic,now 3.0.0-1 amd64 [installed,automatic] > > libarrow-dev/bionic,now 3.0.0-1 amd64 [installed] > > libarrow-glib-dev/bionic,now 3.0.0-1 amd64 [installed,automatic] > > libarrow-glib300/bionic,now 3.0.0-1 amd64 [installed,automatic] > > libarrow300/bionic,now 3.0.0-1 amd64 [installed] > > ``` > > > > Thanks for the help, > > Xander > > > > > > On Tue, Apr 13, 2021 at 1:25 PM, Wes McKinney < wesmckinn@ gmail. com ( > [email protected] ) > wrote: > >> It looks to me like you have the wheel installed, not the conda package. >> Can you reproduce this on Ubuntu from a fresh conda environment? >> >> On Tue, Apr 13, 2021 at 3:16 PM Xander Dunn < xander@ xander. ai ( >> [email protected] ) > wrote: >> >> >>> Typo. The issue remains present. From my Ubuntu machine just now: >>> >>> ``` >>> >>> $ python >>> >>> >>> import pyarrow as pa >>> >>> >>> print(pa.__file__) >>> >>> / home/ xander/ anaconda3/ envs/ plutus_model/ lib/ python3. 7/ >>> site-packages/ >>> pyarrow/ __init__. py ( >>> http://home/xander/anaconda3/envs/plutus_model/lib/python3.7/site-packages/pyarrow/__init__.py >>> ) >>> >>> >>> import pyarrow.plasma >>> >>> >>> import pyarrow.gandiva >>> >>> Traceback (most recent call last): >>> >>> File "<stdin>", line 1, in <module> >>> >>> ModuleNotFoundError: No module named 'pyarrow.gandiva' >>> >>> ``` >>> >>> >>> >>> The .py I'm executing on both machines is identical. Works on mac. Not >>> found on Ubuntu. >>> >>> >>> >>> >>> On Tue, Apr 13, 2021 at 1:01 PM, Micah Kornfield < emkornfield@ gmail. com >>> ( [email protected] ) > wrote: >>> >>>> Hi Xander, >>>> Was there autocorrect on this e-mail? the second example shows "gondiva" >>>> not "gandiva" >>>> >>>> >>>> -Micah >>>> >>>> On Tue, Apr 13, 2021 at 12:59 PM Xander Dunn < xander@ xander. ai ( >>>> [email protected] ) > wrote: >>>> >>>> >>>>> On my local macOS 11.2.3: >>>>> >>>>> ``` >>>>> >>>>> $ python --version >>>>> >>>>> Python 3.7.10 >>>>> >>>>> $ pip --version >>>>> >>>>> pip 21.0.1 from >>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pip (python >>>>> 3.7) >>>>> >>>>> $ pip list | grep pyarrow >>>>> >>>>> pyarrow 3.0.0 >>>>> >>>>> $ which python >>>>> >>>>> /usr/local/anaconda3/envs/my_model/bin/python >>>>> >>>>> $ python >>>>> >>>>> >>> import pyarrow as pa >>>>> >>>>> >>> print(pa.__file__) >>>>> >>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ >>>>> pyarrow/ >>>>> __init__. py ( >>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py >>>>> ) >>>>> >>>>> >>> import pyarrow.plasma >>>>> >>>>> >>> import pyarrow.gandiva as ga >>>>> >>>>> >>> print(ga.__file__) >>>>> >>>>> / usr/ local/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ >>>>> pyarrow/ >>>>> gandiva. cpython-37m-darwin. so ( >>>>> http://usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/gandiva.cpython-37m-darwin.so >>>>> ) >>>>> >>>>> ``` >>>>> >>>>> >>>>> >>>>> On my Ubuntu 14.04 instance: >>>>> >>>>> ``` >>>>> >>>>> $ python --version >>>>> >>>>> Python 3.7.10 >>>>> >>>>> $ pip --version >>>>> >>>>> pip 21.0.1 from >>>>> /home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pip >>>>> (python 3.7) >>>>> >>>>> $ pip list | grep pyarrow >>>>> >>>>> pyarrow 3.0.0 >>>>> >>>>> $ which python >>>>> >>>>> /home/xander/anaconda3/envs/my_model/bin/python >>>>> >>>>> $ python >>>>> >>>>> >>> import pyarrow as pa >>>>> >>>>> >>> print(pa.__file__) >>>>> >>>>> / home/ xander/ anaconda3/ envs/ my_model/ lib/ python3. 7/ site-packages/ >>>>> pyarrow/ __init__. py ( >>>>> http://home/xander/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/__init__.py >>>>> ) >>>>> >>>>> >>> import pyarrow.plasma >>>>> >>>>> >>> import pyarrow.gondiva >>>>> >>>>> Traceback (most recent call last): >>>>> >>>>> File "<stdin>", line 1, in <module> >>>>> >>>>> ModuleNotFoundError: No module named 'pyarrow.gondiva' >>>>> >>>>> ``` >>>>> >>>>> You can see that pyarrow.gondiva is found on mac but not on Ubuntu. Same >>>>> Python version. Same pyarrow version. I installed both of them with `conda >>>>> install -c conda-forge pyarrow==3.0.0`. >>>>> >>>>> >>>>> >>>>> On Mac, I see the expected Cython file and library: >>>>> >>>>> ``` >>>>> >>>>> $ l >>>>> /usr/local/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | >>>>> grep gandiva >>>>> >>>>> -rwxrwxr-x 2 xander staff 221K Apr 1 12:44 gandiva. >>>>> cpython-37m-darwin. >>>>> so ( http://gandiva.cpython-37m-darwin.so/ ) >>>>> >>>>> -rw-rw-r-- 2 xander staff 17K Jan 18 14:00 gandiva.pyx >>>>> >>>>> ``` >>>>> >>>>> >>>>> >>>>> On Ubuntu, I see only the Cython file: >>>>> >>>>> ``` >>>>> >>>>> $ l ~/anaconda3/envs/my_model/lib/python3.7/site-packages/pyarrow/ | grep >>>>> gandiva >>>>> >>>>> -rw-rw-r-- 1 xander xander 17K Apr 13 12:28 gandiva.pyx >>>>> >>>>> ``` >>>>> >>>>> >>>>> >>>>> Is this expected? Should I be able to import pyarrow.gandiva on Ubuntu? >>>>> Everything is run on Ubuntu so if I make use of pyarrow.gandiva I'll need >>>>> to figure out how to call it. >>>>> >>>>> >>>>> >>>>> It's mentioned here that it was removed from Python wheels but should >>>>> still be available in the conda install: https:/ / issues. apache. org/ >>>>> jira/ >>>>> browse/ ARROW-10154 ( https://issues.apache.org/jira/browse/ARROW-10154 ). >>>>> I'm not finding it in my Ubuntu conda install. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Xander >>>>> >>>> >>>> >>> >>> >> >> > >
