[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880948#comment-16880948
]
Suvayu Ali commented on ARROW-5871:
---
Hi [~wesmckinn], I read that issue. Unfortunately my experience with conda has
been rather frustrating. I think for production use I'll stick to 0.13.0 for
now, and try to compile from source for experimental use. Unfortunately I have
never successfully managed to compile pyarrow before (no issues with the C++
library though).
Thanks a lot
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
> Fix For: 1.0.0
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ARROW-5719) [Java] Support in-place vector sorting
[ https://issues.apache.org/jira/browse/ARROW-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Kornfield resolved ARROW-5719. Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 4699 [https://github.com/apache/arrow/pull/4699] > [Java] Support in-place vector sorting > -- > > Key: ARROW-5719 > URL: https://issues.apache.org/jira/browse/ARROW-5719 > Project: Apache Arrow > Issue Type: New Feature > Components: Java >Reporter: Liya Fan >Assignee: Liya Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Support in-place sorting for vectors. An in-place sorter sorts the vector by > directly modifying the vector data, so the input and output vectors are the > same one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5868) [Python] manylinux2010 wheels have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-5868: -- Labels: pull-request-available (was: ) > [Python] manylinux2010 wheels have shared library dependency on liblz4 > -- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Assignee: Wes McKinney >Priority: Blocker > Labels: pull-request-available > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5880) [C++] Update arrow parquet writer to use TypedBufferBuilder
[ https://issues.apache.org/jira/browse/ARROW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-5880: -- Labels: Parquet pull-request-available (was: Parquet) > [C++] Update arrow parquet writer to use TypedBufferBuilder > > > Key: ARROW-5880 > URL: https://issues.apache.org/jira/browse/ARROW-5880 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Micah Kornfield >Assignee: Micah Kornfield >Priority: Major > Labels: Parquet, pull-request-available > Fix For: 1.0.0 > > > In looking at the code it seems the writer.cc hasn't been updated to use the > latest classes. there is also what appears to be a small performance bug in > TypedBufferBuilder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5880) [C++] Update arrow parquet writer to use TypedBufferBuilder
[ https://issues.apache.org/jira/browse/ARROW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Kornfield updated ARROW-5880: --- Fix Version/s: 1.0.0 > [C++] Update arrow parquet writer to use TypedBufferBuilder > > > Key: ARROW-5880 > URL: https://issues.apache.org/jira/browse/ARROW-5880 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Micah Kornfield >Assignee: Micah Kornfield >Priority: Major > Fix For: 1.0.0 > > > In looking at the code it seems the writer.cc hasn't been updated to use the > latest classes. there is also what appears to be a small performance bug in > TypedBufferBuilder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5880) [C++] Update arrow parquet writer to use TypedBufferBuilder
[ https://issues.apache.org/jira/browse/ARROW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Kornfield updated ARROW-5880: --- Labels: Parquet (was: ) > [C++] Update arrow parquet writer to use TypedBufferBuilder > > > Key: ARROW-5880 > URL: https://issues.apache.org/jira/browse/ARROW-5880 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Micah Kornfield >Assignee: Micah Kornfield >Priority: Major > Labels: Parquet > Fix For: 1.0.0 > > > In looking at the code it seems the writer.cc hasn't been updated to use the > latest classes. there is also what appears to be a small performance bug in > TypedBufferBuilder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5880) [C++] Update arrow parquet writer to use TypedBufferBuilder
[ https://issues.apache.org/jira/browse/ARROW-5880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Kornfield updated ARROW-5880: --- Component/s: C++ > [C++] Update arrow parquet writer to use TypedBufferBuilder > > > Key: ARROW-5880 > URL: https://issues.apache.org/jira/browse/ARROW-5880 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Micah Kornfield >Assignee: Micah Kornfield >Priority: Major > > In looking at the code it seems the writer.cc hasn't been updated to use the > latest classes. there is also what appears to be a small performance bug in > TypedBufferBuilder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-5880) [C++] Update arrow parquet writer to use TypedBufferBuilder
Micah Kornfield created ARROW-5880: -- Summary: [C++] Update arrow parquet writer to use TypedBufferBuilder Key: ARROW-5880 URL: https://issues.apache.org/jira/browse/ARROW-5880 Project: Apache Arrow Issue Type: Improvement Reporter: Micah Kornfield Assignee: Micah Kornfield In looking at the code it seems the writer.cc hasn't been updated to use the latest classes. there is also what appears to be a small performance bug in TypedBufferBuilder -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880871#comment-16880871
]
Wes McKinney commented on ARROW-5871:
-
Per https://github.com/pypa/manylinux/issues/305 if you want to use Fedora 30
it looks like you are going to have to use conda to install pyarrow (I'm hoping
that won't have this issue). Seems like using Fedora 30 at all might be a bit
premature
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
> Fix For: 1.0.0
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-5868) [Python] manylinux2010 wheels have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney reassigned ARROW-5868: --- Assignee: Wes McKinney > [Python] manylinux2010 wheels have shared library dependency on liblz4 > -- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Assignee: Wes McKinney >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880751#comment-16880751
]
Wes McKinney commented on ARROW-5871:
-
Asking the Powers That Be
https://discuss.python.org/t/libcrypt-so-1-removal-in-fedora-30-impacting-manylinux-builds/1961
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
> Fix For: 1.0.0
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5871:
Fix Version/s: (was: 0.14.1)
1.0.0
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Assignee: Wes McKinney
>Priority: Major
> Fix For: 1.0.0
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney reassigned ARROW-5871:
---
Assignee: (was: Wes McKinney)
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
> Fix For: 1.0.0
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880746#comment-16880746
]
Wes McKinney commented on ARROW-5871:
-
Ah so I thought this was OpenSSL-related but it's not:
https://fedoraproject.org/wiki/Changes/FullyRemoveDeprecatedAndUnsafeFunctionsFromLibcrypt
libcrypto is in OpenSSL, libcrypt is something else.
I don't know how to fix this, but this is not obviously an Apache Arrow
problem. Removing this from 0.14.1 for now until we can obtain more information
about how to deal with this in manylinux*
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Assignee: Wes McKinney
>Priority: Major
> Fix For: 0.14.1
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880743#comment-16880743
]
Wes McKinney commented on ARROW-5871:
-
Appears we are already supposed to be statically linking openssl
https://github.com/apache/arrow/blob/master/python/manylinux1/build_arrow.sh#L103
Very frustrating.
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Assignee: Wes McKinney
>Priority: Major
> Fix For: 0.14.1
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney reassigned ARROW-5871:
---
Assignee: Wes McKinney
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Assignee: Wes McKinney
>Priority: Major
> Fix For: 0.14.1
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] manylinux2010 wheels have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880735#comment-16880735 ] Haowei Yu commented on ARROW-5868: -- Ok I tried and it seems working. Thanks for the quick response. > [Python] manylinux2010 wheels have shared library dependency on liblz4 > -- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880734#comment-16880734
]
Wes McKinney commented on ARROW-5871:
-
Strangely this seems to only be a problem with the Python 3.7 manylinux1 wheel
{code}
(pyarrow-0.14-wheel-3.7) 16:50 ~/code/arrow (ARROW-5863)$ ldd
~/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/lib.cpython-36m-x86_64-linux-gnu.so
| grep crypt
(pyarrow-0.14-wheel-3.7) 16:50 ~/code/arrow (ARROW-5863)$ ldd
~/miniconda/envs/pyarrow-0.14-wheel-3.7/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-x86_64-linux-gnu.so
| grep crypt
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1
(0x7f0e4e392000)
{code}
It seems that we will need to link OpenSSL static libraries as with
https://github.com/apache/arrow/commit/00505b55625ed47d239ca2fb8087bd708097b414
Really I wish we could stop using wheels, it would allow people like me to get
back to building new things instead of debugging this stuff
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
> Fix For: 0.14.1
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5868) [Python] manylinux2010 wheels have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5868: Summary: [Python] manylinux2010 wheels have shared library dependency on liblz4 (was: [Python] manylinux2010 wheels may have shared library dependency on liblz4) > [Python] manylinux2010 wheels have shared library dependency on liblz4 > -- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] manylinux2010 wheels may have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880728#comment-16880728 ] Wes McKinney commented on ARROW-5868: - manylinux2010 wheels have been pulled from PyPI. If you reinstall and get the manylinux1 wheel it _should_ work for you > [Python] manylinux2010 wheels may have shared library dependency on liblz4 > -- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] manylinux2010 wheels may have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880723#comment-16880723 ] Wes McKinney commented on ARROW-5868: - manylinux1 wheels are OK. I'm going to check some other things, but I'm probably going to revoke the manylinux2010 wheels > [Python] manylinux2010 wheels may have shared library dependency on liblz4 > -- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] Some manylinux* wheels may have shared library dependency on liblz4
[
https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880722#comment-16880722
]
Wes McKinney commented on ARROW-5868:
-
Confirmed that the manylinux2010 wheel at least has a shared library dependency
on liblz4
{code}
$ ldd
~/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow.so.14
linux-vdso.so.1 (0x7ffe6974d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f1de9c68000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f1de9c5d000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x7f1de9c2e000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x7f1de9c12000)
libarrow_boost_system.so.1.68.0 =>
/home/wesm/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow_boost_system.so.1.68.0
(0x7f1de99fc000)
libarrow_boost_filesystem.so.1.68.0 =>
/home/wesm/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow_boost_filesystem.so.1.68.0
(0x7f1de97df000)
libarrow_boost_regex.so.1.68.0 =>
/home/wesm/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow_boost_regex.so.1.68.0
(0x7f1de94e3000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
(0x7f1de9301000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f1de91b3000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x7f1de9199000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x7f1de9178000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f1de8f8b000)
/lib64/ld-linux-x86-64.so.2 (0x7f1deab1e000)
{code}
Checking manylinux1, then will look at the Docker image to see what's going on
> [Python] Some manylinux* wheels may have shared library dependency on liblz4
>
>
> Key: ARROW-5868
> URL: https://issues.apache.org/jira/browse/ARROW-5868
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
>Reporter: Haowei Yu
>Priority: Blocker
> Fix For: 0.14.1
>
>
> I am using pyarrow in my project. It works well for version 0.13.0
> However, it seems recently there is a release for 0.14.0. After upgrading to
> the latest, I got this error.
> AttributeError: module 'pyarrow' has no attribute 'compat'
> Stacktrace:
> 2019-07-06 09:08:21 Traceback (most recent call last):
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py",
> line 12, in
> 2019-07-06 09:08:21 import snowflake.connector
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py",
> line 21, in
> 2019-07-06 09:08:21 from .connection import SnowflakeConnection
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py",
> line 42, in
> 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py",
> line 35, in
> 2019-07-06 09:08:21 from pyarrow.ipc import open_stream
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py",
> line 47, in
> 2019-07-06 09:08:21 import pyarrow.compat as compat
> I can provide more detail if requested.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5868) [Python] manylinux2010 wheels may have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5868: Summary: [Python] manylinux2010 wheels may have shared library dependency on liblz4 (was: [Python] Some manylinux* wheels may have shared library dependency on liblz4) > [Python] manylinux2010 wheels may have shared library dependency on liblz4 > -- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5525) [C++][CI] Enable continuous fuzzing
[ https://issues.apache.org/jira/browse/ARROW-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5525: Summary: [C++][CI] Enable continuous fuzzing (was: Enable continuous fuzzing) > [C++][CI] Enable continuous fuzzing > --- > > Key: ARROW-5525 > URL: https://issues.apache.org/jira/browse/ARROW-5525 > Project: Apache Arrow > Issue Type: Test > Components: C++ >Reporter: Marco Neumann >Assignee: Yevgeny Pats >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Since fuzzing kinda only works if done as a continuous background job, we > should find a way of doing so. This likely requires another service than > Travis. Basic requirements are: > * master builds should be submitted for fuzzing > * project members should be informed about new crashes (ideally not via > public issue due to potential security impact) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ARROW-5863) [Python] Segmentation Fault via pytest-runner
[
https://issues.apache.org/jira/browse/ARROW-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney resolved ARROW-5863.
-
Resolution: Fixed
Issue resolved by pull request 4824
[https://github.com/apache/arrow/pull/4824]
> [Python] Segmentation Fault via pytest-runner
> -
>
> Key: ARROW-5863
> URL: https://issues.apache.org/jira/browse/ARROW-5863
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: $ uname -a
> Linux aleph 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019
> x86_64 GNU/Linux
> $ python --version
> Python 3.7.3
> $ pip freeze | grep -P "(pyarrow|pytest)"
> pyarrow==0.14.0
> pytest==5.0.0
> pytest-benchmark==3.2.2
> pytest-cov==2.7.1
> pytest-env==0.6.2
> pytest-forked==1.0.2
> pytest-html==1.21.1
> pytest-metadata==1.8.0
> pytest-mock==1.10.4
> pytest-runner==5.1
> pytest-sugar==0.9.2
> pytest-xdist==1.29.0
>Reporter: Josh Bode
>Assignee: Wes McKinney
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0, 0.14.1
>
> Attachments: pyarrow-issue.tar.bz2, pytest-runner.log, pytest.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When running {{pytest}} on projects using {{pyarrow==0.14.0}} on Linux, I am
> getting segmentation faults, but interestingly _only_ when run via
> {{pytest-runner}} (which provides the {{setup.py pytest}} command)
> This works (i.e. {{pytest}} directly):
> {code:java}
> $ pytest
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.09s):
> 1 passed{code}
> However, this does not work, ending in a segmentation fault, even though the
> tests pass:
> {code:java}
> $ python setup.py pytest
> running pytest
> running egg_info
> writing pyarrow_issue.egg-info/PKG-INFO
> writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
> writing requirements to pyarrow_issue.egg-info/requires.txt
> writing top-level names to pyarrow_issue.egg-info/top_level.txt
> reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> running build_ext
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.07s):
> 1 passed
> zsh: segmentation fault (core dumped) python setup.py pytest{code}
> backtrace from {{gdb}}
> {code:java}
> Thread 1 "python" received signal SIGSEGV, Segmentation fault.
> 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> (gdb) bt
> #0 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> #1 0x77ae46cc in ?? () from /usr/lib/libpython3.7m.so.1.0
> #2 0x7023a6b3 in arrow::py::PyExtensionType::~PyExtensionType() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow_python.so.14
> #3 0x7fffed5e6467 in std::unordered_map std::shared_ptr, std::hash,
> std::equal_to, std::allocator std::shared_ptr > > >::~unordered_map() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow.so.14
> #4 0x77de5e70 in __run_exit_handlers () from /usr/lib/libc.so.6
> #5 0x77de5fae in exit () from /usr/lib/libc.so.6
> #6 0x77dcfeea in __libc_start_main () from /usr/lib/libc.so.6
> #7 0x505e in _start ()
> {code}
> I have observed this behaviour on my machine running natively, and also via
> docker. Also, 0.13.0 does not exhibit this behaviour
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5839) [Python] Test manylinux2010 in CI
[ https://issues.apache.org/jira/browse/ARROW-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5839: Fix Version/s: 1.0.0 > [Python] Test manylinux2010 in CI > - > > Key: ARROW-5839 > URL: https://issues.apache.org/jira/browse/ARROW-5839 > Project: Apache Arrow > Issue Type: Task > Components: Python >Reporter: Antoine Pitrou >Priority: Major > Fix For: 1.0.0 > > > Currently we test manylinux1 builds on Travis-CI. At some point we should > test manylinux2010 builds too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5839) [Python] Test manylinux2010 in CI
[ https://issues.apache.org/jira/browse/ARROW-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880714#comment-16880714 ] Wes McKinney commented on ARROW-5839: - Ideally in docker-compose so we can move them both out of Travis CI > [Python] Test manylinux2010 in CI > - > > Key: ARROW-5839 > URL: https://issues.apache.org/jira/browse/ARROW-5839 > Project: Apache Arrow > Issue Type: Task > Components: Python >Reporter: Antoine Pitrou >Priority: Major > > Currently we test manylinux1 builds on Travis-CI. At some point we should > test manylinux2010 builds too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5790) [Python] Passing zero-dim numpy array to pa.array causes segfault
[ https://issues.apache.org/jira/browse/ARROW-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5790: Fix Version/s: 0.14.1 > [Python] Passing zero-dim numpy array to pa.array causes segfault > - > > Key: ARROW-5790 > URL: https://issues.apache.org/jira/browse/ARROW-5790 > Project: Apache Arrow > Issue Type: Bug > Components: Python > Environment: OSX, py37 >Reporter: Brock Mendel >Priority: Major > Fix For: 1.0.0, 0.14.1 > > > ``` > import pyarrow as pa > import numpy as np > zerod = np.array(0) > result = pa.array(zerod) # <-- segfault > ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5825) [Python] Exceptions swallowed in ParquetManifest._visit_directories
[
https://issues.apache.org/jira/browse/ARROW-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5825:
Fix Version/s: 1.0.0
> [Python] Exceptions swallowed in ParquetManifest._visit_directories
> ---
>
> Key: ARROW-5825
> URL: https://issues.apache.org/jira/browse/ARROW-5825
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: George Sakkis
>Priority: Major
> Labels: parquet
> Fix For: 1.0.0
>
>
> {{ParquetManifest._visit_directories}} uses a {{ThreadPoolExecutor}} to visit
> partitioned parquet datasets concurrently, it waits for them to finish but
> doesn't check if the respective futures have failed or not. This is quite
> tricky to detect and debug as an exception is either raised later as a a
> side-effect or (perhaps worse) it passes silently.
> Observed on 0.12.1 but appears to be on latest master too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] Some manylinux* wheels may have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880712#comment-16880712 ] Wes McKinney commented on ARROW-5868: - Something seems to be wrong with the C++ build, someone will have to take a closer look at the wheels themselves and the build to understand what is going on > [Python] Some manylinux* wheels may have shared library dependency on liblz4 > > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5859) [Python] Support ExtentionType on conversion to numpy/pandas
[
https://issues.apache.org/jira/browse/ARROW-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5859:
Fix Version/s: 1.0.0
> [Python] Support ExtentionType on conversion to numpy/pandas
>
>
> Key: ARROW-5859
> URL: https://issues.apache.org/jira/browse/ARROW-5859
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Joris Van den Bossche
>Priority: Major
> Fix For: 1.0.0
>
>
> Currently converting a Table of RecordBatch with an ExtensionType array to
> pandas gives:
> {code}
> ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of
> type extension is known.
> {code}
> And similarly converting the array itself to a python object (to_pandas or
> to_pylist) gives an ArrowNotImplementedError or a "KeyError: 28"
> Initial support could be to fall back to the storage type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] Some manylinux* wheels may have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880707#comment-16880707 ] Haowei Yu commented on ARROW-5868: -- Then it is weird. I am not familiar with this but .so seems to be a dynamic linked library. Python somehow tries to import a dynamic library? Or maybe that is a secondary dependency? > [Python] Some manylinux* wheels may have shared library dependency on liblz4 > > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5868) [Python] AttributeError: module 'pyarrow' has no attribute 'compat'
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5868: Fix Version/s: 0.14.1 > [Python] AttributeError: module 'pyarrow' has no attribute 'compat' > --- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Major > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5868) [Python] Some manylinux* wheels may have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5868: Priority: Blocker (was: Major) > [Python] Some manylinux* wheels may have shared library dependency on liblz4 > > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Blocker > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5868) [Python] Some manylinux* wheels may have shared library dependency on liblz4
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5868: Summary: [Python] Some manylinux* wheels may have shared library dependency on liblz4 (was: [Python] AttributeError: module 'pyarrow' has no attribute 'compat') > [Python] Some manylinux* wheels may have shared library dependency on liblz4 > > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Major > Fix For: 0.14.1 > > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] AttributeError: module 'pyarrow' has no attribute 'compat'
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880702#comment-16880702 ] Haowei Yu commented on ARROW-5868: -- I use pip, not using conda. I have a virtual env on python3.6. Is liblz4.so.1 bundled with pyarrow package? > [Python] AttributeError: module 'pyarrow' has no attribute 'compat' > --- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Major > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5878) [Python][C++] Parquet reader not forward compatible for timestamps without timezone
[
https://issues.apache.org/jira/browse/ARROW-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-5878:
--
Labels: pull-request-available (was: )
> [Python][C++] Parquet reader not forward compatible for timestamps without
> timezone
> ---
>
> Key: ARROW-5878
> URL: https://issues.apache.org/jira/browse/ARROW-5878
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Florian Jetter
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0, 0.14.1
>
> Attachments: timezones_pyarrow_14.paquet
>
>
> Timestamps without timezone which are written by pyarrow 0.14.0 cannot be
> read anymore as timestamps by earlier versions. The timestamp is read as an
> integer when reading in with pyarrow 0.13.0
> Looking at the parquet schemas, it seems that the logical type cannot be
> understood by the older versions, see below.
> h4. File generation with pyarrow 0.14.0
> {code:java}
> import datetime
> import pyarrow.parquet as pq
> import pandas as pd
> df = pd.DataFrame(
> {
> "datetime64": pd.Series(["2018-01-01"], dtype="datetime64[ns]"),
> "datetime64_ts": pd.Series(
> [pd.Timestamp(datetime.datetime(2018, 1, 1), tz="Europe/Berlin")],
> dtype="datetime64[ns]",
> ),
> }
> )
> pq.write_table(pa.Table.from_pandas(df), "timezones_pyarrow_14.paquet")
> {code}
> h4. Reading with pyarrow 0.13.0
> {code:java}
> In [1]: import pyarrow.parquet as pq
> In [2]: import pyarrow as pa
> In [3]: with open("timezones_pyarrow_14.paquet", "rb") as fd:
>...: table = pq.read_pandas(fd)
>...:
> In [4]: table.to_pandas()
> Out[4]:
> datetime64 datetime64_ts
> 0 15147648 2018-01-01 00:00:00+01:00
> In [5]: table.to_pandas().dtypes
> Out[5]:
> datetime64 int64
> datetime64_tsdatetime64[ns, Europe/Berlin]
> dtype: object
> {code}
> h3. Parquet schema as seen by pyarrow versions:
> pyarrow 0.13.0 parquet schema
> {code:java}
> datetime64: INT64
> datetime64_ts: INT64 TIMESTAMP_MICROS
> {code}
> pyarrow 0.14.0 parquet schema
> {code:java}
> datetime64: INT64 Timestamp(isAdjustedToUTC=false, timeUnit=microseconds)
> datetime64_ts: INT64 Timestamp(isAdjustedToUTC=true, timeUnit=microseconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] AttributeError: module 'pyarrow' has no attribute 'compat'
[ https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880699#comment-16880699 ] Wes McKinney commented on ARROW-5868: - How did you install pyarrow, with pip or conda (it looks like with conda)? > [Python] AttributeError: module 'pyarrow' has no attribute 'compat' > --- > > Key: ARROW-5868 > URL: https://issues.apache.org/jira/browse/ARROW-5868 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Haowei Yu >Priority: Major > > I am using pyarrow in my project. It works well for version 0.13.0 > However, it seems recently there is a release for 0.14.0. After upgrading to > the latest, I got this error. > AttributeError: module 'pyarrow' has no attribute 'compat' > Stacktrace: > 2019-07-06 09:08:21 Traceback (most recent call last): > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", > line 12, in > 2019-07-06 09:08:21 import snowflake.connector > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py", > line 21, in > 2019-07-06 09:08:21 from .connection import SnowflakeConnection > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", > line 42, in > 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", > line 35, in > 2019-07-06 09:08:21 from pyarrow.ipc import open_stream > 2019-07-06 09:08:21 File > "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py", > line 47, in > 2019-07-06 09:08:21 import pyarrow.compat as compat > I can provide more detail if requested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5868) [Python] AttributeError: module 'pyarrow' has no attribute 'compat'
[
https://issues.apache.org/jira/browse/ARROW-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880696#comment-16880696
]
Haowei Yu commented on ARROW-5868:
--
It failed with following:
{code:java}
(py36env) [hyu@c6x64-hyu-newuser-final-clone trunk]$ python -c "import pyarrow"
Traceback (most recent call last):
File "", line 1, in
File "/home/hyu/py36env/lib/python3.6/site-packages/pyarrow/__init__.py", line
49, in
from pyarrow.lib import cpu_count, set_cpu_count
ImportError: liblz4.so.1: cannot open shared object file: No such file or
directory{code}
> [Python] AttributeError: module 'pyarrow' has no attribute 'compat'
> ---
>
> Key: ARROW-5868
> URL: https://issues.apache.org/jira/browse/ARROW-5868
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
>Reporter: Haowei Yu
>Priority: Major
>
> I am using pyarrow in my project. It works well for version 0.13.0
> However, it seems recently there is a release for 0.14.0. After upgrading to
> the latest, I got this error.
> AttributeError: module 'pyarrow' has no attribute 'compat'
> Stacktrace:
> 2019-07-06 09:08:21 Traceback (most recent call last):
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py",
> line 12, in
> 2019-07-06 09:08:21 import snowflake.connector
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/__init__.py",
> line 21, in
> 2019-07-06 09:08:21 from .connection import SnowflakeConnection
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py",
> line 42, in
> 2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py",
> line 35, in
> 2019-07-06 09:08:21 from pyarrow.ipc import open_stream
> 2019-07-06 09:08:21 File
> "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/__init__.py",
> line 47, in
> 2019-07-06 09:08:21 import pyarrow.compat as compat
> I can provide more detail if requested.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (ARROW-5847) [C++] Remove lazy.h
[ https://issues.apache.org/jira/browse/ARROW-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney closed ARROW-5847. --- > [C++] Remove lazy.h > --- > > Key: ARROW-5847 > URL: https://issues.apache.org/jira/browse/ARROW-5847 > Project: Apache Arrow > Issue Type: Task > Components: C++ >Reporter: Antoine Pitrou >Priority: Trivial > > This header file, its tests and benchmarks, are unused. Remove them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-5852) [Rust] [DataFusion] Compilation error with Rust 1.37.0-nightly
[
https://issues.apache.org/jira/browse/ARROW-5852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney closed ARROW-5852.
---
> [Rust] [DataFusion] Compilation error with Rust 1.37.0-nightly
> --
>
> Key: ARROW-5852
> URL: https://issues.apache.org/jira/browse/ARROW-5852
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Rust, Rust - DataFusion
>Affects Versions: 0.14.0
>Reporter: Andy Grove
>Assignee: Andy Grove
>Priority: Major
>
> {code:java}
> error[E0283]: type annotations required: cannot resolve `std::string::String:
> std::convert::AsRef<_>`
> -->
> /home/andy/.cargo/registry/src/github.com-1ecc6299db9ec823/rustyline-3.0.0/src/lib.rs:655:43
> |
> 655 | editor.add_history_entry(line.as_ref());
> | ^^
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (ARROW-3491) [C++] Experiment with split DWARF
[ https://issues.apache.org/jira/browse/ARROW-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney closed ARROW-3491. --- > [C++] Experiment with split DWARF > - > > Key: ARROW-3491 > URL: https://issues.apache.org/jira/browse/ARROW-3491 > Project: Apache Arrow > Issue Type: Wish > Components: C++ >Reporter: Antoine Pitrou >Priority: Minor > > From the article at > http://www.productive-cpp.com/improving-cpp-builds-with-split-dwarf/, it > should be possible to split debug information from .o files to have faster > C++ builds and smaller disk footprint of libraries / executables. This is > useful for development builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-3253) [CI] Investigate Azure CI
[ https://issues.apache.org/jira/browse/ARROW-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney closed ARROW-3253. --- > [CI] Investigate Azure CI > - > > Key: ARROW-3253 > URL: https://issues.apache.org/jira/browse/ARROW-3253 > Project: Apache Arrow > Issue Type: Task > Components: C++, Continuous Integration >Reporter: Antoine Pitrou >Priority: Major > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > C++ builds on AppVeyor have become slower and slower. Some of it may be due > to the parquet-cpp repository merge, but I also suspect CPU resources on > AppVeyor have become much tighter. > We should perhaps investigate Microsoft's Azure CI services as an alternative: > https://azure.microsoft.com/en-gb/services/devops/pipelines/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-5837) [Release][OSX] Verification script fails to activate conda environment
[ https://issues.apache.org/jira/browse/ARROW-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney closed ARROW-5837. --- > [Release][OSX] Verification script fails to activate conda environment > --- > > Key: ARROW-5837 > URL: https://issues.apache.org/jira/browse/ARROW-5837 > Project: Apache Arrow > Issue Type: Bug > Components: Packaging >Affects Versions: 0.14.0 >Reporter: Krisztian Szucs >Priority: Major > > Faced issue https://github.com/conda/conda/issues/8186 > I was using it on OSX, within tmux and zsh (probably caused by the nested > shells). > Solution is to export PS1="" at > https://github.com/apache/arrow/blob/master/dev/release/verify-release-candidate.sh#L236 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-3165) [CI] Cache externals accross AppVeyor builds
[ https://issues.apache.org/jira/browse/ARROW-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney closed ARROW-3165. --- > [CI] Cache externals accross AppVeyor builds > > > Key: ARROW-3165 > URL: https://issues.apache.org/jira/browse/ARROW-3165 > Project: Apache Arrow > Issue Type: Improvement > Components: C++, Continuous Integration >Reporter: Antoine Pitrou >Priority: Major > > Downloading and building third-party libraries may be consuming a significant > chunk of time in our CI chain. It would be nice to cache third-party build > artifacts accross CI builds, to save time. > Unfortunately I ran accross weird issues (especially on Windows / AppVeyor) > when trying to implement this. I don't plan to waste any more time on this > soon. Attempt at > [https://github.com/apache/arrow/compare/master...pitrou:externals_dir] > Edit: it seems I finally found the issue on Windows: ninja is case-sensitive, > you must be careful to canonicalize paths yourself in custom CMake commands > such as in ExternalProject_Add... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-5840) [C++][OSX] Unused variable warning in python/common.h in release mode
[
https://issues.apache.org/jira/browse/ARROW-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney closed ARROW-5840.
---
> [C++][OSX] Unused variable warning in python/common.h in release mode
> -
>
> Key: ARROW-5840
> URL: https://issues.apache.org/jira/browse/ARROW-5840
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Krisztian Szucs
>Priority: Minor
>
> {code}
> /var/folders/cz/jrwncy5s5cb612sgwscd0z8hgn/T/arrow-0.14.0.X.HSei2zjI/apache-arrow-0.14.0/cpp/src/arrow/python/common.h:66:9:
> warning: unused variable 'r' [-Wunused-variable]
> int r = internal::check_status(result.status());
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-458) [Python] Expose jemalloc MemoryPool
[
https://issues.apache.org/jira/browse/ARROW-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-458:
---
Summary: [Python] Expose jemalloc MemoryPool (was: Python: Expose jemalloc
MemoryPool)
> [Python] Expose jemalloc MemoryPool
> ---
>
> Key: ARROW-458
> URL: https://issues.apache.org/jira/browse/ARROW-458
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Uwe L. Korn
>Assignee: Uwe L. Korn
>Priority: Major
> Fix For: 0.2.0
>
>
> Expose the {{jemalloc::MemoryPool}} to Python users as a separate
> {{pyarrow.jemalloc}} module.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5863) [Python] Segmentation Fault via pytest-runner
[
https://issues.apache.org/jira/browse/ARROW-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-5863:
--
Labels: pull-request-available (was: )
> [Python] Segmentation Fault via pytest-runner
> -
>
> Key: ARROW-5863
> URL: https://issues.apache.org/jira/browse/ARROW-5863
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: $ uname -a
> Linux aleph 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019
> x86_64 GNU/Linux
> $ python --version
> Python 3.7.3
> $ pip freeze | grep -P "(pyarrow|pytest)"
> pyarrow==0.14.0
> pytest==5.0.0
> pytest-benchmark==3.2.2
> pytest-cov==2.7.1
> pytest-env==0.6.2
> pytest-forked==1.0.2
> pytest-html==1.21.1
> pytest-metadata==1.8.0
> pytest-mock==1.10.4
> pytest-runner==5.1
> pytest-sugar==0.9.2
> pytest-xdist==1.29.0
>Reporter: Josh Bode
>Assignee: Wes McKinney
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0, 0.14.1
>
> Attachments: pyarrow-issue.tar.bz2, pytest-runner.log, pytest.log
>
>
> When running {{pytest}} on projects using {{pyarrow==0.14.0}} on Linux, I am
> getting segmentation faults, but interestingly _only_ when run via
> {{pytest-runner}} (which provides the {{setup.py pytest}} command)
> This works (i.e. {{pytest}} directly):
> {code:java}
> $ pytest
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.09s):
> 1 passed{code}
> However, this does not work, ending in a segmentation fault, even though the
> tests pass:
> {code:java}
> $ python setup.py pytest
> running pytest
> running egg_info
> writing pyarrow_issue.egg-info/PKG-INFO
> writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
> writing requirements to pyarrow_issue.egg-info/requires.txt
> writing top-level names to pyarrow_issue.egg-info/top_level.txt
> reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> running build_ext
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.07s):
> 1 passed
> zsh: segmentation fault (core dumped) python setup.py pytest{code}
> backtrace from {{gdb}}
> {code:java}
> Thread 1 "python" received signal SIGSEGV, Segmentation fault.
> 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> (gdb) bt
> #0 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> #1 0x77ae46cc in ?? () from /usr/lib/libpython3.7m.so.1.0
> #2 0x7023a6b3 in arrow::py::PyExtensionType::~PyExtensionType() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow_python.so.14
> #3 0x7fffed5e6467 in std::unordered_map std::shared_ptr, std::hash,
> std::equal_to, std::allocator std::shared_ptr > > >::~unordered_map() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow.so.14
> #4 0x77de5e70 in __run_exit_handlers () from /usr/lib/libc.so.6
> #5 0x77de5fae in exit () from /usr/lib/libc.so.6
> #6 0x77dcfeea in __libc_start_main () from /usr/lib/libc.so.6
> #7 0x505e in _start ()
> {code}
> I have observed this behaviour on my machine running natively, and also via
> docker. Also, 0.13.0 does not exhibit this behaviour
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-5863) [Python] Segmentation Fault via pytest-runner
[
https://issues.apache.org/jira/browse/ARROW-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney reassigned ARROW-5863:
---
Assignee: Wes McKinney
> [Python] Segmentation Fault via pytest-runner
> -
>
> Key: ARROW-5863
> URL: https://issues.apache.org/jira/browse/ARROW-5863
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: $ uname -a
> Linux aleph 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019
> x86_64 GNU/Linux
> $ python --version
> Python 3.7.3
> $ pip freeze | grep -P "(pyarrow|pytest)"
> pyarrow==0.14.0
> pytest==5.0.0
> pytest-benchmark==3.2.2
> pytest-cov==2.7.1
> pytest-env==0.6.2
> pytest-forked==1.0.2
> pytest-html==1.21.1
> pytest-metadata==1.8.0
> pytest-mock==1.10.4
> pytest-runner==5.1
> pytest-sugar==0.9.2
> pytest-xdist==1.29.0
>Reporter: Josh Bode
>Assignee: Wes McKinney
>Priority: Major
> Fix For: 1.0.0, 0.14.1
>
> Attachments: pyarrow-issue.tar.bz2, pytest-runner.log, pytest.log
>
>
> When running {{pytest}} on projects using {{pyarrow==0.14.0}} on Linux, I am
> getting segmentation faults, but interestingly _only_ when run via
> {{pytest-runner}} (which provides the {{setup.py pytest}} command)
> This works (i.e. {{pytest}} directly):
> {code:java}
> $ pytest
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.09s):
> 1 passed{code}
> However, this does not work, ending in a segmentation fault, even though the
> tests pass:
> {code:java}
> $ python setup.py pytest
> running pytest
> running egg_info
> writing pyarrow_issue.egg-info/PKG-INFO
> writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
> writing requirements to pyarrow_issue.egg-info/requires.txt
> writing top-level names to pyarrow_issue.egg-info/top_level.txt
> reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> running build_ext
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.07s):
> 1 passed
> zsh: segmentation fault (core dumped) python setup.py pytest{code}
> backtrace from {{gdb}}
> {code:java}
> Thread 1 "python" received signal SIGSEGV, Segmentation fault.
> 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> (gdb) bt
> #0 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> #1 0x77ae46cc in ?? () from /usr/lib/libpython3.7m.so.1.0
> #2 0x7023a6b3 in arrow::py::PyExtensionType::~PyExtensionType() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow_python.so.14
> #3 0x7fffed5e6467 in std::unordered_map std::shared_ptr, std::hash,
> std::equal_to, std::allocator std::shared_ptr > > >::~unordered_map() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow.so.14
> #4 0x77de5e70 in __run_exit_handlers () from /usr/lib/libc.so.6
> #5 0x77de5fae in exit () from /usr/lib/libc.so.6
> #6 0x77dcfeea in __libc_start_main () from /usr/lib/libc.so.6
> #7 0x505e in _start ()
> {code}
> I have observed this behaviour on my machine running natively, and also via
> docker. Also, 0.13.0 does not exhibit this behaviour
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5527) [C++] HashTable/MemoTable should use Buffer(s)/Builder(s) for heap data
[ https://issues.apache.org/jira/browse/ARROW-5527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francois Saint-Jacques updated ARROW-5527: -- Description: The current implementation uses `std::vector` and `std::string` with unbounded size. The refactor would take a memory pool in the constructor for buffer management and would get rid of vectors. This will have the side effect of propagating Status to some calls (notably insert due to Upsize failing to resize). * MemoTable constructor needs to take a MemoryPool in input * GetOrInsert must return Status/Result * MemoTable should use a TypeBufferBuilder instead of std::vector * BinaryMemoTable should use a BinaryBuilder instead of (std::vector, std::string) pair. was:The current implementation uses `std::vector` and `std::string` with unbounded size. The refactor would take a memory pool in the constructor for buffer management and would get rid of vectors. This will have the side effect of propagating Status to some calls (notably insert due to Upsize failing to resize). > [C++] HashTable/MemoTable should use Buffer(s)/Builder(s) for heap data > --- > > Key: ARROW-5527 > URL: https://issues.apache.org/jira/browse/ARROW-5527 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Francois Saint-Jacques >Assignee: Francois Saint-Jacques >Priority: Major > > The current implementation uses `std::vector` and `std::string` with > unbounded size. The refactor would take a memory pool in the constructor for > buffer management and would get rid of vectors. This will have the side > effect of propagating Status to some calls (notably insert due to Upsize > failing to resize). > * MemoTable constructor needs to take a MemoryPool in input > * GetOrInsert must return Status/Result > * MemoTable should use a TypeBufferBuilder instead of std::vector > * BinaryMemoTable should use a BinaryBuilder instead of > (std::vector, std::string) pair. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5527) [C++] HashTable/MemoTable should use Buffer(s)/Builder(s) for heap data
[ https://issues.apache.org/jira/browse/ARROW-5527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francois Saint-Jacques updated ARROW-5527: -- Description: The current implementation uses `std::vector` and `std::string` with unbounded size. The refactor would take a memory pool in the constructor for buffer management and would get rid of vectors. This will have the side effect of propagating Status to some calls (notably insert due to Upsize failing to resize). (was: The current implementation uses `std::vector` and `std::string` with unbounded size. The refactor would take a memory pool in the constructor for buffer management and would get rid of vectors. This will have the side effect of propagating Status to some calls (notably insert due to Upsize failing to resize).) > [C++] HashTable/MemoTable should use Buffer(s)/Builder(s) for heap data > --- > > Key: ARROW-5527 > URL: https://issues.apache.org/jira/browse/ARROW-5527 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Francois Saint-Jacques >Assignee: Francois Saint-Jacques >Priority: Major > > The current implementation uses `std::vector` and `std::string` with > unbounded size. The refactor would take a memory pool in the constructor for > buffer management and would get rid of vectors. This will have the side > effect of propagating Status to some calls (notably insert due to Upsize > failing to resize). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ARROW-5525) Enable continuous fuzzing
[ https://issues.apache.org/jira/browse/ARROW-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney reassigned ARROW-5525: --- Assignee: Yevgeny Pats > Enable continuous fuzzing > - > > Key: ARROW-5525 > URL: https://issues.apache.org/jira/browse/ARROW-5525 > Project: Apache Arrow > Issue Type: Test > Components: C++ >Reporter: Marco Neumann >Assignee: Yevgeny Pats >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > Since fuzzing kinda only works if done as a continuous background job, we > should find a way of doing so. This likely requires another service than > Travis. Basic requirements are: > * master builds should be submitted for fuzzing > * project members should be informed about new crashes (ideally not via > public issue due to potential security impact) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5873) [Python][C++] Segmentation fault when comparing schema with None
[
https://issues.apache.org/jira/browse/ARROW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880634#comment-16880634
]
Joris Van den Bossche commented on ARROW-5873:
--
Thanks for the report. I can confirm the segfault (although I also see it with
0.13.0)
> [Python][C++] Segmentation fault when comparing schema with None
>
>
> Key: ARROW-5873
> URL: https://issues.apache.org/jira/browse/ARROW-5873
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Florian Jetter
>Priority: Minor
> Fix For: 1.0.0
>
>
> When comparing a schema with a Python {{None}} I get a segmentation fault.
> This is a regression to 0.13.0
> {code:java}
> In [2]: import pyarrow as pa
> In [3]: pa.schema([pa.field("something", pa.int64())]).equals(None)
> [1]82085 segmentation fault ipython
> {code}
> System information:
> System Version: macOS 10.13.6 (17G6030)
> Kernel Version: Darwin 17.7.0
> Python 3.6.7
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5300) [C++] 0.13 FAILED to build with option -DARROW_NO_DEFAULT_MEMORY_POOL
[
https://issues.apache.org/jira/browse/ARROW-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880631#comment-16880631
]
Wes McKinney commented on ARROW-5300:
-
I don't feel strongly about maintaining this build option, maybe we should just
remove it.
> [C++] 0.13 FAILED to build with option -DARROW_NO_DEFAULT_MEMORY_POOL
> -
>
> Key: ARROW-5300
> URL: https://issues.apache.org/jira/browse/ARROW-5300
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 0.13.0
>Reporter: Weihua Jiang
>Priority: Major
> Fix For: 1.0.0
>
>
> I tried to upgrade Apache Arrow to 0.13. But, when building Apache Arrow 0.13
> with option {{-DARROW_NO_DEFAULT_MEMORY_POOL}}, I got a lot of failures.
> It seems 0.13 assuming default memory pool always available.
>
> My cmake command is:
> |{{make .. -DCMAKE_BUILD_TYPE=Release -DARROW_BUILD_TESTS=off
> -DARROW_USE_GLOG=off -DARROW_WITH_LZ4=off -DARROW_WITH_ZSTD=off
> -DARROW_WITH_SNAPPY=off -DARROW_WITH_BROTLI=off -DARROW_WITH_ZLIB=off
> -DARROW_JEMALLOC=off -DARROW_CXXFLAGS=-DARROW_NO_DEFAULT_MEMORY_POOL}}|
> I tried to fix the compilation by adding some missing constructors. However,
> it seems this issue is bigger than I expected. It seems all the builders and
> appenders have this issue as many classes even don't have a memory pool
> associated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5300) [C++] 0.13 FAILED to build with option -DARROW_NO_DEFAULT_MEMORY_POOL
[
https://issues.apache.org/jira/browse/ARROW-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5300:
Fix Version/s: 1.0.0
> [C++] 0.13 FAILED to build with option -DARROW_NO_DEFAULT_MEMORY_POOL
> -
>
> Key: ARROW-5300
> URL: https://issues.apache.org/jira/browse/ARROW-5300
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 0.13.0
>Reporter: Weihua Jiang
>Priority: Major
> Fix For: 1.0.0
>
>
> I tried to upgrade Apache Arrow to 0.13. But, when building Apache Arrow 0.13
> with option {{-DARROW_NO_DEFAULT_MEMORY_POOL}}, I got a lot of failures.
> It seems 0.13 assuming default memory pool always available.
>
> My cmake command is:
> |{{make .. -DCMAKE_BUILD_TYPE=Release -DARROW_BUILD_TESTS=off
> -DARROW_USE_GLOG=off -DARROW_WITH_LZ4=off -DARROW_WITH_ZSTD=off
> -DARROW_WITH_SNAPPY=off -DARROW_WITH_BROTLI=off -DARROW_WITH_ZLIB=off
> -DARROW_JEMALLOC=off -DARROW_CXXFLAGS=-DARROW_NO_DEFAULT_MEMORY_POOL}}|
> I tried to fix the compilation by adding some missing constructors. However,
> it seems this issue is bigger than I expected. It seems all the builders and
> appenders have this issue as many classes even don't have a memory pool
> associated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ARROW-5525) Enable continuous fuzzing
[ https://issues.apache.org/jira/browse/ARROW-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-5525. - Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 4504 [https://github.com/apache/arrow/pull/4504] > Enable continuous fuzzing > - > > Key: ARROW-5525 > URL: https://issues.apache.org/jira/browse/ARROW-5525 > Project: Apache Arrow > Issue Type: Test > Components: C++ >Reporter: Marco Neumann >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Since fuzzing kinda only works if done as a continuous background job, we > should find a way of doing so. This likely requires another service than > Travis. Basic requirements are: > * master builds should be submitted for fuzzing > * project members should be informed about new crashes (ideally not via > public issue due to potential security impact) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ARROW-5777) [C++] BasicDecimal128 is a small object it doesn't always make sense to pass by const ref
[ https://issues.apache.org/jira/browse/ARROW-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-5777. - Resolution: Fixed Fix Version/s: 0.14.1 Issue resolved by pull request 4739 [https://github.com/apache/arrow/pull/4739] > [C++] BasicDecimal128 is a small object it doesn't always make sense to pass > by const ref > - > > Key: ARROW-5777 > URL: https://issues.apache.org/jira/browse/ARROW-5777 > Project: Apache Arrow > Issue Type: Improvement >Reporter: Micah Kornfield >Assignee: Micah Kornfield >Priority: Major > Labels: pull-request-available > Fix For: 0.14.1 > > Time Spent: 12h 50m > Remaining Estimate: 0h > > For the builder use case we saw 10% improvement in performance by not using > const ref. > Add benchmarks and look at where it makes sense to avoid passing by ref. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5777) [C++] BasicDecimal128 is a small object it doesn't always make sense to pass by const ref
[ https://issues.apache.org/jira/browse/ARROW-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5777: Fix Version/s: (was: 0.14.1) 1.0.0 > [C++] BasicDecimal128 is a small object it doesn't always make sense to pass > by const ref > - > > Key: ARROW-5777 > URL: https://issues.apache.org/jira/browse/ARROW-5777 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Micah Kornfield >Assignee: Micah Kornfield >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > Time Spent: 13h > Remaining Estimate: 0h > > For the builder use case we saw 10% improvement in performance by not using > const ref. > Add benchmarks and look at where it makes sense to avoid passing by ref. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5777) [C++] BasicDecimal128 is a small object it doesn't always make sense to pass by const ref
[ https://issues.apache.org/jira/browse/ARROW-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5777: Component/s: C++ > [C++] BasicDecimal128 is a small object it doesn't always make sense to pass > by const ref > - > > Key: ARROW-5777 > URL: https://issues.apache.org/jira/browse/ARROW-5777 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Micah Kornfield >Assignee: Micah Kornfield >Priority: Major > Labels: pull-request-available > Fix For: 0.14.1 > > Time Spent: 12h 50m > Remaining Estimate: 0h > > For the builder use case we saw 10% improvement in performance by not using > const ref. > Add benchmarks and look at where it makes sense to avoid passing by ref. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe L. Korn reassigned ARROW-5874:
--
Assignee: Krisztian Szucs
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Assignee: Krisztian Szucs
>Priority: Critical
> Labels: pull-request-available, pyarrow, wheel
> Fix For: 0.14.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
> #
>
[jira] [Resolved] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe L. Korn resolved ARROW-5874.
Resolution: Fixed
Issue resolved by pull request 4823
[https://github.com/apache/arrow/pull/4823]
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pull-request-available, pyarrow, wheel
> Fix For: 0.14.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
>
[jira] [Resolved] (ARROW-5827) [C++] Require c-ares CMake config
[ https://issues.apache.org/jira/browse/ARROW-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-5827. - Resolution: Fixed Fix Version/s: (was: 1.0.0) 0.14.1 Issue resolved by pull request 4783 [https://github.com/apache/arrow/pull/4783] > [C++] Require c-ares CMake config > - > > Key: ARROW-5827 > URL: https://issues.apache.org/jira/browse/ARROW-5827 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Sutou Kouhei >Assignee: Sutou Kouhei >Priority: Major > Labels: pull-request-available > Fix For: 0.14.1 > > Time Spent: 50m > Remaining Estimate: 0h > > Because gRPC requires c-ares' CMake config. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-5879) [C++][Python] Clean up linking of optional libraries within C++ and to Python extensions
Wes McKinney created ARROW-5879:
---
Summary: [C++][Python] Clean up linking of optional libraries
within C++ and to Python extensions
Key: ARROW-5879
URL: https://issues.apache.org/jira/browse/ARROW-5879
Project: Apache Arrow
Issue Type: Improvement
Components: C++, Python
Reporter: Wes McKinney
Fix For: 1.0.0
Optional modules such as
* Flight (and its dependents, including OpenSSL)
* Parquet
* Gandiva
are all linked unconditionally to {{pyarrow.lib}}. It would be better IMHO to
only link these libraries to the corresponding Cython extension rather than
link everything to every extension.
Relatedly, libraries like OpenSSL are being included in linking with all shared
libraries. We should clean this up to only link to the relevant shared
libraries where it is required, like {{libparquet}} (for encryption support)
and {{libarrow_flight}} (for using gRPC with TLS)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5856) [Python] linking 3rd party cython modules against pyarrow fails since 0.14.0
[ https://issues.apache.org/jira/browse/ARROW-5856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5856: Fix Version/s: 0.14.1 1.0.0 > [Python] linking 3rd party cython modules against pyarrow fails since 0.14.0 > > > Key: ARROW-5856 > URL: https://issues.apache.org/jira/browse/ARROW-5856 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Affects Versions: 0.14.0 >Reporter: Steve Stagg >Priority: Major > Fix For: 1.0.0, 0.14.1 > > Attachments: setup.py, test.pyx > > > Compiling cython modules that link to the pyarrow library, using the > recommended approach for getting the appropriate include and link flags has > stopped working for PyArrow 0.14.0. > > A minimal test case is included in the attachments that demonstrates the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5827) [C++] Require c-ares CMake config
[ https://issues.apache.org/jira/browse/ARROW-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-5827: Fix Version/s: 1.0.0 > [C++] Require c-ares CMake config > - > > Key: ARROW-5827 > URL: https://issues.apache.org/jira/browse/ARROW-5827 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Sutou Kouhei >Assignee: Sutou Kouhei >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0, 0.14.1 > > Time Spent: 1h > Remaining Estimate: 0h > > Because gRPC requires c-ares' CMake config. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ARROW-5826) [Website] Blog post for 0.14.0 release announcement
[ https://issues.apache.org/jira/browse/ARROW-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-5826. - Resolution: Fixed Issue resolved by pull request 4819 [https://github.com/apache/arrow/pull/4819] > [Website] Blog post for 0.14.0 release announcement > --- > > Key: ARROW-5826 > URL: https://issues.apache.org/jira/browse/ARROW-5826 > Project: Apache Arrow > Issue Type: New Feature > Components: Website >Reporter: Wes McKinney >Assignee: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.14.0 > > Time Spent: 50m > Remaining Estimate: 0h > > As with prior releases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5871:
Fix Version/s: 0.14.1
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
> Fix For: 0.14.1
>
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5874:
Fix Version/s: 0.14.1
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pull-request-available, pyarrow, wheel
> Fix For: 0.14.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
> #
>
[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880548#comment-16880548
]
Suvayu Ali commented on ARROW-5871:
---
Hi [~wesmckinn], I see the same issue with the manylinux1 wheel.
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5863) [Python] Segmentation Fault via pytest-runner
[
https://issues.apache.org/jira/browse/ARROW-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5863:
Fix Version/s: 0.14.1
> [Python] Segmentation Fault via pytest-runner
> -
>
> Key: ARROW-5863
> URL: https://issues.apache.org/jira/browse/ARROW-5863
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: $ uname -a
> Linux aleph 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019
> x86_64 GNU/Linux
> $ python --version
> Python 3.7.3
> $ pip freeze | grep -P "(pyarrow|pytest)"
> pyarrow==0.14.0
> pytest==5.0.0
> pytest-benchmark==3.2.2
> pytest-cov==2.7.1
> pytest-env==0.6.2
> pytest-forked==1.0.2
> pytest-html==1.21.1
> pytest-metadata==1.8.0
> pytest-mock==1.10.4
> pytest-runner==5.1
> pytest-sugar==0.9.2
> pytest-xdist==1.29.0
>Reporter: Josh Bode
>Priority: Major
> Fix For: 1.0.0, 0.14.1
>
> Attachments: pyarrow-issue.tar.bz2, pytest-runner.log, pytest.log
>
>
> When running {{pytest}} on projects using {{pyarrow==0.14.0}} on Linux, I am
> getting segmentation faults, but interestingly _only_ when run via
> {{pytest-runner}} (which provides the {{setup.py pytest}} command)
> This works (i.e. {{pytest}} directly):
> {code:java}
> $ pytest
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.09s):
> 1 passed{code}
> However, this does not work, ending in a segmentation fault, even though the
> tests pass:
> {code:java}
> $ python setup.py pytest
> running pytest
> running egg_info
> writing pyarrow_issue.egg-info/PKG-INFO
> writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
> writing requirements to pyarrow_issue.egg-info/requires.txt
> writing top-level names to pyarrow_issue.egg-info/top_level.txt
> reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
> running build_ext
> Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0,
> pytest-sugar 0.9.2)
> benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False
> min_rounds=5 min_time=0.05 max_time=1.0 calibration_precision=10
> warmup=False warmup_iterations=10)
> rootdir: /home/josh/scratch/pyarrow-issue
> plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4,
> xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1,
> html-1.21.1, benchmark-3.2.2, metadata-1.8.0
> collecting ...
> tests/test_pyarrow.py ✓ 100% ██
> Results (0.07s):
> 1 passed
> zsh: segmentation fault (core dumped) python setup.py pytest{code}
> backtrace from {{gdb}}
> {code:java}
> Thread 1 "python" received signal SIGSEGV, Segmentation fault.
> 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> (gdb) bt
> #0 0x77c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
> #1 0x77ae46cc in ?? () from /usr/lib/libpython3.7m.so.1.0
> #2 0x7023a6b3 in arrow::py::PyExtensionType::~PyExtensionType() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow_python.so.14
> #3 0x7fffed5e6467 in std::unordered_map std::shared_ptr, std::hash,
> std::equal_to, std::allocator std::shared_ptr > > >::~unordered_map() ()
> from
> /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow.so.14
> #4 0x77de5e70 in __run_exit_handlers () from /usr/lib/libc.so.6
> #5 0x77de5fae in exit () from /usr/lib/libc.so.6
> #6 0x77dcfeea in __libc_start_main () from /usr/lib/libc.so.6
> #7 0x505e in _start ()
> {code}
> I have observed this behaviour on my machine running natively, and also via
> docker. Also, 0.13.0 does not exhibit this behaviour
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (ARROW-5580) Correct definitions of timestamp functions in Gandiva
[ https://issues.apache.org/jira/browse/ARROW-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Kietzman reopened ARROW-5580: -- The date difference functions have confusing parameter order: conventionally, in a subtraction function the minuend is the left operand while the subtrahend is the right operand. > Correct definitions of timestamp functions in Gandiva > - > > Key: ARROW-5580 > URL: https://issues.apache.org/jira/browse/ARROW-5580 > Project: Apache Arrow > Issue Type: Task > Components: C++ - Gandiva >Reporter: Prudhvi Porandla >Assignee: Prudhvi Porandla >Priority: Minor > Labels: pull-request-available > Fix For: 0.14.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Timestamp functions are unsupported in Gandiva due to definition mismatch. > For example, Gandiva supports timestampAddMonth(timestamp, int32) but the > expected signature is timestampAddMonth(int32, timestamp). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-5878) [Python][C++] Parquet reader not forward compatible for timestamps without timezone
[
https://issues.apache.org/jira/browse/ARROW-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5878:
Fix Version/s: 0.14.1
> [Python][C++] Parquet reader not forward compatible for timestamps without
> timezone
> ---
>
> Key: ARROW-5878
> URL: https://issues.apache.org/jira/browse/ARROW-5878
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Florian Jetter
>Priority: Major
> Fix For: 1.0.0, 0.14.1
>
> Attachments: timezones_pyarrow_14.paquet
>
>
> Timestamps without timezone which are written by pyarrow 0.14.0 cannot be
> read anymore as timestamps by earlier versions. The timestamp is read as an
> integer when reading in with pyarrow 0.13.0
> Looking at the parquet schemas, it seems that the logical type cannot be
> understood by the older versions, see below.
> h4. File generation with pyarrow 0.14.0
> {code:java}
> import datetime
> import pyarrow.parquet as pq
> import pandas as pd
> df = pd.DataFrame(
> {
> "datetime64": pd.Series(["2018-01-01"], dtype="datetime64[ns]"),
> "datetime64_ts": pd.Series(
> [pd.Timestamp(datetime.datetime(2018, 1, 1), tz="Europe/Berlin")],
> dtype="datetime64[ns]",
> ),
> }
> )
> pq.write_table(pa.Table.from_pandas(df), "timezones_pyarrow_14.paquet")
> {code}
> h4. Reading with pyarrow 0.13.0
> {code:java}
> In [1]: import pyarrow.parquet as pq
> In [2]: import pyarrow as pa
> In [3]: with open("timezones_pyarrow_14.paquet", "rb") as fd:
>...: table = pq.read_pandas(fd)
>...:
> In [4]: table.to_pandas()
> Out[4]:
> datetime64 datetime64_ts
> 0 15147648 2018-01-01 00:00:00+01:00
> In [5]: table.to_pandas().dtypes
> Out[5]:
> datetime64 int64
> datetime64_tsdatetime64[ns, Europe/Berlin]
> dtype: object
> {code}
> h3. Parquet schema as seen by pyarrow versions:
> pyarrow 0.13.0 parquet schema
> {code:java}
> datetime64: INT64
> datetime64_ts: INT64 TIMESTAMP_MICROS
> {code}
> pyarrow 0.14.0 parquet schema
> {code:java}
> datetime64: INT64 Timestamp(isAdjustedToUTC=false, timeUnit=microseconds)
> datetime64_ts: INT64 Timestamp(isAdjustedToUTC=true, timeUnit=microseconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5878) [Python][C++] Parquet reader not forward compatible for timestamps without timezone
[
https://issues.apache.org/jira/browse/ARROW-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880520#comment-16880520
]
Wes McKinney commented on ARROW-5878:
-
This is sort of a grey area because of comments in parquet.thrift
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L334
I am OK with always setting TIMESTAMP_MICROS/TIMESTAMP_MILLIS ConvertedType for
data that originates from Arrow.
Do you want to submit a PR? We are probably doing a 0.14.1 release so this can
get fixed fairly soon
> [Python][C++] Parquet reader not forward compatible for timestamps without
> timezone
> ---
>
> Key: ARROW-5878
> URL: https://issues.apache.org/jira/browse/ARROW-5878
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Florian Jetter
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: timezones_pyarrow_14.paquet
>
>
> Timestamps without timezone which are written by pyarrow 0.14.0 cannot be
> read anymore as timestamps by earlier versions. The timestamp is read as an
> integer when reading in with pyarrow 0.13.0
> Looking at the parquet schemas, it seems that the logical type cannot be
> understood by the older versions, see below.
> h4. File generation with pyarrow 0.14.0
> {code:java}
> import datetime
> import pyarrow.parquet as pq
> import pandas as pd
> df = pd.DataFrame(
> {
> "datetime64": pd.Series(["2018-01-01"], dtype="datetime64[ns]"),
> "datetime64_ts": pd.Series(
> [pd.Timestamp(datetime.datetime(2018, 1, 1), tz="Europe/Berlin")],
> dtype="datetime64[ns]",
> ),
> }
> )
> pq.write_table(pa.Table.from_pandas(df), "timezones_pyarrow_14.paquet")
> {code}
> h4. Reading with pyarrow 0.13.0
> {code:java}
> In [1]: import pyarrow.parquet as pq
> In [2]: import pyarrow as pa
> In [3]: with open("timezones_pyarrow_14.paquet", "rb") as fd:
>...: table = pq.read_pandas(fd)
>...:
> In [4]: table.to_pandas()
> Out[4]:
> datetime64 datetime64_ts
> 0 15147648 2018-01-01 00:00:00+01:00
> In [5]: table.to_pandas().dtypes
> Out[5]:
> datetime64 int64
> datetime64_tsdatetime64[ns, Europe/Berlin]
> dtype: object
> {code}
> h3. Parquet schema as seen by pyarrow versions:
> pyarrow 0.13.0 parquet schema
> {code:java}
> datetime64: INT64
> datetime64_ts: INT64 TIMESTAMP_MICROS
> {code}
> pyarrow 0.14.0 parquet schema
> {code:java}
> datetime64: INT64 Timestamp(isAdjustedToUTC=false, timeUnit=microseconds)
> datetime64_ts: INT64 Timestamp(isAdjustedToUTC=true, timeUnit=microseconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5878) [Python][C++] Parquet reader not forward compatible for timestamps without timezone
[
https://issues.apache.org/jira/browse/ARROW-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5878:
Fix Version/s: 1.0.0
> [Python][C++] Parquet reader not forward compatible for timestamps without
> timezone
> ---
>
> Key: ARROW-5878
> URL: https://issues.apache.org/jira/browse/ARROW-5878
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Florian Jetter
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: timezones_pyarrow_14.paquet
>
>
> Timestamps without timezone which are written by pyarrow 0.14.0 cannot be
> read anymore as timestamps by earlier versions. The timestamp is read as an
> integer when reading in with pyarrow 0.13.0
> Looking at the parquet schemas, it seems that the logical type cannot be
> understood by the older versions, see below.
> h4. File generation with pyarrow 0.14.0
> {code:java}
> import datetime
> import pyarrow.parquet as pq
> import pandas as pd
> df = pd.DataFrame(
> {
> "datetime64": pd.Series(["2018-01-01"], dtype="datetime64[ns]"),
> "datetime64_ts": pd.Series(
> [pd.Timestamp(datetime.datetime(2018, 1, 1), tz="Europe/Berlin")],
> dtype="datetime64[ns]",
> ),
> }
> )
> pq.write_table(pa.Table.from_pandas(df), "timezones_pyarrow_14.paquet")
> {code}
> h4. Reading with pyarrow 0.13.0
> {code:java}
> In [1]: import pyarrow.parquet as pq
> In [2]: import pyarrow as pa
> In [3]: with open("timezones_pyarrow_14.paquet", "rb") as fd:
>...: table = pq.read_pandas(fd)
>...:
> In [4]: table.to_pandas()
> Out[4]:
> datetime64 datetime64_ts
> 0 15147648 2018-01-01 00:00:00+01:00
> In [5]: table.to_pandas().dtypes
> Out[5]:
> datetime64 int64
> datetime64_tsdatetime64[ns, Europe/Berlin]
> dtype: object
> {code}
> h3. Parquet schema as seen by pyarrow versions:
> pyarrow 0.13.0 parquet schema
> {code:java}
> datetime64: INT64
> datetime64_ts: INT64 TIMESTAMP_MICROS
> {code}
> pyarrow 0.14.0 parquet schema
> {code:java}
> datetime64: INT64 Timestamp(isAdjustedToUTC=false, timeUnit=microseconds)
> datetime64_ts: INT64 Timestamp(isAdjustedToUTC=true, timeUnit=microseconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-5874:
--
Labels: pull-request-available pyarrow wheel (was: pyarrow wheel)
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pull-request-available, pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
> #
>
[jira] [Comment Edited] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880457#comment-16880457
]
Uwe L. Korn edited comment on ARROW-5874 at 7/8/19 3:17 PM:
We should bundle OpenSSL with the wheel and declare it as unsafe to use in
production. Either compile from source when using {{pip}} in your production
environment or use {{conda}}. This is roughly the same way {{psycopg2}} went.
You cannot manage this type of binary dependencies with {{pip}}, this is why
{{conda}} was created.
I'm aware of {{delocate}} but we are explcitly not using it as we rely on CMake
and {{setup.py}} to bundle all required libraries. In our case it might be
better to statically link OpenSSL to not pollute the global namespace with our
shipped version of OpenSSL.
was (Author: xhochy):
We should bundle OpenSSL with the wheel and declare it as unsafe to use in
production. Either compile from source when using {{pip}} in your production
environment or use {{conda}}. This is roughly the same way {{psycopg2}} went.
You cannot
I'm aware of {{delocate}} but we are explcitly not using it as we rely on CMake
and {{setup.py}} to bundle all required libraries. In our case it might be
better to statically link OpenSSL to not pollute the global namespace with our
shipped version of OpenSSL.
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib
[jira] [Created] (ARROW-5878) [Python][C++] Parquet reader not forward compatible for timestamps without timezone
Florian Jetter created ARROW-5878:
-
Summary: [Python][C++] Parquet reader not forward compatible for
timestamps without timezone
Key: ARROW-5878
URL: https://issues.apache.org/jira/browse/ARROW-5878
Project: Apache Arrow
Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Florian Jetter
Attachments: timezones_pyarrow_14.paquet
Timestamps without timezone which are written by pyarrow 0.14.0 cannot be read
anymore as timestamps by earlier versions. The timestamp is read as an integer
when reading in with pyarrow 0.13.0
Looking at the parquet schemas, it seems that the logical type cannot be
understood by the older versions, see below.
h4. File generation with pyarrow 0.14.0
{code:java}
import datetime
import pyarrow.parquet as pq
import pandas as pd
df = pd.DataFrame(
{
"datetime64": pd.Series(["2018-01-01"], dtype="datetime64[ns]"),
"datetime64_ts": pd.Series(
[pd.Timestamp(datetime.datetime(2018, 1, 1), tz="Europe/Berlin")],
dtype="datetime64[ns]",
),
}
)
pq.write_table(pa.Table.from_pandas(df), "timezones_pyarrow_14.paquet")
{code}
h4. Reading with pyarrow 0.13.0
{code:java}
In [1]: import pyarrow.parquet as pq
In [2]: import pyarrow as pa
In [3]: with open("timezones_pyarrow_14.paquet", "rb") as fd:
...: table = pq.read_pandas(fd)
...:
In [4]: table.to_pandas()
Out[4]:
datetime64 datetime64_ts
0 15147648 2018-01-01 00:00:00+01:00
In [5]: table.to_pandas().dtypes
Out[5]:
datetime64 int64
datetime64_tsdatetime64[ns, Europe/Berlin]
dtype: object
{code}
h3. Parquet schema as seen by pyarrow versions:
pyarrow 0.13.0 parquet schema
{code:java}
datetime64: INT64
datetime64_ts: INT64 TIMESTAMP_MICROS
{code}
pyarrow 0.14.0 parquet schema
{code:java}
datetime64: INT64 Timestamp(isAdjustedToUTC=false, timeUnit=microseconds)
datetime64_ts: INT64 Timestamp(isAdjustedToUTC=true, timeUnit=microseconds)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880457#comment-16880457
]
Uwe L. Korn commented on ARROW-5874:
We should bundle OpenSSL with the wheel and declare it as unsafe to use in
production. Either compile from source when using {{pip}} in your production
environment or use {{conda}}. This is roughly the same way {{psycopg2}} went.
You cannot
I'm aware of {{delocate}} but we are explcitly not using it as we rely on CMake
and {{setup.py}} to bundle all required libraries. In our case it might be
better to statically link OpenSSL to not pollute the global namespace with our
shipped version of OpenSSL.
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
>
[jira] [Comment Edited] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880443#comment-16880443
]
Krisztian Szucs edited comment on ARROW-5874 at 7/8/19 3:10 PM:
According to homebrew: "Apple has deprecated use of OpenSSL in favor of its own
TLS and crypto libraries."
So we cannot expect to have system openssl.
Thanks for the delocate reference, I'm taking a look. cc [~xhochy]
was (Author: kszucs):
According to homebrew: "Apple has deprecated use of OpenSSL in favor of its own
TLS and crypto libraries."
So we cannot expect to have system openssl.
Thanks for the delocate reference, I'm taking a look.
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
>
[jira] [Commented] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880443#comment-16880443
]
Krisztian Szucs commented on ARROW-5874:
According to homebrew: "Apple has deprecated use of OpenSSL in favor of its own
TLS and crypto libraries."
So we cannot expect to have system openssl.
Thanks for the delocate reference, I'm taking a look.
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
>
[jira] [Created] (ARROW-5877) [FlightRPC] Document caveats around usage of auth APIs
lidavidm created ARROW-5877: --- Summary: [FlightRPC] Document caveats around usage of auth APIs Key: ARROW-5877 URL: https://issues.apache.org/jira/browse/ARROW-5877 Project: Apache Arrow Issue Type: Improvement Components: FlightRPC Reporter: lidavidm The Flight Handshake method can be insecure, and currently has a surprising failure mode; we should document these caveats (blocks forever waiting on client/server; insecure depending on deployment configuration) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-5876) [FlightRPC] Implement basic auth across all languages
lidavidm created ARROW-5876: --- Summary: [FlightRPC] Implement basic auth across all languages Key: ARROW-5876 URL: https://issues.apache.org/jira/browse/ARROW-5876 Project: Apache Arrow Issue Type: Improvement Components: FlightRPC Affects Versions: 0.14.0 Reporter: lidavidm We should implement a set of common auth methods in Flight itself to have standardized ways to do things like basic auth. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-5875) [FlightRPC] Test RPC features in integration tests
lidavidm created ARROW-5875: --- Summary: [FlightRPC] Test RPC features in integration tests Key: ARROW-5875 URL: https://issues.apache.org/jira/browse/ARROW-5875 Project: Apache Arrow Issue Type: Test Components: FlightRPC, Integration Affects Versions: 0.14.0 Reporter: lidavidm We should test not just wire-format compatibility, but feature-compatibility in Flight integration tests. This may mean adding a separate suite of tests to the integration script. Features that should be tested include: * Authentication * Error & error code propagation * Cancellation * Flow control/backpressure -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880373#comment-16880373
]
Michael Anselmi commented on ARROW-5874:
Hi [~wesmckinn], thanks for the response.
I haven't used it myself but
[delocate|https://github.com/matthew-brett/delocate] may help with bundling
OpenSSL shared libraries into the wheel if you decide to go that route. It
seems to be the macOS counterpart of
[auditwheel|https://github.com/pypa/auditwheel].
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility
[jira] [Commented] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880369#comment-16880369
]
Wes McKinney commented on ARROW-5874:
-
This was actually intentional to depend on system OpenSSL, but it was
inadequately tested. We probably will need to bundle the Homebrew-provided
OpenSSL libraries
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
>
[jira] [Updated] (ARROW-5805) [Python] Dockerize (add to docker-compose) Python Travis CI job
[ https://issues.apache.org/jira/browse/ARROW-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-5805: -- Labels: pull-request-available (was: ) > [Python] Dockerize (add to docker-compose) Python Travis CI job > --- > > Key: ARROW-5805 > URL: https://issues.apache.org/jira/browse/ARROW-5805 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Reporter: Wes McKinney >Assignee: Rok Mihevc >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > > https://github.com/apache/arrow/blob/master/.travis.yml#L118 > The existing Python Dockerfiles should be expanded to test all of the things > that are being tested currently in Travis CI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880364#comment-16880364
]
Krisztian Szucs commented on ARROW-5874:
I can confirm the issue, with brew installed openssl the same error occurs.
The test section in the build script doesn't remove system openssl, this is why
we have not captured the issue
https://github.com/apache/arrow/blob/master/dev/tasks/python-wheels/travis.osx.yml#L71
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version
[jira] [Updated] (ARROW-5803) [C++] Dockerize C++ with clang 7 Travis CI unit test logic
[ https://issues.apache.org/jira/browse/ARROW-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-5803: -- Labels: pull-request-available (was: ) > [C++] Dockerize C++ with clang 7 Travis CI unit test logic > -- > > Key: ARROW-5803 > URL: https://issues.apache.org/jira/browse/ARROW-5803 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ >Reporter: Wes McKinney >Assignee: Francois Saint-Jacques >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > > Convert to docker-compose (or use one of the current Dockerfiles under cpp/) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880355#comment-16880355
]
Wes McKinney commented on ARROW-5874:
-
Thanks for the report. Since adding the OpenSSL dependency we expected some
hiccups. There's a couple possible solutions
* Bundling OpenSSL shared libraries in the wheel
* Fixing the library path to not be looking for libraries in a hard-coded
location (I thought that the wheel build was supposed to fix this)
* Only linking to OpenSSL in libarrow_flight (I'm not sure that it needs to be
linked with {{pyarrow.lib}}). I think we should do this regardless
FYI: maintaining wheels is extremely difficult for us. The project, and the
Python ecosystem in general, would be much better off if everyone used conda
for complex packages like this
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> #
[jira] [Updated] (ARROW-5873) [Python][C++] Segmentation fault when comparing schema with None
[
https://issues.apache.org/jira/browse/ARROW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5873:
Fix Version/s: 1.0.0
> [Python][C++] Segmentation fault when comparing schema with None
>
>
> Key: ARROW-5873
> URL: https://issues.apache.org/jira/browse/ARROW-5873
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Florian Jetter
>Priority: Minor
> Fix For: 1.0.0
>
>
> When comparing a schema with a Python {{None}} I get a segmentation fault.
> This is a regression to 0.13.0
> {code:java}
> In [2]: import pyarrow as pa
> In [3]: pa.schema([pa.field("something", pa.int64())]).equals(None)
> [1]82085 segmentation fault ipython
> {code}
> System information:
> System Version: macOS 10.13.6 (17G6030)
> Kernel Version: Darwin 17.7.0
> Python 3.6.7
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5873) [Python][C++] Segmentation fault when comparing schema with None
[
https://issues.apache.org/jira/browse/ARROW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5873:
Summary: [Python][C++] Segmentation fault when comparing schema with None
(was: [Python/C++] Segmentation fault when comparing schema with None)
> [Python][C++] Segmentation fault when comparing schema with None
>
>
> Key: ARROW-5873
> URL: https://issues.apache.org/jira/browse/ARROW-5873
> Project: Apache Arrow
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Florian Jetter
>Priority: Minor
>
> When comparing a schema with a Python {{None}} I get a segmentation fault.
> This is a regression to 0.13.0
> {code:java}
> In [2]: import pyarrow as pa
> In [3]: pa.schema([pa.field("something", pa.int64())]).equals(None)
> [1]82085 segmentation fault ipython
> {code}
> System information:
> System Version: macOS 10.13.6 (17G6030)
> Kernel Version: Darwin 17.7.0
> Python 3.6.7
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Anselmi updated ARROW-5874:
---
Description:
Hello, and congrats on the recent release of Apache Arrow 0.14.0!
This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
{code:java}
python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
source ~/virtualenv/pyarrow-0.14.0/bin/activate
pip install --upgrade pip setuptools
pip install pyarrow # installs pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
pip freeze --all
# numpy==1.16.4
# pip==19.1.1
# pyarrow==0.14.0
# setuptools==41.0.1
# six==1.12.0
{code}
However I am unable to import pyarrow:
{code:java}
python -c 'import pyarrow'
# Traceback (most recent call last):
# File "", line 1, in
# File
"/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
line 49, in
# from pyarrow.lib import cpu_count, set_cpu_count
# ImportError:
dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
# Referenced from:
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
# Reason: image not found
{code}
pyarrow is trying to load a shared library (OpenSSL in this case) from a path
under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
provided by Homebrew as part of your build process? Unfortunately this makes
the pyarrow 0.14.0 wheel completely unusable on my system or any system that
doesn't have OpenSSL installed in that location. This is a regression from
pyarrow 0.13.0 as those wheels "just worked".
Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}} and
{{.so}} file in
{{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
and included the output for those with dependencies under {{/usr/local/opt}}:
{code:java}
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
1.2.8)
# @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
current version 0.0.0)
# @rpath/libarrow_boost_filesystem.dylib (compatibility version 0.0.0,
current version 0.0.0)
# @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
current version 0.0.0)
# /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.5.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
# @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.5.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
# @rpath/libarrow_python.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
#
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
(compatibility version 150.0.0, current version 1349.64.0)
# @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/lib/libc++.1.dylib
[jira] [Commented] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880336#comment-16880336
]
Wes McKinney commented on ARROW-5871:
-
You can download the file "pyarrow-0.14.0-cp37-cp37m-manylinux1_x86_64.whl "
from https://pypi.org/project/pyarrow/#files and install it with pip
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5871) [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-5871:
Summary: [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
(was: Can't import pyarrow 0.14.0 due to mismatching libcrypt)
> [Python] Can't import pyarrow 0.14.0 due to mismatching libcrypt
>
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-5871) Can't import pyarrow 0.14.0 due to mismatching libcrypt
[
https://issues.apache.org/jira/browse/ARROW-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880335#comment-16880335
]
Wes McKinney commented on ARROW-5871:
-
Looks like this might be a problem with the manylinux2010 wheels. Can you try
installing the manylinux1 wheel and see if that works?
> Can't import pyarrow 0.14.0 due to mismatching libcrypt
> ---
>
> Key: ARROW-5871
> URL: https://issues.apache.org/jira/browse/ARROW-5871
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 0.14.0
> Environment: 5.1.16-300.fc30.x86_64
> Python 3.7.3
> libxcrypt-4.4.6-2.fc30.x86_64
>Reporter: Suvayu Ali
>Priority: Major
>
> In a freshly created virtual environment, after I install pyarrow 0.14.0
> (using pip), importing pyarrow from the python prompt leads to crash:
> {code:java}
> $ mktmpenv
> [..]
> This is a temporary environment. It will be deleted when you run 'deactivate'.
> $ pip install pyarrow
> Collecting pyarrow
> Using cached
> https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
> Collecting numpy>=1.14 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl
> Collecting six>=1.0.0 (from pyarrow)
> Using cached
> https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
> Installing collected packages: numpy, six, pyarrow
> Successfully installed numpy-1.16.4 pyarrow-0.14.0 six-1.12.0
> $ python --version
> Python 3.7.3
> $ python -m pyarrow
> Traceback (most recent call last):
> File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
> mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
> File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
> return _get_module_details(pkg_main_name, error)
> File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
> __import__(pkg_name)
> File
> "/home/user/.virtualenvs/tmp-8a4d52e7bb62853/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: libcrypt.so.1: cannot open shared object file: No such file or
> directory{code}
> This is surprising because I have older versions of pyarrow (up to 0.13.0)
> working, and libcrypt on my system (Fedora 30, Python 3.7) is libcrypt.so.2!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Anselmi updated ARROW-5874:
---
Labels: pyarrow wheel (was: )
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
> Labels: pyarrow, wheel
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade -- pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library (OpenSSL in this case) from a path
> under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
> provided by Homebrew as part of your build process? Unfortunately this makes
> the pyarrow 0.14.0 wheel completely unusable on my system or any system that
> doesn't have OpenSSL installed in that location. This is a regression from
> pyarrow 0.13.0 as those wheels "just worked".
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
> # @rpath/libarrow_python.14.dylib
[jira] [Updated] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Anselmi updated ARROW-5874:
---
Description:
Hello, and congrats on the recent release of Apache Arrow 0.14.0!
This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
{code:java}
python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
source ~/virtualenv/pyarrow-0.14.0/bin/activate
pip install --upgrade -- pip setuptools
pip install pyarrow # installs pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
pip freeze --all
# numpy==1.16.4
# pip==19.1.1
# pyarrow==0.14.0
# setuptools==41.0.1
# six==1.12.0
{code}
However I am unable to import pyarrow:
{code:java}
python -c 'import pyarrow'
# Traceback (most recent call last):
# File "", line 1, in
# File
"/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
line 49, in
# from pyarrow.lib import cpu_count, set_cpu_count
# ImportError:
dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
# Referenced from:
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
# Reason: image not found
{code}
pyarrow is trying to load a shared library (OpenSSL in this case) from a path
under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
provided by Homebrew as part of your build process? Unfortunately this makes
the pyarrow 0.14.0 wheel completely unusable on my system or any system that
doesn't have OpenSSL installed in that location. This is a regression from
pyarrow 0.13.0 as those wheels "just worked".
Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}} and
{{.so}} file in
{{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
and included the output for those with dependencies under {{/usr/local/opt}}:
{code:java}
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
1.2.8)
# @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
current version 0.0.0)
# @rpath/libarrow_boost_filesystem.dylib (compatibility version 0.0.0,
current version 0.0.0)
# @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
current version 0.0.0)
# /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.5.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
# @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.5.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
# @rpath/libarrow_python.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
#
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
(compatibility version 150.0.0, current version 1349.64.0)
# @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/lib/libc++.1.dylib
[jira] [Updated] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Anselmi updated ARROW-5874:
---
Description:
Hello, and congrats on the recent release of Apache Arrow 0.14.0!
This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
{code:java}
python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
source ~/virtualenv/pyarrow-0.14.0/bin/activate
pip install --upgrade -- pip setuptools
pip install pyarrow # installs pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
pip freeze --all
# numpy==1.16.4
# pip==19.1.1
# pyarrow==0.14.0
# setuptools==41.0.1
# six==1.12.0
{code}
However I am unable to import pyarrow:
{code:java}
python -c 'import pyarrow'
# Traceback (most recent call last):
# File "", line 1, in
# File
"/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
line 49, in
# from pyarrow.lib import cpu_count, set_cpu_count
# ImportError:
dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
# Referenced from:
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
# Reason: image not found
{code}
pyarrow is trying to load a shared library (OpenSSL in this case) from a path
under {{/usr/local/opt}} that doesn't exist; perhaps that OpenSSL had been
provided by Homebrew as part of your build process? Unfortunately this makes
the pyarrow 0.14.0 wheel completely unusable on my system or any system that
doesn't have OpenSSL installed in that location.
Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}} and
{{.so}} file in
{{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
and included the output for those with dependencies under {{/usr/local/opt}}:
{code:java}
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
1.2.8)
# @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
current version 0.0.0)
# @rpath/libarrow_boost_filesystem.dylib (compatibility version 0.0.0,
current version 0.0.0)
# @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
current version 0.0.0)
# /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.5.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
# @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.5.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
otool -L
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
#
/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
# @rpath/libarrow_python.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 1238.50.2)
#
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
(compatibility version 150.0.0, current version 1349.64.0)
# @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
current version 14.0.0)
# @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
version 14.0.0)
# /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version
1.0.0, current version 1.0.0)
# /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
version 1.0.0, current version 1.0.0)
# /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
307.5.0)
otool -L
[jira] [Updated] (ARROW-5874) [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Anselmi updated ARROW-5874:
---
Summary: [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
/usr/local/opt (was: pyarrow 0.14.0 macOS wheels depend on shared libs under
/usr/local/opt)
> [Python] pyarrow 0.14.0 macOS wheels depend on shared libs under
> /usr/local/opt
> ---
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade -- pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library from a path under
> {{/usr/local/opt}} that doesn't exist. This makes the pyarrow 0.14.0 wheel
> completely unusable on my system.
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
> # @rpath/libarrow_python.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> #
[jira] [Updated] (ARROW-5874) pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
[
https://issues.apache.org/jira/browse/ARROW-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Anselmi updated ARROW-5874:
---
Summary: pyarrow 0.14.0 macOS wheels depend on shared libs under
/usr/local/opt (was: pyarrow 0.14.0 macOS wheels appear to depend on shared
libs under /usr/local/opt)
> pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt
> --
>
> Key: ARROW-5874
> URL: https://issues.apache.org/jira/browse/ARROW-5874
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.14.0
> Environment: macOS 10.14.5
> Anaconda Python 3.7.3
>Reporter: Michael Anselmi
>Priority: Critical
>
> Hello, and congrats on the recent release of Apache Arrow 0.14.0!
> This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:
> {code:java}
> python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
> source ~/virtualenv/pyarrow-0.14.0/bin/activate
> pip install --upgrade -- pip setuptools
> pip install pyarrow # installs
> pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
> pip freeze --all
> # numpy==1.16.4
> # pip==19.1.1
> # pyarrow==0.14.0
> # setuptools==41.0.1
> # six==1.12.0
> {code}
> However I am unable to import pyarrow:
> {code:java}
> python -c 'import pyarrow'
> # Traceback (most recent call last):
> # File "", line 1, in
> # File
> "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py",
> line 49, in
> # from pyarrow.lib import cpu_count, set_cpu_count
> # ImportError:
> dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so,
> 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
> # Referenced from:
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> # Reason: image not found
> {code}
> pyarrow is trying to load a shared library from a path under
> {{/usr/local/opt}} that doesn't exist. This makes the pyarrow 0.14.0 wheel
> completely unusable on my system.
> Additional diagnostic output below. I ran {{otool -L}} on each {{.dylib}}
> and {{.so}} file in
> {{/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow}}
> and included the output for those with dependencies under {{/usr/local/opt}}:
> {code:java}
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version
> 1.2.8)
> # @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # @rpath/libarrow_boost_filesystem.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> # @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0,
> current version 0.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
> # @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # @rpath/libarrow.14.dylib (compatibility version 14.0.0, current
> version 14.0.0)
> # /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility
> version 1.0.0, current version 1.0.0)
> # /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current
> version 307.5.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> otool -L
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
> #
> /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
> # @rpath/libarrow_python.14.dylib (compatibility version 14.0.0,
> current version 14.0.0)
> # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.50.2)
> #
>
