[
https://issues.apache.org/jira/browse/ARROW-12585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rodrigo Tobar updated ARROW-12585:
----------------------------------
Attachment: example.tar.gz
> Published apt packages incompatible with pip binary wheels
> ----------------------------------------------------------
>
> Key: ARROW-12585
> URL: https://issues.apache.org/jira/browse/ARROW-12585
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Packaging, Python
> Affects Versions: 4.0.0
> Reporter: Rodrigo Tobar
> Priority: Major
> Attachments: example.tar.gz
>
>
> We have a shared library that uses the shared {{libarrow}} and {{libplasma}}
> plasma libraries. Our shared library is then eventually loaded by a python
> process where we use also {{pyarrow}}. To avoid compilation of arrow/plasma
> we are installing the {{libarrow-dev}} and {{libplasma-dev}} apt packages (as
> per the official [instructions|https://arrow.apache.org/install/]) and the
> binary wheel of {{pyarrow}}.
> Each method brings its own copy of {{libarrow.so.400}}, and it turns out the
> two libraries are not equal: the library contained within {{pyarrow}} is
> compiled most probably with an older gcc version than that installed via apt,
> which is compiled using the newer CXX11 ABI from stdlibc++. This wouldn't
> have any visible effects, except that {{std::string}} is used (and maybe more
> affected types) in some arrow API points. The difference in the ABI used to
> compile {{libarrow.so.400}} eventually means they contain differently named
> symbols.
> Back to our shared library, we load it in a python process. When this
> happens, and if the {{pyarrow}} has already been imported, then *its* copy of
> {{libarrow.so.400}} is already in memory, and loading our shared library
> doesn't load the "apt" copy of {{libarrow.so.400}}. This means our library
> doesn't trigger the loading of the copy of {{libarrow.so.400}} that it was
> compiled against, and if our library refers to one of the symbols that has
> changed name then it fails to load due to this missing symbol.
> I've attached a fairly minimal example: a Dockerfile prepares a system with
> libarrow-dev from apt and a binary pyarrow wheel from PyPI. It then compiles
> a shared library against libarrow-dev. The command ran by default by the
> container is a small test that runs python and loads the example shared
> library, both with and without loading pyarrow first. When pyarrow is loaded
> first then a missing symbol error happens and the shared library fails to
> load.
> I've experienced this in an Ubuntu-based linux distro and against Arrow
> 4.0.0, but I'd assume this happens in other distros and versions.
> The workaround we are using at the moment is simple: we are installing a
> pyarrow version that is different from the arrow version installed via apt.
> We are lucky we can run in this mixed-version, multiple-libraries-loaded
> scenario, but it might not be for everyone.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
