The end goal is to go direct from pyarrow to wasm without intermediate transforms. I can definitely make it work as is, we'll just have to be careful that the code we compile to webassembly matches exactly either our local copy of arrow if the user hasn't installed pyarrow, otherwise their installed copy.
Tim Paine tim.paine.nyc 908-721-1185 > On Sep 10, 2019, at 19:12, Tim Paine <t.paine...@gmail.com> wrote: > > We're building webassembly, so we obviously don't want to introduce a pyarrow > dependency. I don't want to do any pyarrow manipulations in c++, just get the > c++ table. I was hoping pyarrow might expose a raw pointer or have something > castable. > > It seems to be a big limitation, there is no way of communicating a pyarrow > table to a c++ library that uses arrow without that library linking against > pyarrow. > > Tim Paine > tim.paine.nyc > 908-721-1185 > >> On Sep 10, 2019, at 17:44, Wes McKinney <wesmck...@gmail.com> wrote: >> >> The Python extension types are defined in Cython, not C or C++ so you need >> to load the Cython extensions in order to instantiate the classes. >> >> Why do you have 2 copies of the C++ library? That seems easy to fix. If you >> are using wheels from PyPI I would recommend that you switch to conda or >> your own wheels without the C++ libraries bundled. >> >>> On Tue, Sep 10, 2019, 4:23 PM Tim Paine <t.paine...@gmail.com> wrote: >>> >>> Is there no way to do it without PyArrow? My C++ library is building arrow >>> itself, which means if I use PyArow I’ll end up having 2 copies: one from >>> my local C++ only build, and one from PyArrow. >>> >>>> On Sep 10, 2019, at 5:18 PM, Wes McKinney <wesmck...@gmail.com> wrote: >>>> >>>> hi Tim, >>>> >>>> You can use the functions in >>>> >>>> >>> https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/pyarrow.h >>>> >>>> You need to call "import_pyarrow()" from C++ before these APIs can be >>>> used. It's similar to the NumPy C API in that regard >>>> >>>> - Wes >>>> >>>>> On Tue, Sep 10, 2019 at 4:13 PM Tim Paine <t.paine...@gmail.com> wrote: >>>>> >>>>> Hey all, following up on a question I asked on stack overflow < >>> https://stackoverflow.com/questions/57863751/how-to-convert-pyarrow-table-to-arrow-table-when-interfacing-between-pyarrow-in >>>> . >>>>> >>>>> It seems there is some code < >>> https://arrow.apache.org/docs/python/extending.html#_CPPv412unwrap_tableP8PyObjectPNSt10shared_ptrI5TableEE> >>> in PyArrow’s C++ to convert from a PyArrow table to an Arrow table. The >>> problem with this is that my C++ library < >>> https://github.com/finos/perspective> is going to build and link against >>> Arrow on the C++ side rather than PyArrow side (because it will also be >>> consumed in WebAssembly), so I want to avoid also linking against PyArrow’s >>> copy of the arrow library. I also need to look for PyArrow’s header files, >>> which might conflict with the version in the local C++ code. >>>>> >>>>> My solution right now is to just assert that PyArrow version == Arrow >>> version and do some pruning (so I link against local libarrow and PyArrow’s >>> libarrow_python rather than use PyArrow’s libarrow), but ideally it would >>> be great if there was a clean way to hand a PyArrow Table over to C++ >>> without requiring the C++ to have PyArrow (e.g. using only a PyObject *). >>> Please forgive my ignorance/google skills if its already possible! >>>>> >>>>> unwrap_table code: >>>>> >>> https://github.com/apache/arrow/blob/c39e3508f93ea41410c2ae17783054d05592dc0e/python/pyarrow/public-api.pxi#L310 >>> < >>> https://github.com/apache/arrow/blob/c39e3508f93ea41410c2ae17783054d05592dc0e/python/pyarrow/public-api.pxi#L310 >>>> >>>>> >>>>> library pruning: >>>>> >>> https://github.com/finos/perspective/blob/python_arrow/cmake/modules/FindPyArrow.cmake#L53 >>> < >>> https://github.com/finos/perspective/blob/python_arrow/cmake/modules/FindPyArrow.cmake#L53 >>>> >>>>> >>>>> >>>>> >>>>> >>>>> Tim >>> >>>
