hi Pasha, Copying dev@.
You can see how DuckDB interacts with the pyarrow data structures by the C interface here, maybe it's helpful https://github.com/cwida/duckdb/blob/master/tools/pythonpkg/duckdb_python.cpp We haven't defined a Python API (either C API level or Python API level) so that objects can advertise that they support the Arrow C interface -- it's a separate issue from the C interface itself (which doesn't have anything specifically to do with Python), and I agree it would probably be a good idea to have a standard way that we codify and document . Thanks Wes On Mon, Oct 19, 2020 at 12:34 PM Pasha Stetsenko <stpa...@gmail.com> wrote: > > Hi everybody, > > I've been reading http://arrow.apache.org/docs/format/CDataInterface.html, > which has been > "... inspired by the Python buffer protocol", and i can't find any details on > how to connect this > protocol with other libraries/applications. > > Here's what I mean: with the python buffer protocol, i can create a new type > and set its > `tp_as_buffer` field to a `PyBufferProcs` structure. This way any other > library can call > `PyObject_CheckBuffer()` on my object to check whether or not it supports the > buffer interface, > and then `PyObject_GetBuffer()` to use that interface. > > I could not find the corresponding mechanisms in the Arrow C data interface. > For example, consider the "Exporting a simple int32 array" tutorial in the > article above. After creating > `export_int32_type()`, `release_int32_type()`, `export_int32_array()`, > `release_int32_array()` > -- how do i announce to the world that these functions are available? > Conversely, if i want to > talk to an Arrow Table via this interface -- where do i find the endpoints > that return > `ArrowSchema` and `ArrowArray` structures? > > (I understand that there is an additional, more complicated API for accessing > arrow objects http://arrow.apache.org/docs/python/extending.html, but this > seems to be a completely different > API than what CDataInterface describes).
