Thanks Uwe, Wes, Pearu and Antoine. This is in the pyarrow docs, but no
example, so I'll open up a JIRA so that it might be more obvious the
next person.
On 11/8/18 12:59 PM, Uwe L. Korn wrote:
Hello Randy,
you are looking for
https://arrow.apache.org/docs/python/generated/pyarrow.foreign_buffer.html#pyarrow.foreign_buffer
This takes an address, size and a Python object for having a reference on the
object. In your case the last one can be None. Note that this will not do a
copy and thus directly reference the shared memory.
Uwe
On Thu, Nov 8, 2018, at 6:49 PM, Randy Zwitch wrote:
Within OmniSci (MapD), we have the following code that takes a pointer
and length and reads to a NumPy array before calling py_buffer:
https://github.com/omnisci/pymapd/blob/master/pymapd/shm.pyx#L31-L52
Is it possible to eliminate the NumPy step and go directly do an Arrow
buffer? There is both a concern that we're doing an unnecessary memory
copy, as well as wanting to defer to the Arrow way of doing things as
much as possible rather than having our own shims like these.
