Hello,
Excuse me if this is a frequent question but I am trying to find a way to share
data (Feather/Parquet tables for instance) between different processes (IPC),
the ideal would be to use shared memory as I could write data with a process
and read it with another one without any copy. The different processes could be
2 separate C++ processes or 1 C++ program with a Python one. The platform would
be primarily Windows, but it would be better it if it was also compatible with
Linux.
As I understand it Arrow should be able to do something like this, but I can’t
find the proper way to do it.
I looked into mapped files, but it seems like it is only useful to read data as
one needs to have the size of the written data before writing it in a mapped
file. I tried Flight too, but this is not shared memory IPC. There was also
Plasma, but it seems not to be fully maintained anymore (and not available for
Windows for the moment).
Is there a way to achieve this with Arrow ?
Kind regards
Louis C
