Hello Luis, As you already mentioned, mapped files, Windows name for shared memory, need the size to be available ahead. This is the same on other operating systems, too. Flight will copy the data when transferring from one process to another. So there you will have the copy again.
So to actually better understand your use case: Why aren't you able to calculate the size beforehand? To construct an Arrow structure, you also need to know it's size. When using the builders for incremental creation, we have tuned everything to minimize the amount of copies but they still copy when the size doesn't match and we cannot extend the existing memory region in-place. Cheers Uwe > Am 21.09.2020 um 10:59 schrieb Louis C <l...@outlook.fr>: > > > Hello, > > Excuse me if this is a frequent question but I am trying to find a way to > share data (Feather/Parquet tables for instance) between different processes > (IPC), the ideal would be to use shared memory as I could write data with a > process and read it with another one without any copy. The different > processes could be 2 separate C++ processes or 1 C++ program with a Python > one. The platform would be primarily Windows, but it would be better it if it > was also compatible with Linux. > > As I understand it Arrow should be able to do something like this, but I > can’t find the proper way to do it. > > I looked into mapped files, but it seems like it is only useful to read data > as one needs to have the size of the written data before writing it in a > mapped file. I tried Flight too, but this is not shared memory IPC. There was > also Plasma, but it seems not to be fully maintained anymore (and not > available for Windows for the moment). > Is there a way to achieve this with Arrow ? > > Kind regards > > Louis C
