Right - I have a different app that uses sockets in another context for a similar goal.
The thing is - the Stream object is "advertised" (so to say) as a suitable holder for such data. E.g., looking at the docs for `pyarrow.ipc.open_stream()` and `pyarrow.ipc.NativeFile`, they specifically mention how this is the right approach when doing streaming, and I assumed that concurrent reading from that stream is a viable use case for such files. In addition, I haven't seen in the official Arrow docs anything warning against such concurrent reads. Perhaps I'm just completely ignorant of this topic and should've realized that a NativeFile can't support this use case, but I believe that a minimal warning against such "abuse" of the IPC protocol might be helpful in the future. Alternatively, advise users that wish to concurrently read and write to use a socket as the "file" connecting the two processes. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, August 19th, 2021 at 5:47 PM, Antoine Pitrou <[email protected]> wrote: > On Thu, 19 Aug 2021 14:31:56 +0000 > > Hagai Har-Gil [email protected] wrote: > > > Hello, > > > > My IPC use case involves writing a Windows-based real tlime data stream in > > Python, and reading it back concurrently with Rust. I tried using the IPC > > protocol to accomplish that, but I'm currently unsuccessful. The issue I'm > > facing is probably related to concurrent access to a file by the two > > different processes, where one is attempting to write and the other to > > read. Code examples can be found in this issue that I filed against the > > Rust (unofficial) IPC implementation > > (https://github.com/jorgecarleitao/arrow2/issues/301), but I believe that > > this problem isn't due to a faulty implementation. > > > > My question is a bit more broad - is my use case a valid one for the IPC > > protocol? > > Well... what you're trying to do is usually achived by having one > > process writing to e.g. a pipe or socket, and the other process reading > > from the other end of the pipe or socket. Using an actual on-disk file > > for that sounds a bit weird. > > Regards > > Antoine.
