amoeba commented on issue #38617: URL: https://github.com/apache/arrow/issues/38617#issuecomment-1894783486
Hi @pitrou, I did some testing so we're clear what actually breaks and how. I've pushed four separate reproducers to a branch on my fork at https://github.com/amoeba/arrow/tree/gh-38617/gh-38617 (see README in that folder). I tested using `os.fork` directly and also `multiprocessing.Pool` (the latter was how this issue was originally encountered) and the summary of what I found is: - Forking then using a FlightClient from the parent process seems to work fine - Other combinations do not: - Forking then using a FlightClient from the child process: The RPC never appears to run, the server never appears to see it. - Forking then using the FlightClient from both the parent and the child: Different failures happen including segfault, sigbus, hangs, cryptic GRPC errors - Forking using `multiprocesessing.Pool`, giving the workers access to the FlightClient via global (since we can't pickle): No child calls to the RPC ever appear to start, they just hang. Let me know if other tests would be useful and if the ones I've done make sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
