marsupialtail commented on issue #13300: URL: https://github.com/apache/arrow/issues/13300#issuecomment-1145112047
Making a self contained example might be difficult. The code is in here: https://github.com/marsupialtail/quokka/tree/replace-redis. High level: I have a distributed actor system which pushes around Arrow tables. I initialize a FlightServer on each machine. Other actors might push data to servers on remote machines and fetch data from servers on local machine (which have been pushed remotely). There can be many such actors. You can look at how I use the FlightServer in https://github.com/marsupialtail/quokka/blob/replace-redis/pyquokka/quokka_runtime.py#L420 and https://github.com/marsupialtail/quokka/blob/replace-redis/pyquokka/flight.py. How I use the FlightClient is here: https://github.com/marsupialtail/quokka/blob/replace-redis/pyquokka/nodes.py#L604. Each actor constantly calls the get_batches function to try to get its latest batches in the local FlightServer. The behavior is that for some actors this work fine but for others it hangs. In particular the observed behavior is that it gets stuck in the iterator. As in the actor seems to hang in the loop here: https://github.com/marsupialtail/quokka/blob/replace-redis/pyquokka/nodes.py#L604. I'm sorry if this is too much information -- I can try to produce a minimally reproducible example, but that might be kind of difficult. If you are interested in helping, you can also contact me directly at [email protected] or on the Substrait Slack. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
