gRPC has backpressure built in. Is your server in Java or Python/C++? If in Java, the server needs to explicitly poll isReady to respect backpressure. (This is rather wasteful, yes, and this will artificially throttle your peak bandwidth because of a long-standing flaw in gRPC-Java.)
If in Python/C++, the backpressure should be automatic (write calls will block), so the client should just make sure not to read from the reader until it actually wants data (there are lower-level options to tweak this, IIRC) On Thu, Jul 6, 2023, at 23:18, Wenbo Hu wrote: > Hi, > I'm using arrow flight to transfer data in distributed system, but > the lightning speed makes both client and server faces out of memory > issue. > For do_put and do_exchange method, the protocol provides stream > metadata reader/writer for client/server exchange control messages > along data stream. > But do_get only returns a FlightDataStream without any extra > control message can be used to communicate with each other. > Also, the returned FlightDataStream is unaware of client > canceling, while java has a cancel callback. > My solution is to use do_exchange to replace do_get for client > download data, or is there any better way to implement that? > > -- > --------------------- > Best Regards, > Wenbo Hu,
