I have no idea what would be involved in attaching ASAN to Python, and suspect it may be "exciting," so I'm trying to see first if gRPC has any monitoring capability around its buffers.
One thing I did notice while reading through the codebase was unittests like this one <https://github.com/grpc/grpc/blob/master/test/core/end2end/tests/retry_exceeds_buffer_size_in_subsequent_batch.cc> about exceeding buffer sizes -- that does seem to trigger an ABORTED response, but the test was fairly hard to understand (not much commenting there...). Am I right in thinking that if this 4MB buffer is overflowed, that's somehow going to happen? On Fri, Jul 19, 2019 at 12:59 PM Lidi Zheng <li...@google.com> wrote: > Hi Yonatan, > > In gRPC Python side, the consumption of message is sequential, and won't > be kept in memory. > If you recall the batch operations, only if a message is sent to > application, will gRPC Python start another RECV_MESSAGE operation. > It's unlikely that the problem resided in Python space. > > In C-Core space, AFAIK for each TCP read, the size is 4MiB > <https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/tcp_posix.cc#L1177> > per > channel. > I think we have flow control both in TCP level and HTTP2 level. > > For debugging, did you try to use ASAN? For channel arg, I can only find > "GRPC_ARG_TCP_READ_CHUNK_SIZE" and "GRPC_ARG_MAX_RECEIVE_MESSAGE_LENGTH" > that might be related to your case. > > Lidi Zheng > > On Fri, Jul 19, 2019 at 12:48 PM Yonatan Zunger <zun...@humu.com> wrote: > >> Maybe a more concrete way of asking this question: Let's say we have a >> Python gRPC client making a response-streaming request to some gRPC server. >> The server starts to stream back responses. If the client fails to consume >> data as fast as the server generates it, I'm trying to figure out where the >> data would accumulate, and which memory allocator it would be using. >> (Because Python heap profiling won't see calls to malloc()) >> >> If I'm understanding correctly: >> >> * The responses are written by the server to the network socket at the >> server's own speed (no pushback controlling it); >> * These get picked up by the kernel network device on the client, and get >> pulled into userspace ASAP by the event loop, which is in the C layer of >> the gRPC client. This is stored in a grpc_byte_buffer and builds up there. >> * The Python client library exposes a response iterator, which is >> ultimately a _Rendezvous object; its iteration is implemented in >> _Rendezvous._next(), which calls cygrpc.ReceiveMessageOperation, which is >> what drains data from the grpc_byte_buffer and passes it to the protobuf >> parser, which creates objects in the Python memory address space and >> returns them to the caller. >> >> This means that if the client were to drain the iterator more slowly, >> data would accumulate in the grpc_byte_buffer, which is in the C layer and >> not visible to (e.g.) Python heap profiling using the PEP445 malloc hooks. >> >> If I am understanding this correctly, is there any way (without doing a >> massive amount of plumbing) to monitor the state of the byte buffer, e.g. >> with some gRPC debug parameter? And is there any mechanism in the C layer >> which limits the size of this buffer, doing something like failing the RPC >> if the buffer size exceeds some threshold? >> >> Yonatan >> >> On Thu, Jul 18, 2019 at 5:27 PM Yonatan Zunger <zun...@humu.com> wrote: >> >>> Hi everyone, >>> >>> I'm trying to debug a mysterious memory blowout in a Python batch job, >>> and one of the angles I'm exploring is that this may have to do with the >>> way it's reading data. This job is reading from bigtable, which is >>> ultimately fetching the actual data with a unidirectional streaming "read >>> rows" RPC. This takes a single request and returns a sequence of data >>> chunks, the higher-level client reshapes this into an iterator over the >>> individual data cells, and those are consumed by the higher-level program, >>> so that the next response proto is consumed once the program is ready to >>> parse it. >>> >>> Something I can't remember about gRPC internals: What, if anything, is >>> the pushback mechanism in unidirectional streaming? In the zero-pushback >>> case, it would seem that a server could yield results at any speed, which >>> would be accepted by the client and stored in gRPC's internal buffers until >>> it got read by the client code, which could potentially cause a large >>> memory blowout if the server wrote faster than the client read. Is this in >>> fact the case? If so, is there any good way to instrument and detect if >>> it's happening? (Some combination of gRPC debug flags, perhaps) If not, is >>> there some pushback mechanism I'm not thinking of? >>> >>> (Alas, I can't change the protocol in this situation; the server is run >>> by someone else) >>> >>> Yonatan >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "grpc.io" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to grpc-io+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/CAFk%3DnbT16yfxQ_%2BUkudCAkaADECw-XRbqvtC4u%3DbaEQ_Rv9VAA%40mail.gmail.com >> <https://groups.google.com/d/msgid/grpc-io/CAFk%3DnbT16yfxQ_%2BUkudCAkaADECw-XRbqvtC4u%3DbaEQ_Rv9VAA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAFk%3DnbSgze_g9n0CJCD%3D0_%3DBXMmkgaja%2BVwb4c-BxPBppERZ9g%40mail.gmail.com.
