Thanks for the additional information. On Wed, May 13, 2020 at 11:41 PM <ravi.y0...@gmail.com> wrote:
> Thanks Eric for the reply. > > Couple of things. I am not able to get this when you say that gRPC java is > not optimized for this type of large file. My individual message size is > 1024 bytes only. The program reads this much bytes from the file at a time > and sends this to onNext() call of stream observer. So, each message sent > on wire for each onNext() must be little over 1kb only. > That makes sense; what I was considering as a possibility was that the messages were going to onNext() *much* faster than they were going out on the wire. In the worst case, the entire file would be buffered into memory before anything was sent over the wire. > > Also, I just changed the gRPC version to *1.29.0* and saw that the native > memory usage is almost zero with this version. The high usage that I saw > was on version *1.21.0*. > > Even better; most likely the memory usage was optimized or, if a leak, fixed since version 1.21. Thanks for the update! Eric > Regards, > > On Thursday, May 14, 2020 at 3:26:12 AM UTC+5:30, Eric Gribkoff wrote: >> >> gRPC Java may not be optimized for this type of large file transfer, and >> is likely copying the provided bytes more than once which could explain why >> you see additional direct memory allocation. These should be all released >> once the data goes out over the wire. It looks from your code snippet that >> the entire file is immediately buffered - we would expect to see the memory >> consumption decrease as bytes are sent out to the client. >> >> Further investigation of this would probably be best carried out on the >> grpc java repository at https://github.com/grpc/grpc-java/ if you'd care >> to file an issue/reproduction case there. >> >> Thanks, >> >> Eric >> On Monday, May 11, 2020 at 3:22:35 AM UTC-7 ravi....@gmail.com wrote: >> >>> I have been testing gRPC behaviour for larger message size on my >>> machine. I have got a single client to which I am streaming a video file of >>> 603mb via streaming gRPC. I ran into OOM while testing and found that in >>> case of slow clients, response messages were getting queued up and I was >>> getting below error msg: >>> io.netty.util.internal.OutOfDirectMemoryError: failed to allocate >>> 16777216 byte(s) of direct memory (used: 1873805512, max: 1890582528) >>> >>> >>> Now this is fixable via flow control(using onReady() channel handler) >>> but out of curiosity I increased my direct memory to 4GB >>> via -XX:MaxDirectMemorySize=4g jvm flag to force queuing up of all >>> response messages and hence the client can consume on it's own pace. It got >>> completed successfully. But I observed that I ended up using 2.4GB of >>> direct memory. I checked it via usedDirectMemory() eposed by netty >>> for ByteBufAllocatorMetric. >>> >>> *Isn't this too much for a 603mb file as it is 3 times more than the >>> total file size*. Below is the code snippet that I am using: >>> >>> stream = new FileInputStream(file); >>> byte[] buffer = new byte[1024]; >>> ByteBufAllocator byteBufAllocator = ByteBufAllocator.DEFAULT; >>> int length; >>> while ((length = stream.read(buffer)) > 0) { >>> response.onNext(VideoResponse.newBuilder().setVideoBytes(ByteString. >>> copyFrom(buffer)).build()); >>> if (ByteBufAllocator.DEFAULT instanceof ByteBufAllocatorMetricProvider) >>> { >>> ByteBufAllocatorMetric metric = ((ByteBufAllocatorMetricProvider >>> ) byteBufAllocator).metric(); >>> System.out.println(metric.usedDirectMemory()/(1024*1024)); >>> } >>> } >>> >>> -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to grpc-io+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/e9584663-f06e-4a22-9307-520409511eab%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/e9584663-f06e-4a22-9307-520409511eab%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALUXJ7gCp6b1noZGBbBQqMqDjsFrNhnTMonN%2BHmrEWQQGGh0kg%40mail.gmail.com.
