On Sun, May 12, 2024 at 11:03 AM Yunhao Bai <cloudbai1...@gmail.com> wrote:
> When we are performing performance test and try to test the throughput for > the gRPC server, we find there is a very strange behavior: When we increase > the rate for unary RPC call to the gRPC server (e.g., from 15k rps to 50 > rps), we see a performance drop after some peak point. > I'm surprised that you are surprised. I thought it was well known that when you over-saturate a server performance eventually decreases. Although that looks pretty dramatic (not that I can read the image). After our investigate, we find that there are many "default-executor" > threads occupying CPU when there is a performance drop, and when there is > no such phenomenon, everything goes fine. I also post the CPU utils when > there is a performance drop and not with client rps equaling 50k. > Are you using grpc-java? "default-executor" definitely sounds like Java, but you mention event engine later. Event engine is only for the C-based implementation. Assuming you are using grpc-java, specifying executor() on the server (and channels) to a thread pool you manage can yield more performance. The default executor is a basic unbounded Executors.newCachedThreadPool(). If you know your maximum concurrency needs, limiting the maximum number of threads can improve performance (either via ThreadPoolExecutor or newFixedThreadPool()). Also, some workloads see benefit in using ForkJoinPool (which is a fixed-sized thread pool). If your application is mostly asynchronous, then choosing the number of threads based on number of cores makes sense, somewhere between a bit less than the number of cores to 2x. If you have a lot of blocking, then you'll just have to experiment. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CA%2B4M1oOL4jjJd-siQzWromo6W8Ver3b41A0YrqbP%3D6GcdDZ2Tg%40mail.gmail.com.
smime.p7s
Description: S/MIME Cryptographic Signature
