On Monday, October 30, 2023 at 10:12:08 PM UTC-7 Robert Engels wrote:
What is the average wall time of a request? The average latency is a few hundred milliseconds. Based on what you wrote it appears that handling a single request generates a lot of garbage - high allocation rate - and for this to be significant I suspect the runtime is also significant - which implies to me a spawn and destroy request handler is your best bet. I actually didn't quite get your suggestion earlier. We are using gRPC and I think for each request we already have separate goroutines to handle it. Can you explain a little bit more about spawn and destroy request handler? On Oct 30, 2023, at 11:56 PM, Zhihui Jiang <coboy...@gmail.com> wrote: Hi Michael, Jason and Robert, thanks a lot for the replies and suggestions! I did some profiling today, here are some specific findings: 1, CPUs used for GC is around 35% after we enabled soft memory limit, and it was 45%+ before. I don't have too much experience here on how much CPU we should spend on GCs ideally, but my intuition 35% is pretty high. 2, For GC, most of the CPU is on *runtime.scanObject* which I guess is dependent on how many object we allocate and how fast that is. 3, With some further look at the heap profiling, it turns out most of the objects (70%+) allocated are due to complex protobuf messages we use for communications between services which can be big and might have deep-nested submessages. On Monday, October 30, 2023 at 2:19:23 PM UTC-7 Michael Knyszek wrote: I second Jason's message, and +1 to off-heap memory as a last resort. Yes, indeed. One of the advantage using Go is we don't need to manage memory by ourselves, I will try other options first and see how much we can improve. Here are a few more details: For a starting point on how to reduce memory allocations directly, see https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require restructuring your program in places. (e.g. passing byte slices to functions to be filled instead of returning byte slices; that sort of thing.) RE: pooling memory, take a look look at sync.Pool ( https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at reducing the number of memory allocations that are made in the steady-state. Object pooling is actually one of the most promising option we are trying to implement right now. One quick question: is sync.Pool also feasible for complex protobuf messages? any pitfall we should be take into consideration? On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote: Similar to Robert's suggestion, you could just use non-GC-ed memory within the process. https://github.com/glycerine/offheap provides an example. The central idea is that the Go GC will never touch memory that you have requested yourself from the OS. So you can make your own Arenas. https://en.wikipedia.org/wiki/Region-based_memory_management But I would save these as last resorts of course. Before that: a) can you reduce the objects allocated per request? b) can you allocate everything else on the stack? There are flags to see why things are escaping to the heap, use those in your analysis. (This is by far the simplest and fastest thing. Since the stack is automatically unwound when the user request finishes, typically, there is no GC to do.) Will try this out and let you know if we have interesting findings here. c) can you allocate a pool of objects that is just reused instead of allocating for each new user request? d) Is there anything that can be effectively cached and re-used instead of allocated? Good point! We actually have an in-memory cache which already haas very high cache hit ratio of 95%+. Seems not too much headroom here to further reduce the CPUs on GC. Use the profiler pprof to figure out what is going on. Thanks! pprof indeed is very helpful tool and the problem we are facing seems to boil down to too many large/complex protobuf message passed around different services which allocates too many objects during the proto unmarshal. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2e3ac44e-923b-4b6b-88ec-743f8474c83an%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/2e3ac44e-923b-4b6b-88ec-743f8474c83an%40googlegroups.com?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/29875b49-316d-4332-9854-35da4c043005n%40googlegroups.com.
