What is the average wall time of a request? Based on what you wrote it appears that handling a single request generates a lot of garbage - high allocation rate - and for this to be significant I suspect the runtime is also significant - which implies to me a spawn and destroy request handler is your best bet.
> On Oct 30, 2023, at 11:56 PM, Zhihui Jiang <coboy123...@gmail.com> wrote: > > Hi Michael, Jason and Robert, thanks a lot for the replies and suggestions! > > I did some profiling today, here are some specific findings: > 1, CPUs used for GC is around 35% after we enabled soft memory limit, and it > was 45%+ before. I don't have too much experience here on how much CPU we > should spend on GCs ideally, but my intuition 35% is pretty high. > 2, For GC, most of the CPU is on runtime.scanObject which I guess is > dependent on how many object we allocate and how fast that is. > 3, With some further look at the heap profiling, it turns out most of the > objects (70%+) allocated are due to complex protobuf messages we use for > communications between services which can be big and might have deep-nested > submessages. > > On Monday, October 30, 2023 at 2:19:23 PM UTC-7 Michael Knyszek wrote: > I second Jason's message, and +1 to off-heap memory as a last resort. > Yes, indeed. One of the advantage using Go is we don't need to manage memory > by ourselves, I will try other options first and see how much we can improve. > Here are a few more details: > > For a starting point on how to reduce memory allocations directly, see > https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require > restructuring your program in places. (e.g. passing byte slices to functions > to be filled instead of returning byte slices; that sort of thing.) > RE: pooling memory, take a look look at sync.Pool > (https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at > reducing the number of memory allocations that are made in the steady-state. > Object pooling is actually one of the most promising option we are trying to > implement right now. One quick question: is sync.Pool also feasible for > complex protobuf messages? any pitfall we should be take into consideration? > > On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote: > Similar to Robert's suggestion, you could just use non-GC-ed memory within > the process. > > https://github.com/glycerine/offheap provides an example. > > The central idea is that the Go GC will never touch memory that you have > requested > yourself from the OS. So you can make your own Arenas. > https://en.wikipedia.org/wiki/Region-based_memory_management > > But I would save these as last resorts of course. Before that: > > a) can you reduce the objects allocated per request? > b) can you allocate everything else on the stack? There are flags to see why > things are escaping to the heap, use those in your analysis. > (This is by far the simplest and fastest thing. Since the stack is > automatically unwound when the user request finishes, typically, there is no > GC to do.) > Will try this out and let you know if we have interesting findings here. > c) can you allocate a pool of objects that is just reused instead of > allocating for each new user request? > d) Is there anything that can be effectively cached and re-used instead of > allocated? > Good point! We actually have an in-memory cache which already haas very high > cache hit ratio of 95%+. Seems not too much headroom here to further reduce > the CPUs on GC. > > Use the profiler pprof to figure out what is going on. > Thanks! pprof indeed is very helpful tool and the problem we are facing seems > to boil down to too many large/complex protobuf message passed around > different services which allocates too many objects during the proto > unmarshal. > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/2e3ac44e-923b-4b6b-88ec-743f8474c83an%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0A66E87C-1BD6-4486-B152-763188B404C7%40ix.netcom.com.
