Yes. I already monitoring the runtime stat of number of go routines (Can be seen in the screenshot as well) and it's not increasing.
On Sat, Mar 26, 2022, 20:40 robert engels <reng...@ix.netcom.com> wrote: > Are you certain the number of Go routines is not increasing - each Go > routine requires stack. See https://tpaschalis.github.io/goroutines-size/ > > On Mar 24, 2022, at 3:18 PM, Shlomi Amit <shlo...@gmail.com> wrote: > > Hi. > > I’m trying to find a memory leak in my application. > I’ve added some runtime memory stats to the logs (Heap & Stack stats.. go > routine and heapObject counts). > I can see that from time to time m.StackInuse is increasing without any > obvious reason from the application perspective (At least not one that I > can find yet). > Here some additional runtime mem stats (Taken at the same time): > StackInUse 56MB > HeapSys: 147MB. > HeapAlloc: 97MB > > While StackInUse and HeapSys seems to increase from time to time, > HeapAlloc stays about the same during 6 hours while the application is > running. > There seems to be a bit of correlation between StackInUse and HeapSys, but > overtime StackInUse increase more than HeapSys. (Is HeapSize includes > StackInUse?) > > In addition to adding runtime memory logs, I’m also creating periodic heap > profile dumps. > My problem is that when analyzing the heap with pprof, it gives me no clue > why StackInUse is so high. > The pprof inuse_space shows: > “Showing nodes accounting for 55.31MB, 93.25% of 59.31MB total” > What this total MB represent? It doesn’t seem to match HeapAlloc, HeapSys > or StackInUse. > Does pprof heap profile even include StackInUse? > > I really need to understand where the leak is coming from, but after > looking in many places, the memory stats are still not clear to me, and > neither what memory stat pprof heap profile really represent. > Note that I’m also logging HeapObjects count and I don’t see any leak > there… It’s just StackInUse increasing from time to time (As it seems, it's > always double itself... ~8MB->16MB->32), and HeapSys. > > Note that I’m also using Cgo, but my understanding is Cgo memory > allocations will not be reflected by the runtime memory stats. Is this > correct and I assume if runtime memory stats are increasing this is > defiantly because of go code and not C code? > I hope I was clear, but added a screenshot for the different memory stats. > Marked in red the point in times that stackInUse increase. (My current > understanding which might be wrong, is that stackInUse is not included in > HeapSys, this is why they are stacked in the graph). > > I know my write is a bit messed up and you might not really be sure what's > being asked, so in if I'll try to summarize: > > > 1. What stackInUse represents? Is it part of HeapSys? > 2. What HeapSys represents? (Both it and StackInUse are way more high > than heapAlloc) > 3. Why pprof inuse_space doesn't seem to have any notion of stuff > which was allocated by StackInUse or HeapSys? It seems to pretty much > corelate with HeapAlloc. > 4. Since I'm also using Cgo, I do understand C memory allocation will > not be reflected in pprof, perhaps they are being reflected by StackInUse > or HeapInUse? > Considering StackInUse seems to double itself in size when it > increase, it seems like some kind of memory pre-allocation to have enough > headroom than needed, not something which I expect to happen by C code. > 5. Most importantly, with the below memory stats being increased, > what's the approach to investigate? > 6. Currently I'm using docker, and sometimes get OOM. What's best > memory stats which will the most close to describe container RSS memory in > use? (Specifically, my docker container was limited to 300MiB) > > > I know these are many question, but I did try and go over the runtime > memory stats documentation, and I'm still very confused, especially not > after the actual memory stats I'm seeing. > BTW, I'm using go 1.17.5 > > <Untitled.jpg> > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > <Untitled.jpg> > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com.
