I've already checked process threads using ps command, and threads are not leaking. Still, it is more than possible that there are memory leaks within my Cgo code, or gstreamer as well (Which I'm running with Cgo as well)
But before I'm starting to chase this path, I would like to be sure heapSys or StackInUse stats are also including Cgo memory allocations, because my understanding was that Go runtime is not reflecting memory allocations done from C code. I've aleady found Go memory leak before with pprof (Due to leak seen in heapObjects count) and another leak in goRoutine and fix them... But now I'm with the mystery of pprof showing low memory usage but stackInUse & HeapSys are increasing and are not reflected in pprof. So - Are HeapSys or Stack In use includes memory allocations done in C code? If so, I'll move on and start chasing C code (Possibly using Valgrind? Can I use it to profile my Go application with Cgo?) On Sat, Mar 26, 2022, 22:54 robert engels <reng...@ix.netcom.com> wrote: > Are you certain your CGo code isn’t creating threads? > > On Mar 26, 2022, at 1:10 PM, Shlomi Amit <shlo...@gmail.com> wrote: > > Yes. I already monitoring the runtime stat of number of go routines (Can > be seen in the screenshot as well) and it's not increasing. > > On Sat, Mar 26, 2022, 20:40 robert engels <reng...@ix.netcom.com> wrote: > >> Are you certain the number of Go routines is not increasing - each Go >> routine requires stack. See https://tpaschalis.github.io/goroutines-size/ >> >> On Mar 24, 2022, at 3:18 PM, Shlomi Amit <shlo...@gmail.com> wrote: >> >> Hi. >> >> I’m trying to find a memory leak in my application. >> I’ve added some runtime memory stats to the logs (Heap & Stack stats.. go >> routine and heapObject counts). >> I can see that from time to time m.StackInuse is increasing without any >> obvious reason from the application perspective (At least not one that I >> can find yet). >> Here some additional runtime mem stats (Taken at the same time): >> StackInUse 56MB >> HeapSys: 147MB. >> HeapAlloc: 97MB >> >> While StackInUse and HeapSys seems to increase from time to time, >> HeapAlloc stays about the same during 6 hours while the application is >> running. >> There seems to be a bit of correlation between StackInUse and HeapSys, >> but overtime StackInUse increase more than HeapSys. (Is HeapSize includes >> StackInUse?) >> >> In addition to adding runtime memory logs, I’m also creating periodic >> heap profile dumps. >> My problem is that when analyzing the heap with pprof, it gives me no >> clue why StackInUse is so high. >> The pprof inuse_space shows: >> “Showing nodes accounting for 55.31MB, 93.25% of 59.31MB total” >> What this total MB represent? It doesn’t seem to match HeapAlloc, HeapSys >> or StackInUse. >> Does pprof heap profile even include StackInUse? >> >> I really need to understand where the leak is coming from, but after >> looking in many places, the memory stats are still not clear to me, and >> neither what memory stat pprof heap profile really represent. >> Note that I’m also logging HeapObjects count and I don’t see any leak >> there… It’s just StackInUse increasing from time to time (As it seems, it's >> always double itself... ~8MB->16MB->32), and HeapSys. >> >> Note that I’m also using Cgo, but my understanding is Cgo memory >> allocations will not be reflected by the runtime memory stats. Is this >> correct and I assume if runtime memory stats are increasing this is >> defiantly because of go code and not C code? >> I hope I was clear, but added a screenshot for the different memory stats. >> Marked in red the point in times that stackInUse increase. (My current >> understanding which might be wrong, is that stackInUse is not included in >> HeapSys, this is why they are stacked in the graph). >> >> I know my write is a bit messed up and you might not really be sure >> what's being asked, so in if I'll try to summarize: >> >> >> 1. What stackInUse represents? Is it part of HeapSys? >> 2. What HeapSys represents? (Both it and StackInUse are way more high >> than heapAlloc) >> 3. Why pprof inuse_space doesn't seem to have any notion of stuff >> which was allocated by StackInUse or HeapSys? It seems to pretty much >> corelate with HeapAlloc. >> 4. Since I'm also using Cgo, I do understand C memory allocation will >> not be reflected in pprof, perhaps they are being reflected by StackInUse >> or HeapInUse? >> Considering StackInUse seems to double itself in size when it >> increase, it seems like some kind of memory pre-allocation to have enough >> headroom than needed, not something which I expect to happen by C code. >> 5. Most importantly, with the below memory stats being increased, >> what's the approach to investigate? >> 6. Currently I'm using docker, and sometimes get OOM. What's best >> memory stats which will the most close to describe container RSS memory in >> use? (Specifically, my docker container was limited to 300MiB) >> >> >> I know these are many question, but I did try and go over the runtime >> memory stats documentation, and I'm still very confused, especially not >> after the actual memory stats I'm seeing. >> BTW, I'm using go 1.17.5 >> >> <Untitled.jpg> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com >> <https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> <Untitled.jpg> >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com > <https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAbwE0O6Wj2ngcGymiJhrwUxOJLQ_5TZbJgqWnqm_c_BZc8-bQ%40mail.gmail.com.
