Thanks both, immensely helpful in narrowing the problem.
On Tue, Dec 14, 2021 at 7:12 AM Miha Vrhovnik <miha.vrhov...@gmail.com> wrote: > And if you are using a cgo and didn't replace the standard malloc with > jemaloc you are probably suffering from memory fragmentation or whatever > the hell happens. > We had to restart the processors every 1000 iterations because of that. > There was no memory leak according to valgrind. (about 300kb), yet the RSS > was constantly rising. Calling exactly the same C api with a C shim the RSS > was constant about 100M. And api was extremely simplified all large memory > allocations happened on the C side. pprof showed small amount of used > memory yet the RSS was in gigabytes. > > > BR; > Miha > On Monday, December 13, 2021 at 7:24:57 PM UTC+1 kra...@skepticism.us > wrote: > >> If you're using cgo to link with C/C++ code then a memory leak in that >> API is the first place to start looking.. >> >> On Mon, Dec 13, 2021 at 10:14 AM Gareth Owenson <g.ow...@slcyber.io> >> wrote: >> >>> Hi all >>> >>> We have a go application that processes large numbers of documents. It >>> behaves like it has a memory leak, eventually exhausting system ram after a >>> few hours. I've setup pprof and looked at the various outputs, and that is >>> adamant that the application is only using some 50MB of ram, yet Linux top >>> reports it using much more (and growing). >>> >>> I've tried setting madvdontneed to no avail. >>> >>> Any help on better ways to identify the cause would be greatly >>> appreciated. >>> >>> best >>> Gareth >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/409a950d-b270-4ef7-a965-ea3d450e74f5n%40googlegroups.com >>> <https://groups.google.com/d/msgid/golang-nuts/409a950d-b270-4ef7-a965-ea3d450e74f5n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Kurtis Rader >> Caretaker of the exceptional canines Junior and Hank >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "golang-nuts" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/golang-nuts/lGcsCbY282I/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/0707e376-d6d7-40ff-9803-e5534c98b62dn%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/0707e376-d6d7-40ff-9803-e5534c98b62dn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CALjXe529FreQk9eWddEGkCAmU1aKrrEuhPux22gJ1tNNxj%3DNcg%40mail.gmail.com.
