[go-nuts] Re: Debugging memory leak when a GO process crashed

[Vladimir Varankin]

Hey Mariappan,

>From my experience, there are several possible options:

I believe you can use GOTRACEBACK=crash env variable (or its equivalent in 
the runtime/debug package in the std) to get a coredump on the crash. See 
this old post from JBD [1], that explored this.

If you clearly observe, that the application "leaks" its memory (or it 
"leaks goroutines") via the metrics — simple by looking at the trends in 
the app's memory usage — you may collect several pprof files (heap and 
goroutines), with some interval. Then explore the diff between the two 
profiles with pprof tool's diff option. To help automating this, the 
continuous profiling became a fairly trendy thing in the recent years. I 
won't recommend the actual project I explained back then in the [3], but 
the overall idea still valid, and there are several big and small offerings 
for that nowadays.

Hope this will help.

[1]: https://rakyll.org/coredumps/
[2]: https://pkg.go.dev/runtime/debug#SetTraceback
[3]: https://medium.com/@tvii/continuous-profiling-and-go-6c0ab4d2504b

On Tuesday, January 10, 2023 at 8:55:18 AM UTC+1 mariappa...@gmail.com 
wrote:

> Hello Go experts,
>
> I could able to find the solution to debug memory leaks of GO process, 
> when it is running by using PPROF. Is it possible to collect the heap 
> profile for debugging, when GO process crashed using core dump? or what is 
> the recommended way to root cause the memory leak? Based on the need, 
> enabling PPROF and debugging may not be possible in the production 
> environment. Please help. 
>
> Best Regards
> Mariappan
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b33dc147-7099-494c-8a04-e4507d410d8dn%40googlegroups.com.

Re: [go-nuts] Re: Debugging memory leak when a GO process crashed

[Robert Engels]

You can certainly have memory leaks in Go - due to undesired references 
retaining the object. What the op is asking for is a memory dump at OOM crash. 

If the dump shows 1M object X and the author believes there should only be 1 - 
you have a leak   

> On Jan 12, 2023, at 4:01 AM, mariappan balraj  
> wrote:
> 
> 
> Hi Brian,
> 
> A Go program can crash because of Out of Memory. Yes it can be because of 
> goroutine leak. Or it can have some reference to global variables, but it 
> should be freed? How to debug this?
> When the leak is happening while using CGO, is there any tool to debug this?
> 
> Best Regards
> Mariappan
> 
> On Thu, Jan 12, 2023 at 2:19 PM Brian Candler  wrote:
>> I think you're not being clear on your problem you're trying to solve.
>> 
>> There are no "memory leaks" in Go in the traditional sense (omitting to free 
>> an allocation) because of the garbage collector. Therefore, if memory is not 
>> being freed in a pure Go program, it's because you are keeping a reference 
>> to it somewhere - e.g. in a global variable, directly or indirectly. Or it 
>> could be referenced from a local variable in a goroutine which isn't 
>> terminating - in that case I'd say that's a goroutine leak, not a memory 
>> leak.  Is that the sort of thing you're trying to debug?
>> 
>> Or, is the situation that you're using CGO and forgetting to free some 
>> allocations in your C code?
>> 
>> Or, I've seen some people complain that Go doesn't not release unused memory 
>> back to the operating system as quickly as they'd like or expect, showing as 
>> high RSS. If that's your issue then it can be discussed.
>> 
>> On Thursday, 12 January 2023 at 03:58:52 UTC mariappa...@gmail.com wrote:
>>> Hello Go experts,
>>> 
>>> Can someone please help with this?
>>> 
>>> Best Regards
>>> Mariappan
>>> 
>>> On Tue, Jan 10, 2023 at 1:24 PM mariappan balraj  
>>> wrote:
 Hello Go experts,
 
 I could able to find the solution to debug memory leaks of GO process, 
 when it is running by using PPROF. Is it possible to collect the heap 
 profile for debugging, when GO process crashed using core dump? or what is 
 the recommended way to root cause the memory leak? Based on the need, 
 enabling PPROF and debugging may not be possible in the production 
 environment. Please help. 
 
 Best Regards
 Mariappan
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/19a761a1-7707-4300-9128-93f22b9ee342n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAKKWi6SKZr9x7-YoJNLLp57815bzg40wjCsYSYxtd7SQ2VMEUA%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/390B033D-44D7-469B-BB36-88BF5A5D6D57%40ix.netcom.com.

Re: [go-nuts] Re: Debugging memory leak when a GO process crashed

[mariappan balraj]

Hi Brian,

A Go program can crash because of Out of Memory. Yes it can be because of
goroutine leak. Or it can have some reference to global variables, but it
should be freed? How to debug this?
When the leak is happening while using CGO, is there any tool to debug this?

Best Regards
Mariappan

On Thu, Jan 12, 2023 at 2:19 PM Brian Candler  wrote:

> I think you're not being clear on your problem you're trying to solve.
>
> There are no "memory leaks" in Go in the traditional sense (omitting to
> free an allocation) because of the garbage collector. Therefore, if memory
> is not being freed in a pure Go program, it's because you are keeping a
> reference to it somewhere - e.g. in a global variable, directly or
> indirectly. Or it could be referenced from a local variable in a goroutine
> which isn't terminating - in that case I'd say that's a goroutine leak, not
> a memory leak.  Is that the sort of thing you're trying to debug?
>
> Or, is the situation that you're using CGO and forgetting to free some
> allocations in your C code?
>
> Or, I've seen some people complain that Go doesn't not release unused
> memory back to the operating system as quickly as they'd like or expect,
> showing as high RSS. If that's your issue then it can be discussed.
>
> On Thursday, 12 January 2023 at 03:58:52 UTC mariappa...@gmail.com wrote:
>
>> Hello Go experts,
>>
>> Can someone please help with this?
>>
>> Best Regards
>> Mariappan
>>
>> On Tue, Jan 10, 2023 at 1:24 PM mariappan balraj 
>> wrote:
>>
>>> Hello Go experts,
>>>
>>> I could able to find the solution to debug memory leaks of GO process,
>>> when it is running by using PPROF. Is it possible to collect the heap
>>> profile for debugging, when GO process crashed using core dump? or what is
>>> the recommended way to root cause the memory leak? Based on the need,
>>> enabling PPROF and debugging may not be possible in the production
>>> environment. Please help.
>>>
>>> Best Regards
>>> Mariappan
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/19a761a1-7707-4300-9128-93f22b9ee342n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKKWi6SKZr9x7-YoJNLLp57815bzg40wjCsYSYxtd7SQ2VMEUA%40mail.gmail.com.

[go-nuts] Re: Debugging memory leak when a GO process crashed

[Brian Candler]

I think you're not being clear on your problem you're trying to solve.

There are no "memory leaks" in Go in the traditional sense (omitting to 
free an allocation) because of the garbage collector. Therefore, if memory 
is not being freed in a pure Go program, it's because you are keeping a 
reference to it somewhere - e.g. in a global variable, directly or 
indirectly. Or it could be referenced from a local variable in a goroutine 
which isn't terminating - in that case I'd say that's a goroutine leak, not 
a memory leak.  Is that the sort of thing you're trying to debug?

Or, is the situation that you're using CGO and forgetting to free some 
allocations in your C code?

Or, I've seen some people complain that Go doesn't not release unused 
memory back to the operating system as quickly as they'd like or expect, 
showing as high RSS. If that's your issue then it can be discussed.

On Thursday, 12 January 2023 at 03:58:52 UTC mariappa...@gmail.com wrote:

> Hello Go experts,
>
> Can someone please help with this?
>
> Best Regards
> Mariappan
>
> On Tue, Jan 10, 2023 at 1:24 PM mariappan balraj  
> wrote:
>
>> Hello Go experts,
>>
>> I could able to find the solution to debug memory leaks of GO process, 
>> when it is running by using PPROF. Is it possible to collect the heap 
>> profile for debugging, when GO process crashed using core dump? or what is 
>> the recommended way to root cause the memory leak? Based on the need, 
>> enabling PPROF and debugging may not be possible in the production 
>> environment. Please help. 
>>
>> Best Regards
>> Mariappan
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/19a761a1-7707-4300-9128-93f22b9ee342n%40googlegroups.com.

[go-nuts] Re: Debugging memory leak when a GO process crashed

[mariappan balraj]

Hello Go experts,

Can someone please help with this?

Best Regards
Mariappan

On Tue, Jan 10, 2023 at 1:24 PM mariappan balraj 
wrote:

> Hello Go experts,
>
> I could able to find the solution to debug memory leaks of GO process,
> when it is running by using PPROF. Is it possible to collect the heap
> profile for debugging, when GO process crashed using core dump? or what is
> the recommended way to root cause the memory leak? Based on the need,
> enabling PPROF and debugging may not be possible in the production
> environment. Please help.
>
> Best Regards
> Mariappan
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKKWi6THY4LAe1Euc%2BwyOT%2By_Cb90hOAWQzqhcSK%2BE%2BGEWsjYg%40mail.gmail.com.