You've requested the total allocated space (--alloc_space), not only the
heap used (--heap_inuse, or no flag).
So that 17GiB is the total allocated size, does NOT include the released!
2020. március 24., kedd 15:16:46 UTC+1 időpontban Nitish Saboo a következőt
írta:
>
> Hi,
>
> I have already gone through those links. They helped me to gather the mem
> profile and while analyzing the data(as given in those links) I have come
> across the following issue:
>
> While I was running the service for 100 minutes the 'top' command output
> was showing Mem% as 11.1. There was no increase in mem usage since I had
> not called 'LoadPatternDB()' method. I have 8GB of memory on the node where
> I am running the service. My issue is :
>
>
> - Why is it showing memory accounting for around 17GB? 11.1 % of 8GB
> is .88GB and my node is only of 8GB. I feel the way I gathered the mem
> profiling was not correct ..is it ?
>
> Please let me know where am I going wrong?
>
> Thanks,
> Nitish
>
> On Tue, Mar 24, 2020 at 5:32 PM Nitish Saboo <nitish...@gmail.com
> <javascript:>> wrote:
>
>> Hi,
>>
>> >>There is no root analysis available in Go. Read the paper I linked to.
>>
>> Sorry I did not get you. Which paper are you referring to?
>>
>> While I was running the service for 100 minutes the 'top' command output
>> was showing Mem% as 11.1. There was no increase in mem usage since I had
>> not called 'LoadPatternDB()' method.I have 8GB of memory on the node where
>> I am running the service. My issue is :
>>
>>
>> - Why is it showing memory accounting for around 17GB? 11.1 % of 8GB
>> is .88GB and my node is only of 8GB. I feel the way I gathered the mem
>> profiling was not correct ..is it ?
>>
>> Please advise me what am I missing?
>>
>> Thanks,
>> Nitish
>>
>> On Tue, Mar 24, 2020 at 1:28 AM Robert Engels <ren...@ix.netcom.com
>> <javascript:>> wrote:
>>
>>> Yes. You have a leak in your Go code. It shows you the object types that
>>> are taking up all of the space. There is no root analysis available in Go.
>>> Read the paper I linked to.
>>>
>>> On Mar 23, 2020, at 9:12 AM, Nitish Saboo <nitish...@gmail.com
>>> <javascript:>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> I used something like the following to generate a memprof for 100 minutes
>>>
>>> func main() {
>>> flag.Parse()
>>> if *cpuprofile != "" {
>>> f, err := os.Create(*cpuprofile)
>>> if err != nil {
>>> fmt.Println("could not create CPU profile: ", err)
>>> }
>>> defer f.Close() // error handling omitted for example
>>> if err := pprof.StartCPUProfile(f); err != nil {
>>> fmt.Print("could not start CPU profile: ", err)
>>> }
>>> defer pprof.StopCPUProfile()
>>> }
>>> timeout := time.After(100 * time.Minute)
>>> A_chan := make(chan bool)
>>> B_chan := make(chan bool)
>>> go util.A(A_chan)
>>> go util.B(B_chan)
>>> (..Rest of the code..)
>>>
>>> for {
>>> select {
>>> case <-A_chan:
>>> continue
>>> case <-B_chan:
>>> continue
>>> case <-timeout:
>>> break
>>> }
>>> break
>>> }
>>>
>>> if *memprofile != "" {
>>> count = count + 1
>>> fmt.Println("Generating Mem Profile:")
>>> fmt.Print(count)
>>> f, err := os.Create(*memprofile)
>>> if err != nil {
>>> fmt.Println("could not create memory profile: ", err)
>>> }
>>> defer f.Close() // error handling omitted for example
>>> runtime.GC() // get up-to-date statistics
>>> if err := pprof.WriteHeapProfile(f); err != nil {
>>> fmt.Println("could not write memory profile: ", err)
>>> }
>>>
>>> }
>>> }
>>>
>>> /Desktop/memprof:go tool pprof --alloc_space main mem3.prof
>>> Fetched 1 source profiles out of 2
>>> File: main
>>> Build ID: 99b8f2b91a4e037cf4a622aa32f2c1866764e7eb
>>> Type: alloc_space
>>> Time: Mar 22, 2020 at 7:02pm (IST)
>>> Entering interactive mode (type "help" for commands, "o" for options)
>>> (pprof) top
>>> Showing nodes accounting for 17818.11MB, 87.65% of 20329.62MB total
>>> <<<<<<<<<<<<<<<
>>> Dropped 445 nodes (cum <= 101.65MB)
>>> Showing top 10 nodes out of 58
>>> flat flat% sum% cum cum%
>>> 11999.09MB 59.02% 59.02% 19800.37MB 97.40%
>>> home/nsaboo/project/aws.Events
>>> 1675.69MB 8.24% 67.27% 1911.69MB 9.40%
>>> home/nsaboo/project/utils.Unflatten
>>> 627.21MB 3.09% 70.35% 1475.10MB 7.26%
>>> encoding/json.mapEncoder.encode
>>> 626.76MB 3.08% 73.43% 626.76MB 3.08%
>>> encoding/json.(*Decoder).refill
>>> 611.95MB 3.01% 76.44% 4557.69MB 22.42%
>>> home/nsaboo/project/lib.format
>>> 569.97MB 2.80% 79.25% 569.97MB 2.80% os.(*File).WriteString
>>> 558.95MB 2.75% 82.00% 2034.05MB 10.01% encoding/json.Marshal
>>> 447.51MB 2.20% 84.20% 447.51MB 2.20% reflect.copyVal
>>> 356.10MB 1.75% 85.95% 432.28MB 2.13% compress/flate.NewWriter
>>> 344.88MB 1.70% 87.65% 590.38MB 2.90% reflect.Value.MapKeys
>>>
>>> 1)Is this the correct way of getting a memory profile?
>>>
>>> 2)I ran the service for 100 minutes on a machine with 8GB memory. How am
>>> I seeing memory accounting for around 17GB?
>>>
>>> 3)I understand 'flat' means memory occupied within that method, but how
>>> come it shot up more than the available memory? Hence, asking if the above
>>> process is the correct way of gathering the memory profile.
>>>
>>> Thanks,
>>> Nitish
>>>
>>> On Fri, Mar 13, 2020 at 6:22 PM Michael Jones <michae...@gmail.com
>>> <javascript:>> wrote:
>>>
>>>> hi. get the time at the start, check the elapsed time in your infinite
>>>> loop, and trigger the write/exit after a minute, 10 minutes, 100 minutes,
>>>> ...
>>>>
>>>> On Fri, Mar 13, 2020 at 5:45 AM Nitish Saboo <nitish...@gmail.com
>>>> <javascript:>> wrote:
>>>>
>>>>> Hi Michael,
>>>>>
>>>>> Thanks for your response.
>>>>>
>>>>> That code looks wrong. I see the end but not the start. Look here and
>>>>> copy carefully:
>>>>>
>>>>> >>Since I did not want cpu profiling I omitted the start of the code
>>>>> and just added memory profiling part.
>>>>>
>>>>> Call at end, on way out.
>>>>>
>>>>> >>Oh yes, I missed that.I have to call memory profiling code at the
>>>>> end on the way out.But the thing is that it runs as a service in infinite
>>>>> for loop.
>>>>>
>>>>> func main() {
>>>>> flag.Parse()
>>>>> if *cpuprofile != "" {
>>>>> f, err := os.Create(*cpuprofile)
>>>>> if err != nil {
>>>>> fmt.Println("could not create CPU profile: ", err)
>>>>> }
>>>>> defer f.Close() // error handling omitted for example
>>>>> if err := pprof.StartCPUProfile(f); err != nil {
>>>>> fmt.Print("could not start CPU profile: ", err)
>>>>> }
>>>>> defer pprof.StopCPUProfile()
>>>>> }
>>>>>
>>>>> A_chan := make(chan bool)
>>>>> B_chan := make(chan bool)
>>>>> go util.A(A_chan)
>>>>> go util.B(B_chan)
>>>>> (..Rest of the code..)
>>>>>
>>>>> for {
>>>>> select {
>>>>> case <-A_chan:
>>>>> continue
>>>>> case <-B_chan:
>>>>> continue
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> What would be the correct way to add the memprofile code changes,
>>>>> since it is running in an infinite for loop ?
>>>>>
>>>>> Also, as shared by others above, there are no promises about how soon
>>>>> the dead allocations go away, The speed gets faster and faster version to
>>>>> version, and is impressive indeed now, so old versions are not the best
>>>>> to
>>>>> use, ubt even so, if the allocation feels small to th GC the urgency to
>>>>> free it will be low. You need to loop in allocating and see if the memory
>>>>> grows and grows.
>>>>>
>>>>> >> Yes, got it.I will try using the latest version of Go and check the
>>>>> behavior.
>>>>>
>>>>> Thanks,
>>>>> Nitish
>>>>>
>>>>> On Fri, Mar 13, 2020 at 6:20 AM Michael Jones <michae...@gmail.com
>>>>> <javascript:>> wrote:
>>>>>
>>>>>> That code looks wrong. I see the end but not the start. Look here and
>>>>>> copy carefully:
>>>>>> https://golang.org/pkg/runtime/pprof/
>>>>>>
>>>>>> Call at end, on way out.
>>>>>>
>>>>>> Also, as shared by others above, there are no promises about how soon
>>>>>> the dead allocations go away, The speed gets faster and faster version
>>>>>> to
>>>>>> version, and is impressive indeed now, so old versions are not the best
>>>>>> to
>>>>>> use, ubt even so, if the allocation feels small to th GC the urgency to
>>>>>> free it will be low. You need to loop in allocating and see if the
>>>>>> memory
>>>>>> grows and grows.
>>>>>>
>>>>>> On Thu, Mar 12, 2020 at 9:22 AM Nitish Saboo <nitish...@gmail.com
>>>>>> <javascript:>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have compiled my Go binary against go version 'go1.7 linux/amd64'.
>>>>>>> I added the following code change in the main function to get the
>>>>>>> memory profiling of my service
>>>>>>>
>>>>>>> var memprofile = flag.String("memprofile", "", "write memory profile
>>>>>>> to `file`")
>>>>>>>
>>>>>>> func main() {
>>>>>>> flag.Parse()
>>>>>>> if *memprofile != "" {
>>>>>>> f, err := os.Create(*memprofile)
>>>>>>> if err != nil {
>>>>>>> fmt.Println("could not create memory profile: ", err)
>>>>>>> }
>>>>>>> defer f.Close() // error handling omitted for example
>>>>>>> runtime.GC() // get up-to-date statistics
>>>>>>> if err := pprof.WriteHeapProfile(f); err != nil {
>>>>>>> fmt.Println("could not write memory profile: ", err)
>>>>>>> }
>>>>>>> }
>>>>>>> ..
>>>>>>> ..
>>>>>>> (Rest code to follow)
>>>>>>>
>>>>>>> I ran the binary with the following command:
>>>>>>>
>>>>>>> nsaboo@ubuntu:./main -memprofile=mem.prof
>>>>>>>
>>>>>>> After running the service for couple of minutes, I stopped it and
>>>>>>> got the file 'mem.prof'
>>>>>>>
>>>>>>> 1)mem.prof contains the following:
>>>>>>>
>>>>>>> nsaboo@ubuntu:~/Desktop/memprof$ vim mem.prof
>>>>>>>
>>>>>>> heap profile: 0: 0 [0: 0] @ heap/1048576
>>>>>>>
>>>>>>> # runtime.MemStats
>>>>>>> # Alloc = 761184
>>>>>>> # TotalAlloc = 1160960
>>>>>>> # Sys = 3149824
>>>>>>> # Lookups = 10
>>>>>>> # Mallocs = 8358
>>>>>>> # Frees = 1981
>>>>>>> # HeapAlloc = 761184
>>>>>>> # HeapSys = 1802240
>>>>>>> # HeapIdle = 499712
>>>>>>> # HeapInuse = 1302528
>>>>>>> # HeapReleased = 0
>>>>>>> # HeapObjects = 6377
>>>>>>> # Stack = 294912 / 294912
>>>>>>> # MSpan = 22560 / 32768
>>>>>>> # MCache = 2400 / 16384
>>>>>>> # BuckHashSys = 2727
>>>>>>> # NextGC = 4194304
>>>>>>> # PauseNs = [752083 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>> 0 0
>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>> 0 0
>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>> 0 0
>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>> 0 0
>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>> 0 0
>>>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>>> 0 0
>>>>>>> 0 0 0]
>>>>>>> # NumGC = 1
>>>>>>> # DebugGC = false
>>>>>>>
>>>>>>> 2)When I tried to open the file using the following command, it just
>>>>>>> goes into interactive mode and shows nothing
>>>>>>>
>>>>>>> a)Output from go version go1.7 linux/amd64 for mem.prof
>>>>>>>
>>>>>>> nsaboo@ubuntu:~/Desktop/memprof$ go tool pprof mem.prof
>>>>>>> Entering interactive mode (type "help" for commands)
>>>>>>> (pprof) top
>>>>>>> profile is empty
>>>>>>> (pprof)
>>>>>>>
>>>>>>> b)Output from go version go1.12.4 linux/amd64 for mem.prof
>>>>>>>
>>>>>>> nsaboo@ubuntu:~/Desktop/memprof$ go tool pprof mem.prof
>>>>>>> Type: space
>>>>>>> No samples were found with the default sample value type.
>>>>>>> Try "sample_index" command to analyze different sample values.
>>>>>>> Entering interactive mode (type "help" for commands, "o" for options)
>>>>>>> (pprof) o
>>>>>>> call_tree = false
>>>>>>> compact_labels = true
>>>>>>> cumulative = flat //: [cum | flat]
>>>>>>> divide_by = 1
>>>>>>> drop_negative = false
>>>>>>> edgefraction = 0.001
>>>>>>> focus = ""
>>>>>>> granularity = functions //: [addresses |
>>>>>>> filefunctions | files | functions | lines]
>>>>>>> hide = ""
>>>>>>> ignore = ""
>>>>>>> mean = false
>>>>>>> nodecount = -1 //: default
>>>>>>> nodefraction = 0.005
>>>>>>> noinlines = false
>>>>>>> normalize = false
>>>>>>> output = ""
>>>>>>> prune_from = ""
>>>>>>> relative_percentages = false
>>>>>>> sample_index = space //: [objects |
>>>>>>> space]
>>>>>>> show = ""
>>>>>>> show_from = ""
>>>>>>> tagfocus = ""
>>>>>>> taghide = ""
>>>>>>> tagignore = ""
>>>>>>> tagshow = ""
>>>>>>> trim = true
>>>>>>> trim_path = ""
>>>>>>> unit = minimum
>>>>>>> (pprof) space
>>>>>>> (pprof) sample_index
>>>>>>> (pprof) top
>>>>>>> Showing nodes accounting for 0, 0% of 0 total
>>>>>>> flat flat% sum% cum cum%
>>>>>>>
>>>>>>>
>>>>>>> 3)Please let me know if it is this the correct way of getting the
>>>>>>> memory profiling ?
>>>>>>>
>>>>>>> 4)Can we deduce something from this memory stats that points us to
>>>>>>> increase in memory usage?
>>>>>>>
>>>>>>> 5)I am just thinking out loud, since I am using go1.7, can that be
>>>>>>> the reason for the issue of increase in memory usage that might get
>>>>>>> fixed
>>>>>>> with latest go versions ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nitish
>>>>>>>
>>>>>>> On Tue, Mar 10, 2020 at 6:56 AM Jake Montgomery <jake...@gmail.com
>>>>>>> <javascript:>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On Monday, March 9, 2020 at 1:37:00 PM UTC-4, Nitish Saboo wrote:
>>>>>>>>>
>>>>>>>>> Hi Jake,
>>>>>>>>>
>>>>>>>>> The memory usage remains constant when the rest of the service is
>>>>>>>>> running.Only when LoadPatternDB() method is called within the
>>>>>>>>> service,
>>>>>>>>> Memory Consumption increases which actually should not happen.
>>>>>>>>> I am assuming if there is a memory leak while calling this method
>>>>>>>>> because the memory usage then becomes constant after getting
>>>>>>>>> increased and
>>>>>>>>> then further increases on next call.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Its possible that I am not fully understanding, perhaps a language
>>>>>>>> problem. But from what you have written above I still don't see that
>>>>>>>> this
>>>>>>>> means you definitely have a memory leak. To test for that you would
>>>>>>>> need to *continuously
>>>>>>>> *call LoadPatternDB() and monitor memory for a *considerable time*.
>>>>>>>> If it eventually stabilizes to a constant range then there is no leak,
>>>>>>>> just
>>>>>>>> normal Go-GC variation. If it never stops climbing, and eventually
>>>>>>>> consumes
>>>>>>>> all the memory, then it would probably be a leak. Just because it goes
>>>>>>>> up
>>>>>>>> after one call, or a few calls doe not mean there is a leak.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to golan...@googlegroups.com <javascript:>.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/f897fdb1-8968-4435-9fe9-02e167e09a36%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/f897fdb1-8968-4435-9fe9-02e167e09a36%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "golang-nuts" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to golan...@googlegroups.com <javascript:>.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq6DC98p4M4V2QCbQFTcsL1PtOWELvg8MEcMYj9EM9ui_A%40mail.gmail.com
>>>>>>>
>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq6DC98p4M4V2QCbQFTcsL1PtOWELvg8MEcMYj9EM9ui_A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *Michael T. jonesmichae...@gmail.com <javascript:>*
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> *Michael T. jonesmichae...@gmail.com <javascript:>*
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golan...@googlegroups.com <javascript:>.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7_d8s%3DmS5WVgV9K1m5VCBUoep2mitvX4o%3D%2BHVqf1APmQ%40mail.gmail.com
>>>
>>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq7_d8s%3DmS5WVgV9K1m5VCBUoep2mitvX4o%3D%2BHVqf1APmQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/6455e855-3f1f-4836-ab58-2256e97f7eef%40googlegroups.com.
