Basically using the Top command at end of test. Let me give a quick summary of the scenario.
Basically we have an application which keeps getting data on a TCP connection, we allocate a global slice to temporarily hold the data and then store in nested maps. For every TCP data transaction, we store the data in a global slice temporarily and then operate on the maps. For a certain amount of data, i know what should be the total allocation, but when i check using top command at end of test its almost always double or slightly more. if I call runtime.GC() on every tcp transaction, memory looks good and end of test it is as expected ( everything reclaimed) Is it because the system is heavily loaded ( per sec we may be getting huge no of transactions) so garbage collector cant keep track of the objects to be freed. As a test I even tried to block the delete operations on Map, just addition to maps on a smaller scale though, even then memory is almost double of what is expected. only solution is calling runtime.GC() on every tcp transaction, then everything is fine. I am really not sure whether the map deletions don't work properly or is it just the GC which is not able to work efficiently enough on its own. Calling GC so often is an overhead and would cause CPU spikes. On Tue, Apr 28, 2020 at 3:21 AM Ian Lance Taylor <i...@golang.org> wrote: > On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak <naveen.ka...@gmail.com> wrote: > >> I have my system up and running for let's say 10 hours or so where these >> operations on map ( continuous add/delete) keep happening. Memory keeps >> growing and goes very high. >> Eventually at end of test we clear all map entries, memory is reclaimed >> and is good. >> It's basically kind of load test. >> Appreciate your engagement. >> > > Thanks for the information, but you didn't really answer my question. How > exactly are you measuring whether the memory has been garbage collected? > > Ian > > > >> On Mon, 27 Apr, 2020, 3:41 AM Ian Lance Taylor, <i...@golang.org> wrote: >> >>> On Sun, Apr 26, 2020 at 2:48 PM <naveen.ka...@gmail.com> wrote: >>> >>>> https://play.golang.org/p/e22ufH-T2M1 >>>> >>>> This is my sample data structure. >>>> >>>> package main >>>> >>>> import ( >>>> "fmt" >>>> ) >>>> >>>> type MicroChkpt struct { >>>> comprtype uint32 >>>> MicroChkptInfoMap map[uint32][]byte >>>> } >>>> >>>> type CallChkpt struct { >>>> FullChkptData []byte >>>> MicroChkptMap map[uint32]*MicroChkpt >>>> ckey uint32 >>>> comprtype uint32 >>>> AuditInProgress bool >>>> } >>>> >>>> var CallChkptMap map[uint32]*CallChkpt >>>> >>>> func main() { >>>> fmt.Println("Hello, playground") >>>> } >>>> >>>> So its a nested map structure, >>>> CallChkptMap->MicroChkptMap->MicroChkptInfoMap >>>> So i was expecting on deleting an entry from the top level map >>>> CallChkptMap, whole underlying memory used by nested maps would be >>>> reclaimed. >>>> >>>> Its not happening till all entries are removed from the top level map( >>>> then only i see memory dipping), map has ongoing insert and delete >>>> operations and grows pretty big. >>>> Any workarounds to reclaim the memory on deleting the specific entry >>>> please? >>>> >>>> >>>> Should i go to the nested maps first, set them to nil and then delete >>>> the entry from the top level map? >>>> Appreciate all your time and inputs. >>>> >>> >>> >>> How exactly are you measuring whether the memory has been garbage >>> collected? >>> >>> Ian >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAHB4VaAHgKv_FyB7FnHGkLtgpc9K_E9cte6gYBDLThfpVCy8jQ%40mail.gmail.com.
