Guessing based on your latest description: are you aware that there is no partial slice collection in GC? That is:
> bigSlice := make([]int, 1000*1000) subSlice := bigSlice[0:1:1] bigSlice = nil runtime.GC() // At this point, bigSlice is still allocated! It cannot be freed by the GC > (in the current implementation) useTheSlice(subSlice) > You have to explicitly reallocate smaller slices (and copy over the data from the big slice into each smaller slice) in order for the GC to be able to collect each smaller slice individually. As suggested earlier in the thread, if you can create and share a small, self-contained program which exhibits the behavior you're speaking of, someone on this forum is much more likely to be able to help you. Trying to guess how a memory leak is manifesting is just... guesswork without looking at full code. There are lots of reasons why memory may not be collected by the GC, including random bugs in your program :) On Tuesday, April 28, 2020 at 2:00:55 PM UTC-6, Naveen Kak wrote: > > Basically using the Top command at end of test. > Let me give a quick summary of the scenario. > > Basically we have an application which keeps getting data on a TCP > connection, we allocate a global slice to temporarily hold the data and > then store in nested maps. > For every TCP data transaction, we store the data in a global slice > temporarily and then operate on the maps. > For a certain amount of data, i know what should be the total allocation, > but when i check using top command at end of test its almost always double > or slightly more. > > if I call runtime.GC() on every tcp transaction, memory looks good and end > of test it is as expected ( everything reclaimed) > Is it because the system is heavily loaded ( per sec we may be getting > huge no of transactions) so garbage collector cant keep track of > the objects to be freed. > As a test I even tried to block the delete operations on Map, just > addition to maps on a smaller scale though, even then memory is almost > double of what is expected. > only solution is calling runtime.GC() on every tcp transaction, then > everything is fine. > I am really not sure whether the map deletions don't work properly or is > it just the GC which is not able to work efficiently enough on its own. > > Calling GC so often is an overhead and would cause CPU spikes. > > > > > > > > On Tue, Apr 28, 2020 at 3:21 AM Ian Lance Taylor <ia...@golang.org > <javascript:>> wrote: > >> On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak <navee...@gmail.com >> <javascript:>> wrote: >> >>> I have my system up and running for let's say 10 hours or so where these >>> operations on map ( continuous add/delete) keep happening. Memory keeps >>> growing and goes very high. >>> Eventually at end of test we clear all map entries, memory is reclaimed >>> and is good. >>> It's basically kind of load test. >>> Appreciate your engagement. >>> >> >> Thanks for the information, but you didn't really answer my question. >> How exactly are you measuring whether the memory has been garbage collected? >> >> Ian >> >> >> >>> On Mon, 27 Apr, 2020, 3:41 AM Ian Lance Taylor, <ia...@golang.org >>> <javascript:>> wrote: >>> >>>> On Sun, Apr 26, 2020 at 2:48 PM <navee...@gmail.com <javascript:>> >>>> wrote: >>>> >>>>> https://play.golang.org/p/e22ufH-T2M1 >>>>> >>>>> This is my sample data structure. >>>>> >>>>> package main >>>>> >>>>> import ( >>>>> "fmt" >>>>> ) >>>>> >>>>> type MicroChkpt struct { >>>>> comprtype uint32 >>>>> MicroChkptInfoMap map[uint32][]byte >>>>> } >>>>> >>>>> type CallChkpt struct { >>>>> FullChkptData []byte >>>>> MicroChkptMap map[uint32]*MicroChkpt >>>>> ckey uint32 >>>>> comprtype uint32 >>>>> AuditInProgress bool >>>>> } >>>>> >>>>> var CallChkptMap map[uint32]*CallChkpt >>>>> >>>>> func main() { >>>>> fmt.Println("Hello, playground") >>>>> } >>>>> >>>>> So its a nested map structure, >>>>> CallChkptMap->MicroChkptMap->MicroChkptInfoMap >>>>> So i was expecting on deleting an entry from the top level map >>>>> CallChkptMap, whole underlying memory used by nested maps would be >>>>> reclaimed. >>>>> >>>>> Its not happening till all entries are removed from the top level map( >>>>> then only i see memory dipping), map has ongoing insert and delete >>>>> operations and grows pretty big. >>>>> Any workarounds to reclaim the memory on deleting the specific entry >>>>> please? >>>>> >>>>> >>>>> Should i go to the nested maps first, set them to nil and then delete >>>>> the entry from the top level map? >>>>> Appreciate all your time and inputs. >>>>> >>>> >>>> >>>> How exactly are you measuring whether the memory has been garbage >>>> collected? >>>> >>>> Ian >>>> >>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f4b8f164-085c-4d3e-8f51-c16138c3ac83%40googlegroups.com.
