Also, it may just be that the runtime is better off allocating more and not doing a GC based on available memory and CPU usage. The “max heap” feature in development may help here.
> On Apr 28, 2020, at 3:18 PM, 'Kevin Chowski' via golang-nuts > <golang-nuts@googlegroups.com> wrote: > > > Guessing based on your latest description: are you aware that there is no > partial slice collection in GC? That is: > > > >> bigSlice := make([]int, 1000*1000) >> subSlice := bigSlice[0:1:1] >> bigSlice = nil >> runtime.GC() >> // At this point, bigSlice is still allocated! It cannot be freed by the GC >> (in the current implementation) >> useTheSlice(subSlice) > > You have to explicitly reallocate smaller slices (and copy over the data from > the big slice into each smaller slice) in order for the GC to be able to > collect each smaller slice individually. > > As suggested earlier in the thread, if you can create and share a small, > self-contained program which exhibits the behavior you're speaking of, > someone on this forum is much more likely to be able to help you. Trying to > guess how a memory leak is manifesting is just... guesswork without looking > at full code. There are lots of reasons why memory may not be collected by > the GC, including random bugs in your program :) > > >> On Tuesday, April 28, 2020 at 2:00:55 PM UTC-6, Naveen Kak wrote: >> Basically using the Top command at end of test. >> Let me give a quick summary of the scenario. >> >> Basically we have an application which keeps getting data on a TCP >> connection, we allocate a global slice to temporarily hold the data and then >> store in nested maps. >> For every TCP data transaction, we store the data in a global slice >> temporarily and then operate on the maps. >> For a certain amount of data, i know what should be the total allocation, >> but when i check using top command at end of test its almost always double >> or slightly more. >> >> if I call runtime.GC() on every tcp transaction, memory looks good and end >> of test it is as expected ( everything reclaimed) >> Is it because the system is heavily loaded ( per sec we may be getting huge >> no of transactions) so garbage collector cant keep track of the objects to >> be freed. >> As a test I even tried to block the delete operations on Map, just addition >> to maps on a smaller scale though, even then memory is almost double of what >> is expected. >> only solution is calling runtime.GC() on every tcp transaction, then >> everything is fine. >> I am really not sure whether the map deletions don't work properly or is it >> just the GC which is not able to work efficiently enough on its own. >> >> Calling GC so often is an overhead and would cause CPU spikes. >> >> >> >> >> >> >> >>> On Tue, Apr 28, 2020 at 3:21 AM Ian Lance Taylor <ia...@golang.org> wrote: >>>> On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak <navee...@gmail.com> wrote: >>> >>>> I have my system up and running for let's say 10 hours or so where these >>>> operations on map ( continuous add/delete) keep happening. Memory keeps >>>> growing and goes very high. >>>> Eventually at end of test we clear all map entries, memory is reclaimed >>>> and is good. >>>> It's basically kind of load test. >>>> Appreciate your engagement. >>> >>> Thanks for the information, but you didn't really answer my question. How >>> exactly are you measuring whether the memory has been garbage collected? >>> >>> Ian >>> >>> >>>>> On Mon, 27 Apr, 2020, 3:41 AM Ian Lance Taylor, <ia...@golang.org> wrote: >>>>>> On Sun, Apr 26, 2020 at 2:48 PM <navee...@gmail.com> wrote: >>>>> >>>>>> https://play.golang.org/p/e22ufH-T2M1 >>>>>> >>>>>> This is my sample data structure. >>>>>> >>>>>> package main >>>>>> >>>>>> import ( >>>>>> "fmt" >>>>>> ) >>>>>> >>>>>> type MicroChkpt struct { >>>>>> comprtype uint32 >>>>>> MicroChkptInfoMap map[uint32][]byte >>>>>> } >>>>>> >>>>>> type CallChkpt struct { >>>>>> FullChkptData []byte >>>>>> MicroChkptMap map[uint32]*MicroChkpt >>>>>> ckey uint32 >>>>>> comprtype uint32 >>>>>> AuditInProgress bool >>>>>> } >>>>>> >>>>>> var CallChkptMap map[uint32]*CallChkpt >>>>>> >>>>>> func main() { >>>>>> fmt.Println("Hello, playground") >>>>>> } >>>>>> >>>>>> So its a nested map structure, >>>>>> CallChkptMap->MicroChkptMap->MicroChkptInfoMap >>>>>> So i was expecting on deleting an entry from the top level map >>>>>> CallChkptMap, whole underlying memory used by nested maps would be >>>>>> reclaimed. >>>>>> >>>>>> Its not happening till all entries are removed from the top level map( >>>>>> then only i see memory dipping), map has ongoing insert and delete >>>>>> operations and grows pretty big. >>>>>> Any workarounds to reclaim the memory on deleting the specific entry >>>>>> please? >>>>>> >>>>>> >>>>>> >>>>>> Should i go to the nested maps first, set them to nil and then delete >>>>>> the entry from the top level map? >>>>>> Appreciate all your time and inputs. >>>>>> >>>>> >>>>> >>>>> How exactly are you measuring whether the memory has been garbage >>>>> collected? >>>>> >>>>> Ian > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/f4b8f164-085c-4d3e-8f51-c16138c3ac83%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7E778BC9-5358-464A-8490-D61412A30982%40ix.netcom.com.
