Hi folks, I am trying to figure out if someone has a decent solution for max memory usage/mem-pressure so far. I went through some of the github issues related to this (SetMaxHeap proposals and related discussions) but most of them are still under review:
- https://go-review.googlesource.com/c/go/+/46751 - https://github.com/golang/go/issues/16843 - https://github.com/golang/go/issues/23044 My use case is that I have an in-memory data store and producers can potentially drive it to OOM, causing the server to drop all the data. I'd like to avoid it as much as possible by potentially informing the producers to slow down/refuse new jobs for a bit till some data is drained if I has some sort of a warning system. I think rabbitmq does something similar https://www.rabbitmq.com/memory.html - How are others in the community handling these situations? It seems like most of database-ish implementations will just OOM and die. - Has anyone implemented a mem pressure warning mechanism? Does something like this make sense for detecting high mem usage? (Of course other programs can randomly ask for memory from the OS so this isn't going to be leak-proof) func memWatcher(highWatermark uint64, memCheckInterval time.Duration) { > s := &runtime.MemStats{} > // Count how many times we are consecutively beyond a mem watermark > highWatermarkCounter := -1 > forceGCInterval := 10 > for range time.NewTicker(memCheckInterval).C { > runtime.ReadMemStats(s) > if s.NextGC >= highWatermark { > log.Println("Approaching highWatermark") // Approaching highWatermark > continue > } > // Crossing highWatermark > if s.HeapAlloc >= highWatermark { > if highWatermarkCounter < 0 { > // Transitioning beyond highWatermark > log.Println("High mem usage detected! Crossed high watermark") > // Start counters > highWatermarkCounter = 0 > forceGCInterval = 10 > log.Println("Reject/Throttle new work till usage reduces") > continue > } else { > // Still above highWatermark > highWatermarkCounter++ > if highWatermarkCounter >= forceGCInterval { > log.Println("Forcing GC...") > runtime.GC() > forceGCInterval = forceGCInterval * 2 // Some kind of back-off > } > } > } else { > if highWatermarkCounter >= 0 { > // reset counters - back under the highWatermark > log.Warn().Msg("Mem usage is back under highWatermark") > highWatermarkCounter = -1 > forceGCInterval = 10 > } > } > } > } > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3408da0e-f420-4b3d-98f0-d0a06e95d398%40googlegroups.com.
