The memory usage of containers are ever increasing and has grown from 100 MB to ~ 1 GB in couple of hours. Not able to understand where the gap is, even though the the builder is being reset. Would keeping a sync.Pool of builders help here?
Regards On Sunday, August 30, 2020 at 8:55:15 PM UTC+5:30 Sarath Prabath Redlapalli Jaya wrote: > Hi, we've been observing high memory usage in the following code > https://github.com/Masterminds/squirrel/blob/master/placeholder.go#L113 > > We measured from pprof of heap > > (pprof) top20 > > Showing nodes accounting for 360.07MB, 91.31% of 394.32MB total > > Dropped 142 nodes (cum <= 1.97MB) > > Showing top 20 nodes out of 139 > > flat flat% sum% cum cum% > > 101.21MB 25.67% 25.67% 101.21MB 25.67% bytes.(*Buffer).String > > 48.01MB 12.18% 37.84% 48.01MB 12.18% > github.com/newrelic/go-agent/v3/newrelic.(*spanAttributeMap).add > > 42.61MB 10.81% 48.65% 43.11MB 10.93% > github.com/newrelic/go-agent/v3/newrelic.(*txnTrace).witnessNode > > 38.54MB 9.77% 58.42% 38.54MB 9.77% > go.uber.org/zap/buffer.NewPool.func1 > > 19MB 4.82% 63.24% 19MB 4.82% > github.com/newrelic/go-agent/v3/newrelic.segmentEnd.spanEvent > > 16.51MB 4.19% 67.43% 91.04MB 23.09% > github.com/newrelic/go-agent/v3/newrelic.endDatastoreSegment > > 14.01MB 3.55% 70.98% 18.01MB 4.57% > github.com/newrelic/go-agent/v3/newrelic.newTxn > > 13.51MB 3.43% 74.41% 93.13MB 23.62% > github.com/newrelic/go-agent/v3/newrelic.endExternalSegment > > 12.01MB 3.05% 77.45% 12.01MB 3.05% > github.com/newrelic/go-agent/v3/newrelic.getStackTrace > > 10.51MB 2.66% 80.12% 10.51MB 2.66% > github.com/newrelic/go-agent/v3/newrelic.(*slowQueries).insertAtIndex > > 8MB 2.03% 82.15% 8MB 2.03% > github.com/newrelic/go-agent/v3/newrelic.newSlowQueries > > 6.51MB 1.65% 85.73% 6.51MB 1.65% > github.com/DataDog/datadog-go/statsd.newStatsdBuffer > > 4MB 1.01% 86.75% 52.01MB 13.19% > github.com/newrelic/go-agent/v3/newrelic.(*spanAttributeMap).addString > > 3.50MB 0.89% 87.64% 3.50MB 0.89% > go.uber.org/zap.(*Logger).clone > > 3.50MB 0.89% 88.52% 3.50MB 0.89% > github.com/newrelic/go-agent/v3/newrelic.externalMetricKey.scopedMetric > > 3MB 0.76% 89.28% 3MB 0.76% > github.com/newrelic/go-agent/v3/newrelic.addUserAttribute > > 3MB 0.76% 90.05% 3MB 0.76% strings.(*Builder).WriteString > > 2.50MB 0.63% 90.68% 6.06MB 1.54% encoding/json.Unmarshal > > 2.50MB 0.63% 91.31% 2.50MB 0.63% context.(*cancelCtx).Done > > ===== > > Resetting the buffer by defer didn't help either. So moved to string > builder like this > > func replacePositionalPlaceholders(sql, prefix string) (string, error) { > buf := &strings.Builder{} > defer buf.Reset() > > i := 0 > for { > p := strings.Index(sql, "?") > if p == -1 { > break > } > > if len(sql[p:]) > 1 && sql[p:p+2] == "??" { // escape ?? => ? > buf.WriteString(sql[:p]) > buf.WriteString("?") > if len(sql[p:]) == 1 { > break > } > sql = sql[p+2:] > } else { > i++ > buf.WriteString(sql[:p]) > fmt.Fprintf(buf, "%s%d", prefix, i) > sql = sql[p+1:] > } > } > > buf.WriteString(sql) > return buf.String(), nil > } > > ====== > > This didn't help either, from the results below > > (pprof) top20 > > Showing nodes accounting for 220.09MB, 90.37% of 243.55MB total > > Dropped 126 nodes (cum <= 1.22MB) > > Showing top 20 nodes out of 135 > > flat flat% sum% cum cum% > > 33.97MB 13.95% 13.95% 33.97MB 13.95% strings.(*Builder).Write > > 29.07MB 11.94% 25.89% 29.57MB 12.14% > github.com/newrelic/go-agent/v3/newrelic.(*txnTrace).witnessNode > > 28.01MB 11.50% 37.39% 28.01MB 11.50% > github.com/newrelic/go-agent/v3/newrelic.(*spanAttributeMap).add > > 24.34MB 9.99% 47.38% 24.34MB 9.99% strings.(*Builder).WriteString > > 18.52MB 7.60% 54.98% 18.52MB 7.60% > go.uber.org/zap/buffer.NewPool.func1 > > 11.01MB 4.52% 59.50% 60.59MB 24.88% > github.com/newrelic/go-agent/v3/newrelic.endExternalSegment > > 10.01MB 4.11% 63.61% 13.51MB 5.55% > github.com/newrelic/go-agent/v3/newrelic.newTxn > > 10MB 4.11% 67.72% 10MB 4.11% > github.com/newrelic/go-agent/v3/newrelic.segmentEnd.spanEvent > > 7MB 2.88% 73.73% 7MB 2.88% > github.com/newrelic/go-agent/v3/newrelic.(*slowQueries).insertAtIndex > > 7MB 2.88% 76.60% 50.02MB 20.54% > github.com/newrelic/go-agent/v3/newrelic.endDatastoreSegment > > 6.51MB 2.67% 79.28% 6.51MB 2.67% > github.com/DataDog/datadog-go/statsd.newStatsdBuffer > > 6MB 2.46% 81.74% 6.50MB 2.67% > github.com/newrelic/go-agent/v3/newrelic.addUserAttribute > > 5.50MB 2.26% 84.00% 5.50MB 2.26% > github.com/newrelic/go-agent/v3/newrelic.getStackTrace > > 3MB 1.23% 85.23% 3MB 1.23% > github.com/newrelic/go-agent/v3/newrelic.newSlowQueries > > 3MB 1.23% 86.47% 3MB 1.23% > go.uber.org/zap.(*Logger).clone > > 3MB 1.23% 87.70% 3MB 1.23% > github.com/newrelic/go-agent/v3/newrelic.externalMetricKey.scopedMetric > > 2.50MB 1.03% 88.72% 30.51MB 12.53% > github.com/newrelic/go-agent/v3/newrelic.(*spanAttributeMap).addString > > 2MB 0.82% 90.37% 2MB 0.82% encoding/hex.EncodeToString > > (pprof) peek Write > > Showing nodes accounting for 243.55MB, 100% of 243.55MB total > > ----------------------------------------------------------+------------- > > flat flat% sum% cum cum% calls calls% + context > > ----------------------------------------------------------+------------- > > 33.97MB 100% | fmt.Fprintf > > 33.97MB 13.95% 13.95% 33.97MB 13.95% | > strings.(*Builder).Write > > ----------------------------------------------------------+------------- > > 22.34MB 91.78% | > github.com/XXXXX/squirrel.replacePositionalPlaceholders > > 2MB 8.22% | > net/url.(*URL).String > > 24.34MB 9.99% 23.94% 24.34MB 9.99% | > strings.(*Builder).WriteString > > ----------------------------------------------------------+------------- > > 1.16MB 100% | > encoding/json.compact > > 0 0% 23.94% 1.16MB 0.48% | > bytes.(*Buffer).Write > > 1.16MB 100% | > bytes.(*Buffer).grow > > ----------------------------------------------------------+------------- > > 6.51MB 100% | > github.com/DataDog/datadog-go/statsd.New > > 0 0% 23.94% 6.51MB 2.67% | > github.com/DataDog/datadog-go/statsd.newWithWriter > > 6.51MB 100% | > github.com/DataDog/datadog-go/statsd.newBufferPool > > ----------------------------------------------------------+------------- > > 12.01MB 100% | > go.uber.org/zap.(*SugaredLogger).log > > 0 0% 23.94% 12.01MB 4.93% | > go.uber.org/zap/zapcore.(*CheckedEntry).Write > > 12.01MB 100% | > go.uber.org/zap/zapcore.(*ioCore).Write > > ----------------------------------------------------------+------------- > > 12.01MB 100% | > go.uber.org/zap/zapcore.(*CheckedEntry).Write > > 0 0% 23.94% 12.01MB 4.93% | > go.uber.org/zap/zapcore.(*ioCore).Write > > 12.01MB 100% | > go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry > > ----------------------------------------------------------+------------- > > 1MB 100% | > golang.org/x/net/http2.(*Framer).readMetaFrame > > 0 0% 23.94% 1MB 0.41% | > golang.org/x/net/http2/hpack.(*Decoder).Write > > 1MB 100% | > golang.org/x/net/http2/hpack.(*Decoder).parseHeaderFieldRepr > > ----------------------------------------------------------+------------- > > (pprof) > > > Please help how to debug this more > > > Regards > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c28ac10f-ea81-411d-9236-b4ec476f8109n%40googlegroups.com.
