You need to configure Kafka for how long it retains messages for replay - or some other option to store on disk.
> On Mar 10, 2022, at 10:07 AM, Rakesh K R <rakeshkr1...@gmail.com> wrote: > > Tamas, > Thanks you. So any suggestion on how to make application release this 900MiB > memory back to OS so that pod will not end up in OOMKilled state? > >> On Thursday, March 10, 2022 at 1:45:18 PM UTC+5:30 Tamás Gulácsi wrote: >> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes >> >> says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of >> memory is there... >> >> >> Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 UTC+1): >>> HI, >>> I have a micro service application deployed on kubernetes cluster(with 1gb >>> of pod memory limit). This app receive continuous messages(500 message per >>> second) from producer app over kafka interface(these messages are encoded >>> in protobuf format.) >>> >>> Basic application flow: >>> 1. Get the message one by one from kafka >>> 2. unmarshal proto message >>> 3. apply business logic >>> 4. write the message to redis cache(in []byte format) >>> >>> When pod starts memory will be around 50mb and memory starts increasing as >>> traffic flows into the application. It is never released back to OS. As a >>> result pod restarts with error code OOMKilled. >>> I have integrated grafana to see memory usage like RSS, heap, stack. >>> During this traffic flow, in-use heap size is 80mb, idle heap is 80mb where >>> as process resident memory is at 800-1000MB. Stopping the traffic >>> completely for hours did not help and RSS continue to remain in 1000mb. >>> Tried to analyze this with pprof and it reports only 80mb are in in-use >>> section. So I am wondering where these remaining 800-1000mb of pods memory >>> went. Also application allocates memory like slices/maps/strings to perform >>> business logic(see alloc_space pprof output below) >>> >>> I tried couple of experiments: >>> 1. Calling FreeOsMemory() in the app but that did not help >>> 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and did >>> not help >>> 3. Leaving the application for 5-6hrs without any traffic to see whether >>> memory comes down. It did not help >>> 4. pprof shows only 80mb of heap in use >>> 5. Tried upgrading golang version from 1.13 to 1.16 as there were some >>> improvements in runtime. It did not help >>> >>> pprof output for alloc_space: >>> >>> (pprof) top20 >>> Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total >>> Dropped 566 nodes (cum <= 2.63GB) >>> Showing top 20 nodes out of 114 >>> flat flat% sum% cum cum% >>> 78.89GB 14.99% 14.99% 78.89GB 14.99% >>> github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply >>> 67.01GB 12.73% 27.72% 285.33GB 54.21% >>> airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups >>> 58.75GB 11.16% 38.88% 58.75GB 11.16% >>> google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf >>> 52.26GB 9.93% 48.81% 52.26GB 9.93% reflect.unsafe_NewArray >>> 45.78GB 8.70% 57.50% 46.38GB 8.81% >>> encoding/json.(*decodeState).literalStore >>> 36.98GB 7.02% 64.53% 36.98GB 7.02% reflect.New >>> 28.20GB 5.36% 69.89% 28.20GB 5.36% >>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes >>> 25.60GB 4.86% 74.75% 63.62GB 12.09% >>> google.golang.org/protobuf/proto.MarshalOptions.marshal >>> 12.79GB 2.43% 77.18% 165.56GB 31.45% >>> encoding/json.(*decodeState).object >>> 12.73GB 2.42% 79.60% 12.73GB 2.42% reflect.mapassign >>> 11.05GB 2.10% 81.70% 63.31GB 12.03% reflect.MakeSlice >>> 10.06GB 1.91% 83.61% 12.36GB 2.35% >>> filterServersForDestinationDevicesAndSendToDistributionChan >>> 6.92GB 1.32% 84.92% 309.45GB 58.79% >>> groupAndSendToConfigPolicyChannel >>> 6.79GB 1.29% 86.21% 48.85GB 9.28% >>> publishInternalMsgToDistributionService >>> 6.79GB 1.29% 87.50% 174.81GB 33.21% encoding/json.Unmarshal >>> 6.14GB 1.17% 88.67% 6.14GB 1.17% >>> google.golang.org/protobuf/internal/impl.consumeBytes >>> 4.64GB 0.88% 89.55% 14.39GB 2.73% GetAllDevDataFromGlobalDevDataDb >>> 4.11GB 0.78% 90.33% 18.47GB 3.51% GetAllServersFromServerRecordDb >>> 3.27GB 0.62% 90.95% 3.27GB 0.62% net.HardwareAddr.String >>> 3.23GB 0.61% 91.57% 3.23GB 0.61% reflect.makemap >>> (pprof) >>> >>> >>> Need experts help in analyzing this issue. >>> >>> Thanks in advance!! > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/e9b91937-7bf5-4526-940f-2a60b2989ddfn%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6FA0FBD6-8E0E-4DE9-9560-91473D70DFBC%40ix.netcom.com.
