Re: [SR-Users] Logs suddenly inundated with qm_malloc errors
Maybe there is a memory leak or maybe the system is not provisioned with enough memory to cope with the traffic. Increase the amount of allocated memory and monitor it, see how it evolves over time. Regards, Ovidiu Sas On Tue, Dec 13, 2022 at 6:14 AM George Diamantopoulos wrote: > > Hello all, > > I'm testing out release 5.6.2 on one production kamailio instance, and today > our monitoring system triggered an alarm for lack of disk space on /. Indeed, > kamailio.log started growing at an alarming rate at some point. > > At about the same time disk space started being depleted, the following line > was frequently logged in the logs (no instances of it before that time): > > qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c: > __bencode_piece_new(79), module: rtpengine; Free fragment not found! > > They don't only concern rtpengine, core has produced similar "free fragment > not found" errors as well. There's additional multiple "had header field", > "bad header" and "invalid header name" errors appearing after that too, but > I'm guessing that's only a symptom of the qm_malloc issue? > > kamailio is started with -m 2048 -M 16 and there's another instance with > almost identical traffic and configuration, but running on 5.6.1, which has > never exhibited such failures in several months uptime. So I'm wondering, is > there anything in 5.6.2 that could explain this? Thanks! > > Best regards, > George > __ > Kamailio - Users Mailing List - Non Commercial Discussions > sr-users@lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- VoIP Embedded, Inc. http://www.voipembedded.com __ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] Logs suddenly inundated with qm_malloc errors
Hello, difficult to say, it could be also some random memory error or virtual/physical hardware issue. Are you seeing more error related to that, maybe it’s a memory leak? Make probably sense to observe it further. If it happens again, a diff between the rtpengine in 5.6.1 and 5.6.2 might get more insight. Regarding a memory leak, you can observe the memory pool by kamcmd or different tools by looking at statistics or memory status. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com<https://gilawa.com/> From: sr-users On Behalf Of George Diamantopoulos Sent: Tuesday, December 13, 2022 12:14 PM To: Kamailio (SER) - Users Mailing List Subject: [SR-Users] Logs suddenly inundated with qm_malloc errors Hello all, I'm testing out release 5.6.2 on one production kamailio instance, and today our monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log started growing at an alarming rate at some point. At about the same time disk space started being depleted, the following line was frequently logged in the logs (no instances of it before that time): qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not found! They don't only concern rtpengine, core has produced similar "free fragment not found" errors as well. There's additional multiple "had header field", "bad header" and "invalid header name" errors appearing after that too, but I'm guessing that's only a symptom of the qm_malloc issue? kamailio is started with -m 2048 -M 16 and there's another instance with almost identical traffic and configuration, but running on 5.6.1, which has never exhibited such failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that could explain this? Thanks! Best regards, George __ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
[SR-Users] Logs suddenly inundated with qm_malloc errors
Hello all, I'm testing out release 5.6.2 on one production kamailio instance, and today our monitoring system triggered an alarm for lack of disk space on /. Indeed, kamailio.log started growing at an alarming rate at some point. At about the same time disk space started being depleted, the following line was frequently logged in the logs (no instances of it before that time): qm_malloc(): qm_malloc(0x7f5ade9c0010, 536) called from rtpengine: bencode.c: __bencode_piece_new(79), module: rtpengine; Free fragment not found! They don't only concern rtpengine, core has produced similar "free fragment not found" errors as well. There's additional multiple "had header field", "bad header" and "invalid header name" errors appearing after that too, but I'm guessing that's only a symptom of the qm_malloc issue? kamailio is started with -m 2048 -M 16 and there's another instance with almost identical traffic and configuration, but running on 5.6.1, which has never exhibited such failures in several months uptime. So I'm wondering, is there anything in 5.6.2 that could explain this? Thanks! Best regards, George __ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
