Good day I am trying to diagnose and resolve an issue whereby the memory consumed by the rsyslog daemon increases linearly over time. This continues until it consumes most of the memory (including swap) of the system and the service has to be restarted to free up memory. There are two servers with identical configurations. What I noticed is that the server receiving a higher volume of messages also consumes memory at a higher rate. In other word it appears as if the message rate, or message volume, is directly proportional to the rate at which memory is consumed.
Below is the version information for the rsyslogd daemon: rsyslogd 8.2310.0 (aka 2023.10) compiled with: PLATFORM: x86_64-pc-linux-gnu PLATFORM (lsb_release -d): FEATURE_REGEXP: Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: Yes memory allocator: system default Runtime Instrumentation (slow code): No uuid support: Yes systemd support: Yes Config file: /etc/rsyslog.conf PID file: /var/run/rsyslogd.pid Number of Bits in RainerScript integers: 64 It is running on Debian 12 servers. To provide you with more background detail, initially I configured three listeners: one UDP (port 514), one TCP (port 514) and one TLS (port 6514). A single system was configured to push logs to the TLS port and that worked well (no increase in memory usage over time). Recently I added another UDP listener (port 10514) and started configured a number of systems to push their logs to this port, but since then I've observed the described gradual memory increase. This new listener is configured as follows: A ruleset was created and bound to this listener (the ruleset doesn't have its own queue). The ruleset first runs the mmutf8fix action then calls a different ruleset (named "normalise"), which normalises the data (just sets specific variables that is later used in a template to construct a JSON object). After the call to the "normalise" ruleset returns, a mmnormalize action is performed and some additional variables are set. Lastly the ruleset (the one bound to the listener) then calls yet another ruleset (named "kafka_output"), which is used to construct a JSON object from the various variables and uses the omkafka action to push this to a Kafka cluster. The flow of the above can be visualised as: Source -> Syslog Server [10514/UDP] -> [listener ruleset] -> [normalise ruleset] -> [kafka_output ruleset] It should also be noted the original listeners are configured in much the same way, apart from having calls to even more rulesets. I haven't tested if the UDP listener on port 514 exhibits the same behaviour (it isn't currently being used). This rsyslog daemon is also used to capture locally generated logs and the statistics (impstats) module is also loaded. What can I do to troubleshoot what's causing this "memory leak"? Kind Regards --- _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.