On Mon, 23 Jun 2025, Norberto Nuñez via rsyslog wrote:
Hi there.
I am configuring a rsyslog server as "syslog distributor" for two external
system, but it does not work as spected. By interface counters, the machine
receives about 220K UDP syslog packets per second on RX, but on TX counters
only some 300k UDP packets per second, when it supposed to be 440K packets per
second.
what is your network speed? 300k packets/sec is gig-E wire speed when you are
talking ~256 byte average message size.
at these rates, you may need to tweak the OS network stack buffer sizes
Environment:
* OS: Ubuntu 24.04 LTS
* Machine: LXC, X86_64
* Rsyslog: v8.2502.0
The machine has sufficient resources.
Here the config:
#
#
#
global(
internalmsg.ratelimit.interval="1"
internalmsg.ratelimit.burst="100000"
)
#
#
#
main_queue( queue.size="220000" queue.type="fixedArray" queue.workerThreads="32"
queue.dequeueBatchSize="65536" queue.minDequeueBatchSize="16384" queue.workerThreadMinimumMessages="16384")
large numbers of worker threads are almost always a bad idea, start low and
increase only if you find a thread maxing out the CPU
too many worker threads ends up with the threads fighting for locks and actually
slows things down.
#
#
#
module(load="impstats" interval="60" severity="7" resetcounters="on")
resetting counters is not perfect when you have multiple threads
#
# Template forward
#
template(name="Format_fw" type="string" string="<%PRI%>1 %TIMESTAMP:::date-rfc3339%
%.searchKey% %.appName% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")
#
module(load="omrelp")
#
ruleset(name="fw_dist"){
#
# Search key
#
set $.searchKey = $HOSTNAME;
#
# App name
set $.appName = $app-name;
#
# Forward to Rsyslog writer
#
action(type="omfwd" protocol="udp" Target="192.168.100.13" Port="1514" name="fw_udp_p" Template="Format_fw" RateLimit.Interval="0"
RateLimit.Burst="0" queue.size="220000" queue.type="fixedArray" queue.workerThreads="32" queue.dequeueBatchSize="65536" queue.minDequeueBatchSize="16384"
queue.workerThreadMinimumMessages="16384")
I would not set the min batchsize or thread min messages. Batch sizes have
diminishing returns as they get larger (say above 100 or so), I would not set
them super large unless testing showed they were needed.
again, lots of worker threads is a bad idea
#
# Forward to Grafana Alloy+Loki
#
action(type="omfwd" protocol="udp" Target="192.168.100.15" Port="1514" name="fw_udp_b" Template="Format_fw" RateLimit.Interval="0"
RateLimit.Burst="0" queue.size="220000" queue.type="fixedArray" queue.workerThreads="32" queue.dequeueBatchSize="65536" queue.minDequeueBatchSize="16384"
queue.workerThreadMinimumMessages="16384")
#
#
#
stop
}
#
# https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html
#
module(load="imudp" timeRequery="8" Threads="4" BatchSize="128")
probably do not needs multiple threads. imudp has the ability to read multiple
messages from the OS
input(type="imudp" port="514" ruleset="fw_dist" RateLimit.Interval="0"
RateLimit.Burst="0")
#
# https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html
#
module(load="imtcp")
input(type="imtcp" port="514" ruleset="fw_dist" RateLimit.Interval="0"
RateLimit.Burst="0")
#
#
#
module(load="imrelp")
input(type="imrelp" port="515" ruleset="fw_dist")
#
# https://www.rsyslog.com/how-to-use-impstats/
#
if $syslogtag contains 'rsyslogd-pstats' then {
action(name="FW_PSTATS" type="omfwd" Target="192.168.100.68" Port="1514" Protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format"
TCP_Framing="octet-counted" KeepAlive="on" KeepAlive.Interval="1" KeepAlive.Time="1" KeepAlive.Probes="10" queue.filename="rsyslog-psstats"
queue.type="LinkedList")
stop
}
I would make a ruleset for the pstats and configure the module to use that
ruleset (put a queue on the ruleset so it never touches the main queue)
you should not need the octet framing for pstats data, it's needed if you have
bad message content that contains nulls
$WorkDirectory /var/spool/rsyslog
¿Whats wrong?
please provide the pstats output
David Lang
Kind regards.
Norberto Núñez
________________________________
AVISO LEGAL: Esta información es reservada y privada y está dirigida únicamente
a su destinatario. Si usted no es el destinatario original de este mensaje y
por este medio pudo acceder a dicha información por favor elimine el mensaje.
La distribución o copia de este mensaje está estrictamente prohibida. Esta
comunicación es sólo para propósitos de información y no debe ser considerada
como propuesta, aceptación ni como una declaración de voluntad oficial de
NUCLEO S.A. La transmisión de e-mails no garantiza que el correo electrónico
sea seguro o libre de error. Por consiguiente, no manifestamos que esta
información sea completa o precisa. Toda información está sujeta a alterarse
sin previo aviso.
This information is private and confidential and intended for the recipient
only. If you are not the intended recipient of this message you are hereby
notified that any review, dissemination, distribution or copying of this
message is strictly prohibited. This communication is for information purposes
only and shall not be regarded neither as a proposal, acceptance nor as a
statement of will or official statement from NUCLEO S.A. . Email transmission
cannot be guaranteed to be secure or error-free. Therefore, we do not represent
that this information is complete or accurate and it should not be relied upon
as such. All information is subject to change without notice.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
