the format is
filter action
or
if filter then action
every section where the filter matches will result in an output, so you either
issue a 'stop' (don't look at anything else after this)
if filter then {
action
stop
}
or you setup if then else logic
to help parsing the logs, you need to look at the format the log arrives in.
write some logs using the templare RSUSLOG_DebugFormat and you will see all the
values that are alreasy parsed that you can use.
there are may ways to match parts of a message, or you can use mmnormalize to
extract lots of stuff efficiently.
sorry I don't have time to go into more detail at the moment, but this should
help you find what to read up on.
David Lang
On Tue, 3 Jun 2025, Brendan Kearney via rsyslog wrote:
Date: Tue, 3 Jun 2025 14:18:09 -0400
From: Brendan Kearney via rsyslog <[email protected]>
To: [email protected]
Cc: Brendan Kearney <[email protected]>
Subject: [rsyslog] Output messages to different DB based on SysLogTag
List members,
I am using rsyslog and RELP to collect all logs from several sources,
with various services/processes, and inserting the messages into a
database with ommysql. I would like to begin taking a single service's
logs out of the "main" feed and insert the those messages into a
different DB. What I am wondering is how to parse the message and
create a logic structure to insert this one service's logs into the
separate DB, while leaving all other logs to be processed as is
currently being done. for example, the current setup:
#### GLOBAL DIRECTIVES ####
global(workDirectory="/var/lib/rsyslog")
$template bpk2_SqlWithSeparatePID,"insert into SystemEvents
(Message, Facility, FromHost, Priority, DeviceReportedTime,
ReceivedAt, InfoUnitID, SysLogTag, processid) values ('%msg%',
%syslogfacility%, '%HOSTNAME%', %syslogpriority%,
'%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
%iut%, '%programname%', '%PROCID%')",SQL
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
include(file="/etc/rsyslog.d/*.conf" mode="optional")
#### MODULES ####
module(load="imuxsock" # provides support for local system
logging (e.g. via logger command)
SysSock.Use="off") # Turn off message reception via local
log socket;
# local messages are retrieved through imjournal now.
module(load="imjournal" # provides access to the systemd journal
StateFile="imjournal.state") # File to store the position in
the journal
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
module(load="imrelp") # needs to be done just once
input(type="imrelp" port="20514" keepAlive="on")
module(load="ommysql") # needs to be done just once
#### RULES ####
action(type="ommysql" server="database.bpk2.com" serverport="3306"
db="Syslog" uid="<redacted>" pwd="<redacted>"
template="bpk2_SqlWithSeparatePID"
queue.filename="SyslogQueue" action.resumeRetryCount="-1")
Using a second template for the Squid proxy service, I would need to
define a template for the DB, and I am not sure how to extract the
values out of the message, in order to complete the "insert" statement.
So far, I have:
$template bpk2_Squid,"insert into access_log (time_since_epoch,
date_day, date_time, event_date, response_time, client_src_ip_addr,
client_src_hostname, user_agent, client_proto_ver,
squid_request_status, http_status_code, reply_size, request_method,
request_url, username, proxy_ip, proxy_port, squid_hier_status,
server_ip_addr, server_hostname, mime_type) values (...)",SQL
I have the DB already setup, and it reflects the custom log fields that
I am using in Squid. I am wondering how it extract the different fields
out of the %msg itself, for the insert template to be complete. Then,
once the extract of fields is complete, I presume an "if... then...
else..." kind of structure would be needed to send the Squid logs to one
DB and all the rest of the messages to the existing Syslog DB. It would
be a "fall-through" logic where first match of the service, or
SysLogTag, wins with the Syslog DB insert being a catchall kind of rule.
Does the "action" directive need to have the logic within it, or do I
need to setup a logic tree, with each case having its own "action"
stanza? In essence, I see the flow being:
Take in Message
Determine SysLogTag
If SysLogTag == "(squid-1)" then insert Message into Squid DB, using
bpk2_Squid template, and quit
If SysLogTag == "*" then insert Message into Syslog DB, using
bpk2_SqlWithSeparatePID template
Can you provide insight into how these efforts can be best achieved?
Any insight would be greatly appreciated.
Thank you,
Brendan Kearney
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
