So for the record, here is my awful hack.
I'm parsing messages like "[21/05/2024 23:56:37] [pcc->cct] 00100t7cc"
from existing files using imfile and want to re-use the timestamp from
msg payload in Rsyslog. Timestamp formatting is dd/mm/YYYY HH:MM:ss in
local server timezone
So first, I used mmnormalize with the following rule to extract
individual timestamp fields:
rule=:[%day:number%/%month:number%/%year:number%
%hour:number%:%minute:number%:%second:number%]
[%dsd-src:string-to:->%->%dsd-dst:char-to:]%] %dsd-message:rest%
Then I used a custom static mapping table to convert month as digits
into RFC 3164 "3 letters" month name:
{
"version": 1,
"nomatch": "",
"type": "array",
"table": [
{ "index": 1, "value": "Jan" },
{ "index": 2, "value": "Feb" },
{ "index": 3, "value": "Mar" },
{ "index": 4, "value": "Apr" },
{ "index": 5, "value": "May" },
{ "index": 6, "value": "Jun" },
{ "index": 7, "value": "Jul" },
{ "index": 8, "value": "Aug" },
{ "index": 9, "value": "Sep" },
{ "index": 10, "value": "Oct" },
{ "index": 11, "value": "Nov" },
{ "index": 12, "value": "Dec" },
]
}
Finally, I used a ruleset to apply this liblognorm pattern and use so
internal Rsyslog scripting to build a RFC 3164 timestamp:
lookup_table(
name="month-num-to-three-letters-lookup-table"
file="/etc/rsyslog.d/month-num-to-three-letters-lookup-table.json"
reloadonhup="off"
)
ruleset(
name="extractData"
)
{
set
$.pcc=re_extract($!metadata!filename,"/REP_RECORD_([a-z0-9]{2})/",0,1,"UNMATCHED_PCC");
action(
type="mmnormalize"
path="$!extracted"
rulebase="/etc/rsyslog.d/dsd-trace-parser.rules"
useRawMsg="off"
)
set $!extracted!monthInt = cnum($!extracted!month);
set $!extracted!dayInt = cnum($!extracted!day);
if ($!extracted!dayInt < 10) then {
set $!extracted!dayLeadingSpace = " " & $!extracted!dayInt;
} else {
set $!extracted!dayLeadingSpace = "" & $!extracted!dayInt;
}
set $!extracted!monthThreeLetters =
lookup("month-num-to-three-letters-lookup-table", $!extracted!monthInt);
set $!extracted!timestamp!rfc3164 = $!extracted!!monthThreeLetters
& " " & $!extracted!dayLeadingSpace & " " & $!extracted!hour & ":" &
$!extracted!minute & ":" & $!extracted!second;
action(
type="omfile"
dirCreateMode="0755"
FileCreateMode="0644"
File="/var/log/rep-record.log"
template="repRecordRfc3164FileOutput"
# template="RSYSLOG_DebugFormat"
)
}
So... Yeah it works, but this is ugly and I really thought I'd come up
with a cleaner solution...
I have a couple of questions, maybe someone can points me to improvement
or confirm I did the best that was possible:
1. I did not find any way to parse my timestamp into date-time object
form liblognorm rule directly, did I missed something ?
2. Is there any better way to convert 1 / "01" -> Jan or 12 / "12" ->
Dec than using the static lookup table hack ?
3. Is there any way to localize a my timestamp with server local
timezone ? If so I could use RFC 5424 date-time instead of RFC 3164
4. I tried to use parse_time with my RFC 3164 date-time string but it
returned 0 or and empty string, I think such function does not exist on
CentOS 7 / Rsyslog 8.24, confirmed ?
5. I there any other way to convert this string timestamp into a
date-time object ?
6. I considered re-injecting my generated full RFC 3164 syslog message
into rsyslog itself, so it can then be handled by normal rules, is that
stupid ? Is there any output module to re-feed the message into internal
rsyslog processing queue ?
7. At last, but not least, is there any "bounty" system or something to
contribute to date-time parsing improvments for Rsyslog ?
Best regards, Adam.
On 5/22/24 08:03, Adam Cecile via rsyslog wrote:
Hello,
Thanks for your answer but David figured out: my CentOS 7 rsyslog package does
not accept inline rule definition but wants a rulebase file.
It works fine like this, however, recreating timestamp from the msg payload
itself has been a nightmare but I got something (quite ugly) that works.
I'll update this thread with my solution, hopefully it'll help someone else and
maybe I'll get some comments.
Adam.
On May 22, 2024 7:58:16 AM GMT+02:00, Mariusz Kruk via
rsyslog<rsyslog@lists.adiscon.com> wrote:
Check your spaces in msg and rawmsg and compare it with your pattern because
they are inconsistent.
$msg contains [something]-two spaces-[something]-two spaces-something
$rawmsg contains [something]-two spaces-[something]-one space-something
Your pattern contains [something]-one space-[something]-one space-something
On 22.05.2024 00:24, Adam Cecile via rsyslog wrote:
No ;-)
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'gz-tuma', PRI: 133,
syslogtag 'trace.log', programname: 'trace.log', APP-NAME: 'trace.log', PROCID:
'-', MSGID: '-',
TIMESTAMP: 'May 21 23:56:46', STRUCTURED-DATA: '-',
msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
escaped msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
inputname: imfile rawmsg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
$!:{ "metadata": { "filename": "<redacted>", "fileoffset": "3396674" } }
$.:{ "pcc": "t7" }
$/:
On 5/22/24 00:20, David Lang wrote:
if you look at the msg field in the RSYSLOG_DebugFormat output, you will see
that it does have a leading space. your pattern doesn't
David Lang
On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
Date: Tue, 21 May 2024 23:58:23 +0200
From: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
To: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
Cc: Adam Cecile<acec...@letz-it.lu>
Subject: Re: [rsyslog] Unable to re-use variable generated by mmnormalize
Just tried, no leading space, message is correct however the log does not
contains any trace indicating mmnormalize has tried to do something.
I guess it is just NOT active for some reason...
On 5/21/24 23:50, David Lang wrote:
log the message with RSYSLOG_DebugFormat so that you can see the variables that
exist.
my guess is that your rule needs a leading space, because the msg field you are
parsing starts with a space (a very common problem when you are starting to use
mmnormalize)
David Lang
On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
Date: Tue, 21 May 2024 23:37:14 +0200
From: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
To:rsyslog@lists.adiscon.com
Cc: Adam Cecile<acec...@letz-it.lu>
Subject: [rsyslog] Unable to re-use variable generated by mmnormalize
Hello,
I'm struggling to understand how to use mmnormalize module. My configuration
snippet is the following:
template(name="recordRfc3164FileOutput" type="list") {
property(name="timestamp" dateFormat="rfc3164")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="$.pcc")
constant(value="@")
property(name="syslogtag")
constant(value=":")
constant(value=" ")
property(name="msg")
constant(value="\n")
}
ruleset(
name="extractPccNameFromRecordLog"
)
{
set
$.pcc=re_extract($!metadata!filename,"/record_([a-z0-9]{2})/",0,1,"UNMATCHED_PCC");
action(
type="mmnormalize"
rule=["rule=:[%day:number%/%month:number%/%year:number%
%hour:number%:%minutes:number%:%seconds:number%]
[%dsd-src:string-to:->%->%dsd-dst:char-to:]%] %dsd-message:rest%"]
useRawMsg="off"
)
action(
type="omfile"
dirCreateMode="0755"
FileCreateMode="0644"
File="/var/log/record.log"
template="recordRfc3164FileOutput"
)
}
input(
type="imfile"
file="/path/to/record_*/*.log"
tag="trace.log"
addmetadata="on"
ruleset="extractPccNameFromRecordLog"
)
Extraction of pattern from imfile filepath is working fine, mmnormalize rule is working
fine too, according to my test using "lognormalizer" command line tool to test
it.
So I guessed, I could be able to replace property(name="msg") with
property(name="$.dsd-message") or property(name="$!dsd-message") but all I get is an
empty string.
Is there something I missed ?
Thanks in advance,
Regards, Adam.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
