List members,
I am trying to get my squid logs parsed by liblognorm and inserted into
a database, but I am running into trouble with the rulebase and not able
to parse logs. I have tried a myriad of different ways, but I'm not
making headway. Can someone point out my mistake, and why I can't parse
my logs properly?
Log entries (custom log format in squid):
192.168.1.78,desktop.bpk2.com,-,[02/Jan/2024:11:59:29
-0500],192.168.88.2,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,"Mozilla/5.0
(X11; Linux x86_64; rv:100.0) Gecko/20100101
Firefox/100.0",200,1247,60302,TCP_TUNNEL/FIRSTUP_PARENT,-
192.168.24.83,192.168.24.83,-,[02/Jan/2024:11:59:28
-0500],192.168.88.8,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,"Mozilla/5.0
(X11; Linux x86_64; rv:120.0) Gecko/20100101
Firefox/120.0",200,1247,60205,TCP_TUNNEL/FIRSTUP_PARENT,-
192.168.24.83,192.168.24.83,-,[02/Jan/2024:12:00:03
-0500],192.168.88.1,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,"Mozilla/5.0
(X11; Linux x86_64; rv:120.0) Gecko/20100101
Firefox/120.0",200,946,170381,TCP_TUNNEL/FIRSTUP_PARENT,-
rulebase:
version=2
rule=:%clientip:ipv4%,%hostname:word%,%username:word%,[%timestamp:char-to:]%],%proxyip:ipv4%,%proxyport:number%,%originserverip:ipv4%,%originservertype:word%,%method:word%,%protocolversion:string%,%url:string%,"%useragent:char-to:"%",%responsecode:number%,%totalsize:number%,%totaltime:duration%,%cachestatus:string%,%mimetype:string%
or in "exploded" form:
version=2
rule=:%
clientip:ipv4
%,%
hostname:word
%,%
username:word
%,[%
timestamp:char-to:]
%],%
proxyip:ipv4
%,%
proxyport:number
%,%
originserverip:ipv4
%,%
originservertype:word
%,%
method:word
%,%
protocolversion:string
%,%
url:string
%,"%
useragent:char-to:"
%",%
responsecode:number
%,%
totalsize:number
%,%
totaltime:duration
%,%
cachestatus:string
%,%
mimetype:string%
output of cat "squid.logs |lognormalizer -r squid.rb -U |jq .":
{
"originalmsg": "192.168.1.78,desktop.bpk2.com,-,[02/Jan/2024:11:59:29
-0500],192.168.88.2,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,\"Mozilla/5.0
(X11; Linux x86_64; rv:100.0) Gecko/20100101
Firefox/100.0\",200,1247,60302,TCP_TUNNEL/FIRSTUP_PARENT,-",
"unparsed-data": "
-0500],192.168.88.2,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,\"Mozilla/5.0
(X11; Linux x86_64; rv:100.0) Gecko/20100101
Firefox/100.0\",200,1247,60302,TCP_TUNNEL/FIRSTUP_PARENT,-"
}
{
"originalmsg": "192.168.24.83,192.168.24.83,-,[02/Jan/2024:11:59:28
-0500],192.168.88.8,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,\"Mozilla/5.0
(X11; Linux x86_64; rv:120.0) Gecko/20100101
Firefox/120.0\",200,1247,60205,TCP_TUNNEL/FIRSTUP_PARENT,-",
"unparsed-data": "
-0500],192.168.88.8,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,\"Mozilla/5.0
(X11; Linux x86_64; rv:120.0) Gecko/20100101
Firefox/120.0\",200,1247,60205,TCP_TUNNEL/FIRSTUP_PARENT,-"
}
{
"originalmsg": "192.168.24.83,192.168.24.83,-,[02/Jan/2024:12:00:03
-0500],192.168.88.1,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,\"Mozilla/5.0
(X11; Linux x86_64; rv:120.0) Gecko/20100101
Firefox/120.0\",200,946,170381,TCP_TUNNEL/FIRSTUP_PARENT,-",
"unparsed-data": "
-0500],192.168.88.1,3128,127.0.0.1,-,CONNECT,HTTP/1.1,push.services.mozilla.com:443,\"Mozilla/5.0
(X11; Linux x86_64; rv:120.0) Gecko/20100101
Firefox/120.0\",200,946,170381,TCP_TUNNEL/FIRSTUP_PARENT,-"
}
the parsing is hanging up on the "<space>-0500]" in the timestamp, and
does not seem to be honoring the "char-to:]" indicating the field ends
with the <close bracket> character. is there something I am doing
wrong? Any help would be appreciated. I am using
liblognorm/lognormalizer version 2.0.6, if that matters.
Thank you,
Brendan Kearney
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
