Since I haven't gotten any response to this... can anyone at least give
me a yes or no answer:
Has anyone tried calling omruleset from within a ruleset bound to an
input module? Is this supported?
I'm getting all sorts of segfaults, malloc errors, etc. on Centos 5.5
x86_64, and before I try and debug this, I'd at least like to know
whether or not I'm trying something that either nobody else has ever
done, or isn't supposed to work.
Thanks,
Jason Antman
Rutgers University
Jason Antman wrote:
> Greetings,
>
> I'm trying to implement multiple rulesets with per-ruleset queues in
> order to handle messages going to MySQL. Rsyslog is doing template/regex
> based parsing for the MySQL inserts - it's not raw log messages, or the
> Adiscon format. I seem to be running into some errors.
>
> I tried creating a disk-assisted queue, stopping mysql for a minute or
> so, grabbing some relatively unique strings from the text files
> containing the raw log data, starting MySQL back up, and then searching
> for the strings - which I presumed would eventually be flushed from the
> queue to the database.
>
> Not only did I never see any log data collected when MySQL is down end
> up in the database, but after about 3 minutes with MySQL down, not only
> did I not get any disk queue files, but I saw slightly garbled versions
> of the log messages ending up in /var/log/messages - which isn't even
> mentioned anywhere in the rulesets!
>
> I'm binding imtcp and imudp to a ruleset ("remote") that first logs
> everything to dynfiles based on hostname, and then (for a few specific
> source IPs) passes messages on to omruleset for further processing.
>
> Aside from some repetitive rules for string matching, my ruleset looks like:
> $RuleSet DHCP-parsing
>
> # create a Disk-Assisted LinkedList main queue specific to this ruleset
> $WorkDirectory /var/rsyslog/work
> $RulesetCreateMainQueue on
> $MainMsgQueueFilename DHCPqueue
> $MainMsgQueueType LinkedList
> $MainMsgQueueMaxDiskSpace 500M
> $MainMsgQueueSaveOnShutdown on # persist queue contents to disk on shutdown
>
> # TESTING ONLY
> *.* /var/log/dhcp-parsing
> # END TESTING ONLY
>
> $ActionResumeRetryCount -1 # retry infinitely
>
> :msg, startswith, " DHCPREQUEST for"
> :ommysql:localhost,test,syslogger,syslogger;DHCPREQUESTMAC
> & :ommysql:localhost,test,syslog,foo;DHCPREQUESTIP
> & ~ ### DISCARD
>
> :msg, startswith, " DHCPACK on"
> :ommysql:localhost,test,syslogger,syslogger;DHCPACKonIP
> & :ommysql:localhost,test,syslog,foo;DHCPACKonMAC
> & ~ ### DISCARD
>
> :msg, startswith, " DHCPINFORM from"
> :ommysql:localhost,test,syslog,foo;DHCPINFORM
> & ~ ### DISCARD
>
> if $msg startswith ' DHCPACK to' and ( not ( $msg contains 'no client
> hardware address' ) ) \
> then :ommysql:localhost,test,syslog,foo;DHCPACKtoMAC
> & :ommysql:localhost,test,syslog,foo;DHCPACKtoIP
> & ~ ### DISCARD
>
>
> The garbage that I was seeing in /var/log/messages looked like (as you
> can see, right after the timestamp, it's obviously not properly
> formatted, though prior to killing mysql, it looked fine in the correct
> log files):
> Dec 22 15:20:41 : dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
> (Courtney-PC) via eth0
> Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 from
> 01:23:45:67:89:ab (Courtney-PC) via 123.456.78.123
> Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
> (Courtney-PC) via 123.456.78.123
> Dec 22 15:20:41 dhcpd: DHCPOFFER on 123.456.78.123 to
> 01:23:45:67:89:ab (LeVoyageur) via 123.456.78.123
> Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 (123.456.78.123)
> from 01:23:45:67:89:ab (LeVoyageur) via 123.456.78.123
> Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
> (LeVoyageur) via 123.456.78.123
> Dec 22 15:20:41 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via
> 123.456.78.123
> Dec 22 15:20:41 dhcpd: DHCPREQUEST for 123.456.78.123 from
> 01:23:45:67:89:ab (F52F2867C1364CC) via eth0
> Dec 22 15:20:41 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
> (F52F2867C1364CC) via eth0
> Dec 22 15:20:42 dhcpd: DHCPDISCOVER from 01:23:45:67:89:ab via
> 123.456.78.123
> Dec 22 15:20:42 l dhcpd: DHCPOFFER on 123.456.78.123 to
> 01:23:45:67:89:ab via 123.456.78.123
> Dec 22 15:20:42 dhcpd: DHCPREQUEST for 123.456.78.123 from
> 01:23:45:67:89:ab via eth0
> Dec 22 15:20:42 l dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
> via eth0
> Dec 22 15:20:42 dhcpd: DHCPREQUEST for 123.456.78.123 from
> 01:23:45:67:89:ab via 123.456.78.123
> Dec 22 15:20:42 ast dhcpd: DHCPACK on 123.456.78.123 to
> 01:23:45:67:89:ab via 172.31.16
> Dec 22 15:20:47 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
> via 123.456.78.123
> Dec 22 15:20:47 l dhcpd: DHCPREQUEST for 123.456.78.123 from
> 01:23:45:67:89:ab (Blue-PC) via eth0
> Dec 22 15:20:47 eER from 01:23:45:67:89:ab via 172.dhcpd: dhcpd: DHCPACK
> on 123.456.78.123 to 01:23:45:67:89:ab (Blue-PC) via eth0
> Dec 22 15:20:47 l.25.114 dhcpd: DHCPREQUEST for 123.456.78.123 from
> 01:23:45:67:89:ab (Blue-PC) via 123.456.78.123
> Dec 22 15:20:47 dhcpd: DHCPACK on 123.456.78.123 to 01:23:45:67:89:ab
> (Blue-PC) via 123.456.78.123
>
>
> Any ideas? I chose rsyslog in the hopes that I'd be able to replace our
> current cron-triggered log parsing scripts with rsyslog builtin
> templates and regex functions, but it seems like I'm running into more
> and more problems trying to develop complex rules that can also handle
> decent message volume and things like restarts of mysql.
>
> Thanks for any advice,
> Jason
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
