[rsyslog] Wildcard Support to log all the file logs of a directory
Hi Support, I want to monitor file logs at Loggly. I have many files in a directory and want to monitor them all with a wildcard. I have a configuration like below: $ModLoad imfile $InputFilePollInterval 10 $PrivDropToGroup adm $WorkDirectory /var/spool/rsyslog # Input for FILE1 $InputFileName /FILE1 $InputFileTag APPNAME1 $InputFileStateFile stat-APPNAME1 #this must be unique for each file being polled $InputFileSeverity info $InputFilePersistStateInterval 2 $InputRunFileMonitor # Add a tag for file events $template LogglyFormatFile,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag=\"file\"] %msg%\n" # Send to Loggly then discard if $programname == 'APPNAME1' then @@logs-01.loggly.com:514;LogglyFormatFile if $programname == 'APPNAME1' then ~ What I want is to use wildcard as $InputFileName /directory/*.txt or $InputFileName /directory/* Please consider my case on priority. Thanks -- View this message in context: http://rsyslog-users.1305293.n2.nabble.com/Wildcard-Support-to-log-all-the-file-logs-of-a-directory-tp7592048.html Sent from the rsyslog-users mailing list archive at Nabble.com. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Handling "log explosions"
On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote: While testing our current infrastructure we have suffered a /log explosion/, ie: errors when processing logs caused error logs on the machine that also caused errors when processed...and finally, disk became full and everything died. I'm wondering if worrying about this is useful, or how could it be managed/prevented (as automatically as possible). For example: * Rate-limiting for specific log events (eg: rate limiting events with syslogtag="foo" or matching a filter) * Having counters and ignore events of /type/ if more than N / last_X_minutes * Being able to reduce rsyslog verbosity, logging "fail and recover" messages, instead of logging an error on each failure. How do you handle those situations? Should we stop worrying about things that haven't happened and probably won't ever happen? monitor disk space and alert if it starts filling up. monitor logs/sec and alert if they jump much higher than normal ideally setup anomoly detection and alert when the rate of disk usage/logs per sec are unusually high OR unusually low (see https://www.usenix.org/legacy/publications/library/proceedings/lisa2000/full_papers/brutlag/brutlag_html/index.html for more info) overall, this isn't likely to happen once you get the system setup and running, so many places don't do anything special for this at all. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?
On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote: Even more: does it make sense to have queues when using omfile? usually not, it's usually less effort to write the data to the file than to move it to a new queue. You want a queue on an output (or on a ruleset with multiple outputs) if you think there can be problems with an output (either that it can't keep up with peak load, or that it may fail entirely for some time[1]) and you want to me sure that you can keep processing logs to other outputs imfile keeps track of where it is, so if it can't process messages, none are lost[2], so you may not need a queue if your only source is imfile. The other reason for having a queue involved is to allow batching of messages. There are a lot of cases where it is far more efficient to process multiple messages at a time instead of one at a time. For example, RELP can send multiple messages while waiting for acks for the first message sent and so will operate much faster if it has a queue of multiple messages available to send then if it must wait for the ack for each message before starting to send the next message. Another example, I've measured databases where inserting 1000 messages as a batch took the same time as inserting 2 messages independently. As a general rule, you will want to have a queue on anything that sends to a network/remote system, and may want to have a queue on sending to complex servers on local systems (i.e. databases) so that you can keep accepting new messages when the destinations are slow or unavailable. But to have a queue on omfile is seldom appropriate because if you do have a problem that prevents the log from being written, you are unlikely to be able to do anything else anyway. David Lang [1] network outages, remote systems being rebooted, etc [2] except in the case where you have file rotations going on and multiple rotations happen while you are stopped ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] collect and forward w/o change
On Fri, 23 Dec 2016, at 19:16, Woodruff, Dan wrote: > Then later on if you want to get real fancy, you can set up a queue on > the > ForwardToQRadar ruleset so when QRadar is down for patching logs will be > buffered and forwarded once QRadar is back up. I just got this worked a > few > weeks ago and can share the full ruleset config if you're interested. > > Hope that helps, > Dan An example of buffer/forward would be super interesting - please share Dan. A+ Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Handling "log explosions"
Hello While testing our current infrastructure we have suffered a /log explosion/, ie: errors when processing logs caused error logs on the machine that also caused errors when processed...and finally, disk became full and everything died. I'm wondering if worrying about this is useful, or how could it be managed/prevented (as automatically as possible). For example: * Rate-limiting for specific log events (eg: rate limiting events with syslogtag="foo" or matching a filter) * Having counters and ignore events of /type/ if more than N / last_X_minutes * Being able to reduce rsyslog verbosity, logging "fail and recover" messages, instead of logging an error on each failure. How do you handle those situations? Should we stop worrying about things that haven't happened and probably won't ever happen? Regards ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?
Even more: does it make sense to have queues when using omfile?
El 28/12/16 a las 15:52, mosto...@gmail.com escribió:
Hi
Does it make any sense to use queues when reading a file (imfile) and
forwarding to a central location using RELP (omrelp) ?
We would like to read & send events in a reliable way (reading
offset/position is only updated if successfully received on server).
but I don't understand why queues would make sense in such scenario.
Would the following configuration be enough then?
input(
file="*.log"
ruleset="files"
)
ruleset(name="files"
) {
action(
name="relp"
type="omrelp"
server="relay-cluster"
)
}
Regards (and happy holidays)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] Does it make sense to have a queue with imfile+omrelp?
Hi
Does it make any sense to use queues when reading a file (imfile) and
forwarding to a central location using RELP (omrelp) ?
We would like to read & send events in a reliable way (reading
offset/position is only updated if successfully received on server). but
I don't understand why queues would make sense in such scenario.
Would the following configuration be enough then?
input(
file="*.log"
ruleset="files"
)
ruleset(name="files"
) {
action(
name="relp"
type="omrelp"
server="relay-cluster"
)
}
Regards (and happy holidays)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
