Re: [rsyslog] Can I use multiple stop in filters

David Lang Tue, 29 Dec 2015 00:08:14 -0800

you have rulesets for imudp, imtcp, and imfile, but not for pstats, internalrsyslog messages, and stuff written to /dev/log by applications

I think you need to define queues for the rulesets rather than for each action(just move all the queue.* things to the ruleset () section)

by the way, since you have the reset turned on on your impstats line, thenumbers are for the prior 10 seconds


David Lang


On Tue, 29 Dec 2015, Muhammad Asif wrote:

Date: Tue, 29 Dec 2015 12:56:30 +0500
From: Muhammad Asif <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Can I use multiple stop in filters

Here is my config file.  http://pastebin.com/k4EWRwL7

I am using ruleset tied to input but my main queue still receiving
messages. How can I trouble shoot this issue. Should I remove main queue
configs. Moreover impstats are being reset after each pool time.

Queues stats:    http://pastebin.com/asMECzaS

Thanks

On Tue, Dec 29, 2015 at 12:36 PM, David Lang <[email protected]> wrote:

you need to give the config when you ask questions like this. with
impstats the answer could be either way, by default the counters are not
reset, they are a running total since startup, but there is an option to
reset the counters each time they are reported.

 On Mon, 28 Dec 2015, Muhammad Asif wrote:

Date: Mon, 28 Dec 2015 18:45:09 +0500

From: Muhammad Asif <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Can I use multiple stop in filters

Dear Sir,

Please have a look on it http://pastebin.com/X2iNWmSh .
Please throw some light.
Mon Dec 28 18:05:58 2015: imtcp(514): origin=imtcp submitted=14101
Mon Dec 28 18:06:08 2015: imtcp(514): origin=imtcp submitted=34825
Mon Dec 28 18:06:19 2015: imtcp(514): origin=imtcp submitted=26688

1- Are these values accumulated or new in 10 seconds on tcp port.



yes (see above)

Mon Dec 28 18:05:58 2015: flows-queue queue: origin=core.queue
size=1000000
enqueued=18007 full=16 discarded.full=7
Mon Dec 28 18:06:08 2015: flows-queue queue: origin=core.queue
size=1000000
enqueued=14007 full=14 discarded.full=7
Mon Dec 28 18:06:19 2015: flows-queue queue: origin=core.queue
size=1000000
enqueued=10008 full=13 discarded.full=8

2- Are enqueued are new messages come into action queue from main queue.


yes

Does it also mean 18007+14007+10008=42022 messages dropped or how many

messages dropped due to discard.full=7 here?


it means 8 were dropped due ot the queue being full, the queue was full 13
times

since later values can be smaller than earlier ones, this looks like it is
resetting this counter each time it's being reported.

since size is always being reported at the same, very round, value It
looks like you have the queue full each time you are reporting.

3- Messages read from file are also first go to main queue and then come to

action queue or just come to action queue and then forward.


if you are not using rulesets, things go to the main queue. If you are
using rulesets and have a ruleset tied to an input and have a queue for
that ruleset, that queue is the 'main' queue for that input, the logs never
touch the MAIN queue.


Thanks




On Mon, Dec 28, 2015 at 5:41 PM, Rainer Gerhards <
[email protected]>
wrote:

Define the queue settings on the ruleset. That's faster and achieves the

same result for this configuration.

You can drop the stop statements. At end of ruleset processing always
stops.

Hth Rainer

Sent from phone, thus brief.
Am 28.12.2015 12:38 schrieb "Muhammad Asif" <[email protected]>:

Sorry I was wrong. ruleset is available in imfile. I am acheiving my goal

as shown below. Please give you valuable comments.


main_queue(
queue.dequeueBatchSize="4000"
queue.workerthreads="2"
queue.size="2000000"
)


module(load="imfile" PollingInterval="30" )
input(type="imfile" ruleset="flows"
File="/opt/parser/flows/aggregated_flows.csv"
Tag=""
)


ruleset(name="flows"){
    action(type="omfwd" target="127.0.0.1" port="5172" protocol="tcp"
name="flows-queue" template="msgonly" queue.size="1000000"
   # queue.filename="forwarding" queue.maxdiskspace="1g"
queue.highwatermark="900000" queue.lowwatermark= "500000"
    queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000"
queue.workerthreads="2" queue.type="LinkedList" )
    stop
}


input(type="imtcp" port="514" ruleset="events")

ruleset(name="events"){

 action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp"
name="events-queue" template="msgonly" queue.size="1000000"
   # queue.filename="forwarding" queue.maxdiskspace="1g"
queue.highwatermark="900000" queue.lowwatermark= "500000"
    queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000"
queue.workerthreads="2" queue.type="LinkedList" )

     stop
   }

Please answer some queries.
1- Flows taking from csv file also first go to main queue and then come

to

respective action queue?
2- Is there any better way?

Thanks


On Mon, Dec 28, 2015 at 2:09 PM, Muhammad Asif <[email protected]>
wrote:

Hi David,


As you know ruleset is not available in imfile module then what is the
best way to deal with logs processing from file and receiving on tcp

port

514 differently and avoid being written in any file even not syslog.


Thanks

On Mon, Dec 28, 2015 at 12:57 PM, David Lang <[email protected]> wrote:

yes, you can use stop as many times as you want.


David Lang

On Mon, 28 Dec 2015, Muhammad Asif wrote:

Date: Mon, 28 Dec 2015 11:19:49 +0500

From: Muhammad Asif <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Can I use multiple stop in filters


Hi geeks,

Can I use "stop" (To avoid writing in syslog file) in multiple

filters

like

this.

input(type="imptcp" port="514" ruleset="events");



ruleset(name="events"){
   action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp"
name="events-queue" )

stop
}



module(load="imfile" PollingInterval="30"  ruleset="flows")

input(type="imfile" File="/opt/parser/flows/aggregated_flows.csv"

Tag=""

)

ruleset(name="flows"){
   action(type="omfwd" target="127.0.0.1" port="5172" protocol="tcp"
name="flows-queue")

stop
}


Thanks
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

DON'T LIKE THAT.


_______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you

DON'T LIKE THAT.


_______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Can I use multiple stop in filters

Reply via email to