On Mon, 21 Dec 2015, Muhammad Asif wrote:
Dear David,
You always have been very helpful for us. I have solved many problem of our
rsyslog. Could you please give me some suggestions.
If I received messages at about 20000EPS, what main queue and action queue
size dequeuebatch size should I use.
Is there anyway to get stats of messages received on TCP port and how can
we clearly check how much RAM is being used by queues.
configure the impstats module, it gives you all these details
Logs are being written in syslog file with 20000 EPS but to control
processing at fluentd, i am using dequeuebatch size 2000 EPS with delay of
1 sec. I think there is some wrong.
This will cauuse rsyslog to send up to 20000 messages in one chunk, then sleep
for a second. you probably want to do something more like batch size of 2000 and
sleep for 1/10 of a second (or 200 and 1/100 of a second) so that the logs are
far more spread out
what are the limits in fluentd and what happens when you exceed them?
David Lang
main_queue(
#queue.filename="main_queue" # write to disk if needed
#queue.maxdiskspace="1g" # when to stop writing to disk
#queue.highwatermark="400000" # start spilling to disk at this size
#queue.lowwatermark="100000" # stop spilling when it gets back to this size
#queue.saveonshutdown="on" # write queue contents to disk on shutdown
queue.dequeueBatchSize="2000"
queue.workerthreads="2"
queue.size="500000" # absolute max queue size
)
if ($fromhost-ip != "127.0.0.1" and $fromhost-ip != "10.16.28.160" and
$fromhost-ip != "192.168.10.34" and $fromhost-ip != "172.18.40.134" and
$fromhost-ip != "192.168.122.1") then {
action(type="omfwd" target="127.0.0.1" port="5170" protocol="tcp"
template="msgonly" queue.size="100000"
queue.dequeuebatchsize="2000" queue.dequeueslowdown="1000000"
queue.type="LinkedList" )
stop ## To Avoid writing in syslog
}
I only have 6 GB of RAM.
Thanks.
On Fri, Dec 18, 2015 at 10:32 AM, David Lang <[email protected]> wrote:
1- There are about 13000 files of 1.1 MB accumulated to 13 GB on disk. When
I remove low and high water marks on main and action queue to make it only
In-memmory queues, it still create these files. Why is that?
what makes the disk queues isn't the watermark settings, but the fact that
you have a queue.filename
2- How about if rsyslog receive at 30000EPS but main queue is about 500k
and action queue 100k.
what's your question? a 500K main queue and 100K action queue may be just
fine at 30K logs/sec, if you can deliver the logs at that rate. If you
can't deliver the logs that fast, no amount of queue is going to be enough.
3- what is meant by dequeuebatch size and how can i add delay to control
speed.
rsyslog adds logs to the queues as they arrive with threads running the
input modules.
Other threads (worker threads) grab a 'batch' of messages from the queue,
and then run through the ruleset, doing the tests and then running the code
from output modules to deliver the logs (or if there is an action or
ruleset that's called that has it's own queue, the worker thread delivers
the message(s) to that queue and another layer of worker threads runs to
read messages from those queues)
dequeuebatchsize is the maximum number of logs that a worker thread will
pull from the queue in one loop through the process.
The reason for batches instead of always pulling individual messages is
twofold
1. there is a significant amount of overhead to manipulate the locks on
the queue, only having to do so once per 100 (or 1000) messages instead of
once per message is a very significant speed improvement
2. for some output modules (think datbases), there is very little
difference in the cost of delivering a lot of messages instead of one
message (for example, I tested inserts into postgres about a decade ago, I
could insert 1000 records as one transaction in the same time that I could
insert 2 records as 2 separate transactions). Batches give the output
modules the ability to handle multiple records at once.
so if you have the dequeuebatchdelay set for 1/10 of a second, then
rsyslog will deliver slightly less than 10 batches of logs every second
max. If each batch is 1 message, this will be just under 10 logs per
second. If each batch can be 100 logs, than this could be up to 1000 logs
per second
I say just under because it takes time to process the logs, so 1/10 sec of
sleep + log processing time will be > 1 second, so you would not quite
process 10 batches in a second, but you would probably be close to
processing 99 batches in 100 seconds, so it's not much less.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
