On Fri, 1 Mar 2013, Ben Bradley wrote:
Hi everyone
So I've got Rsyslog happily transmitting log messages over the network to Logstash. I
have disk assisted queueing on the rsyslog log "clients".
Sometimes I don't think Logstash can keep up on the other end, it blocks
because it can't get data into ElasticSearch fast enough. I've not got logstash
using elasticsearch_http which bulks the messages into ElasticSearch so we'll
see how that goes.
But this is for relatively low logging volume (15/sec to 40/sec).
I probably need some sort of queueing system on the receiving end, in front of
Logstash. We will probably be increasing our log volume gradually to about 5x
the current level. Though I probably also need some queueing between Logstash
and ElasticSearch.
I could ditch Logstash altogether and have Rsyslog pumping directly into
ElasticSearch with omelasticsearch. But I'm not sure about the reliability of
this (mention of ElasticSearch crashes) and how I need to structure the schema
to maintain compatibility with Kibana. Also Logstash's filtering and pattern
matching is (unfortunately) much easier to get working at this point (though I
have got Rsyslog doing this in testing).
Or I could install Rsyslog in front of Logstash and use Rsyslog's queue options
(and zip compression).
But what would be the best output module to use with Rsyslog to then send to
the local Logstash?
Using omfwd to send via TCP to Logstash on the same machine seems a bit
wasteful so I looked at omuxsock but then I'm not sure Logstash can receive
through a socket.
Or is this the point where I need some sort of proper queueing system. What
would be great is if there was something that could just receive the TCP
packets from Rsyslog and queue them up. Then Logstash just reads from the queue
when it can. Is that what systems like 0mq and AMQP do?
Are there any queueing systems in particular that are best to use for queueing
messages from Rsyslog?
You have correctly identified the problem as being on the recieving side. If
logstash can't keep up, then adding a rsyslog daemon on the reciever to forward
to logstash is only going to move the bottleneck to that new rsyslog instance,
you still will have the problem of logstash not keeping up with the rate of new
traffic.
You need to find out if the problem is in logstash or elasticsearch.
If it's in logstash, then eliminating it and having rsyslog push the traffic to
elasticsearch can help, but if the bottleneck is in elasticsearch, then you need
to speed it up.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
