Hello Ben, I'm not sure another rsyslog on the Logstash side will help if the bottleneck is on indexing to Elasticsearch.
AFAIK logstash has an internal buffer of 20 or so entries, and when that's full (because the output is not fast enough) it blocks the input. At this point, you need to queue on whatever outputs to Logstash. Whether that's done by rsyslog on the client or on the Logstash side, I don't think it matters. I would try a performance test using elasticsearch_http with bulks. After you find a good bulk size for your setup, 10 to 100x gain in performance won't be a surprise. If that doesn't help, I see two options: - scale Logstash on multiple instances/servers - use omelasticsearch. I've used it quite a lot, and will be using it a whole lot more. No significant issues so far - if ES goes down, it will queue. As for performance, I've indexed 20K logs/s on a laptop. Best regards, Radu 2013/3/1 Ben Bradley <bbradle...@gmail.com> > On Fri, 1 Mar 2013 10:20:28 +0000 > Ben Bradley <bbradle...@gmail.com> wrote: > > > Hi everyone > > > > So I've got Rsyslog happily transmitting log messages over the network > to Logstash. I have disk assisted queueing on the rsyslog log "clients". > > Sometimes I don't think Logstash can keep up on the other end, it blocks > because it can't get data into ElasticSearch fast enough. I've not got > logstash using elasticsearch_http which bulks the messages into > ElasticSearch so we'll see how that goes. > > But this is for relatively low logging volume (15/sec to 40/sec). > > > > I probably need some sort of queueing system on the receiving end, in > front of Logstash. We will probably be increasing our log volume gradually > to about 5x the current level. Though I probably also need some queueing > between Logstash and ElasticSearch. > > > > I could ditch Logstash altogether and have Rsyslog pumping directly into > ElasticSearch with omelasticsearch. But I'm not sure about the reliability > of this (mention of ElasticSearch crashes) and how I need to structure the > schema to maintain compatibility with Kibana. Also Logstash's filtering and > pattern matching is (unfortunately) much easier to get working at this > point (though I have got Rsyslog doing this in testing). > > > > Or I could install Rsyslog in front of Logstash and use Rsyslog's queue > options (and zip compression). > > But what would be the best output module to use with Rsyslog to then > send to the local Logstash? > > Using omfwd to send via TCP to Logstash on the same machine seems a bit > wasteful so I looked at omuxsock but then I'm not sure Logstash can receive > through a socket. > > > > Or is this the point where I need some sort of proper queueing system. > What would be great is if there was something that could just receive the > TCP packets from Rsyslog and queue them up. Then Logstash just reads from > the queue when it can. Is that what systems like 0mq and AMQP do? > > > > Are there any queueing systems in particular that are best to use for > queueing messages from Rsyslog? > > > > Cheers, Ben > > > > > Ah... just noticed rsyslog-zmq3 and zeromq3 RPMs in the repo! > Do they do what I think? > > Only CentOS 5 though. I guess it needs to be installed as an output module > in order to send to a zeromq socket. > > Cheers, Ben > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
