Hi everyone So I've got Rsyslog happily transmitting log messages over the network to Logstash. I have disk assisted queueing on the rsyslog log "clients". Sometimes I don't think Logstash can keep up on the other end, it blocks because it can't get data into ElasticSearch fast enough. I've not got logstash using elasticsearch_http which bulks the messages into ElasticSearch so we'll see how that goes. But this is for relatively low logging volume (15/sec to 40/sec).
I probably need some sort of queueing system on the receiving end, in front of Logstash. We will probably be increasing our log volume gradually to about 5x the current level. Though I probably also need some queueing between Logstash and ElasticSearch. I could ditch Logstash altogether and have Rsyslog pumping directly into ElasticSearch with omelasticsearch. But I'm not sure about the reliability of this (mention of ElasticSearch crashes) and how I need to structure the schema to maintain compatibility with Kibana. Also Logstash's filtering and pattern matching is (unfortunately) much easier to get working at this point (though I have got Rsyslog doing this in testing). Or I could install Rsyslog in front of Logstash and use Rsyslog's queue options (and zip compression). But what would be the best output module to use with Rsyslog to then send to the local Logstash? Using omfwd to send via TCP to Logstash on the same machine seems a bit wasteful so I looked at omuxsock but then I'm not sure Logstash can receive through a socket. Or is this the point where I need some sort of proper queueing system. What would be great is if there was something that could just receive the TCP packets from Rsyslog and queue them up. Then Logstash just reads from the queue when it can. Is that what systems like 0mq and AMQP do? Are there any queueing systems in particular that are best to use for queueing messages from Rsyslog? Cheers, Ben _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.