I am sending logs from rsyslog 8.14.0 to ElasticSearch 1.6. I configured
omelasticsearch module as shown below. In cases where log traffic is low
the logs don't get sent to elasticsearch until I restart rsyslog. I am
assuming that's because omelasticsearch module keeps batching the logs
until dequeuebatchsize is reached. This is a problem for applications that
generate startup logs and after that infrequently generate big bursts of
logs, e.g. during some maintenance process.
Is my theory about dequeuebatchsize correct? If so, is there a way to
define a timeout within which logs should be sent to elasticsearch even if
dequeuebatchsize has not been reached?
Thanks
action(
type = "omelasticsearch"
template = "es-payload"
dynSearchIndex = "on"
searchIndex = "logstash-index"
searchType = "<%= logfile['tag'] %>"
server = "<%= elasticsearch_config['host'] %>"
serverport = "<%= elasticsearch_config['port'] %>"
errorFile = "<%= @log_dir %>/ES-error.log"
bulkmode = "on"
action.resumeretrycount="5" # retry if ES is unreachable (-1 for
infinite retries)
action.resumeInterval="60"
queue.dequeuebatchsize="1000" # ES bulk size
queue.type="linkedlist"
queue.size="100000"
queue.workerthreads="5"
queue.spoolDirectory="<%= @spool_dir %>"
queue.filename="omelasticsearch-queue"
queue.maxfilesize="100m"
queue.maxdiskspace="1g"
queue.highwatermark="50000" # when to start spilling to disk
queue.lowwatermark="20000" # when to stop spilling to disk
queue.saveonshutdown="on"
)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
