Hi Alec,
Actually, Rsyslog doesn't wait for the batch size to fill. Once it gets a
log, it starts preparing a batch and sends it as soon as it's ready to send
it. We use queue.dequeueslowdown to slow it down to get more logs in a
batch.
Do you log suspensions?
global (
action.reportSuspension="on"
action.reportSuspensionContinuation="on"
)
You might also want to run Rsyslog manually using "rsyslogd -dn". It will
be very verbose, but you may get an idea of what happens.
Regards,
Ciprian
---
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Sat, Dec 12, 2015 at 8:35 PM, Alec Swan <[email protected]> wrote:
> I am sending logs from rsyslog 8.14.0 to ElasticSearch 1.6. I configured
> omelasticsearch module as shown below. In cases where log traffic is low
> the logs don't get sent to elasticsearch until I restart rsyslog. I am
> assuming that's because omelasticsearch module keeps batching the logs
> until dequeuebatchsize is reached. This is a problem for applications that
> generate startup logs and after that infrequently generate big bursts of
> logs, e.g. during some maintenance process.
>
> Is my theory about dequeuebatchsize correct? If so, is there a way to
> define a timeout within which logs should be sent to elasticsearch even if
> dequeuebatchsize has not been reached?
>
> Thanks
>
> action(
> type = "omelasticsearch"
> template = "es-payload"
> dynSearchIndex = "on"
> searchIndex = "logstash-index"
> searchType = "<%= logfile['tag'] %>"
> server = "<%= elasticsearch_config['host'] %>"
> serverport = "<%= elasticsearch_config['port'] %>"
> errorFile = "<%= @log_dir %>/ES-error.log"
> bulkmode = "on"
> action.resumeretrycount="5" # retry if ES is unreachable (-1 for
> infinite retries)
> action.resumeInterval="60"
> queue.dequeuebatchsize="1000" # ES bulk size
> queue.type="linkedlist"
> queue.size="100000"
> queue.workerthreads="5"
> queue.spoolDirectory="<%= @spool_dir %>"
> queue.filename="omelasticsearch-queue"
> queue.maxfilesize="100m"
> queue.maxdiskspace="1g"
> queue.highwatermark="50000" # when to start spilling to disk
> queue.lowwatermark="20000" # when to stop spilling to disk
> queue.saveonshutdown="on"
> )
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
