[rsyslog] how to force a larger omelasticsearch bulk size?

I'm using rsyslog-elasticsearch to writing nginx accesslog into Elasticsearch cluster. I found the document told that the plugin would use queue.dequeuesize as the bulk size.But my tcpdump show that every POST only has 8-9 events in the bulk body while my input flow is nearly 10k per second. How c

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

On Tue, 5 May 2015, chenlin rao wrote: I'm using rsyslog-elasticsearch to writing nginx accesslog into Elasticsearch cluster. I found the document told that the plugin would use queue.dequeuesize as the bulk size.But my tcpdump show that every POST only has 8-9 events in the bulk body while my i

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

So how can I define the output queue configuration? I found the omelasticsearch action process 6/min, and the queue.discarded.nf was 60. I run `tcpdump -i eth1 -s0 -A 'tcp dst port 9200' | grep Content-Length` and saw the length is 1.6k. As my msgline size is 0.1k, the bulk size is only 10

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

One thing is that Rsyslog has very smart batching. Instead of delaying processing messages so that they can be combined with others that arrive later, rsyslog processes all messages as fast as it can, and only batches messages when it falls behind. So instead of message arrives, logstash wai

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

Maybe this went overlooked, but David suggested earlier that you can slowdown the queue to let more messages arrive before sending a bulk. queue.dequeueslowdown is the option and it's in microseconds. I think you have a vali

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

] how to force a larger omelasticsearch bulk size? Maybe this went overlooked, but David suggested earlier that you can slowdown the queue to let more messages arrive before sending a bulk. queue.dequeueslowdown <http://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html> is the

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

15, Radu Gheorghe wrote: > > Date: Wed, 17 Jun 2015 10:20:46 +0300 >> From: Radu Gheorghe >> Reply-To: rsyslog-users >> To: rsyslog-users >> Subject: Re: [rsyslog] how to force a larger omelasticsearch bulk size? >> >> Maybe this went overlooked, but

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

David Lang On Wed, 17 Jun 2015, Radu Gheorghe wrote: Date: Wed, 17 Jun 2015 10:20:46 +0300 From: Radu Gheorghe Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] how to force a larger omelasticsearch bulk size? Maybe this went overlooked, but David suggested earlier that you

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

speed (even if >>> less optimized than if there were larger batches) But if anything else on >>> the system need the resources, the indexing threads work slower, which >>> will >>> result in larger batches. >>> >>> all self tuning. >>> >>&g

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

rch Analytics > >> Solr & Elasticsearch Support * http://sematext.com/ > >> > >> On Wed, Jun 17, 2015 at 11:53 AM, David Lang wrote: > >> > >> Thinking about it, probably the best thing to do is to renice the ES > >>> threads that accept

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

exing >> >> might fall behind more than usual. Am I getting it right? >> >> >> >> -- >> >> Performance Monitoring * Log Analytics * Search Analytics >> >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> &

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

/modules-threadpool.html > >> >> >) > >> >> > >> >> would automatically have higher priority so, with heavy searches, > >> indexing > >> >> might fall behind more than usual. Am I getting it right? > >> >>

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

http://sematext.com/ On Wed, Jun 17, 2015 at 11:53 AM, David Lang wrote: Thinking about it, probably the best thing to do is to renice the ES threads that accept the messages from rsyslog. That way if nothing else needs the capacity, everything works at the fastest insert speed (even if less

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

n Wed, 17 Jun 2015, Radu Gheorghe wrote: >>>>>> >>>>>> This sounds interesting, David. I guess it's possible to renice just >>>>>> >>>>> some >>>>> >>>>>> threads from an app and make it "nice

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

gt; seems > > >> it > > >> >> is possible. > > >> >> > > >> >> The only problem I see with this approach is that searches (and > other > > >> >> kinds > > >> >> of requests from other threadpools

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

t;> >> threads from an app and make it "nicer", right? Googling a bit it >> > seems >> > >> it >> > >> >> is possible. >> > >> >> >> > >> >> The only problem I see with t

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

gt; >> > >> > > >> > https://www.usenix.org/publications/login/feb14/logging-reports-dashboards > >> > >> > ) > >> > >> > > >> > >> > David Lang > >> > >> > > >> > >> > On Wed, 17 Jun 2015, Ra