Re: Indexing is becoming slow, what to look for?

2014-09-09 Thread Thomas
By setting this parameter, some additional questions of mine have been generated: By setting indices.memory.index_buffer_size to a specific node and not to all nodes of the cluster, will this configuration be taken into account from all nodes? Is it going to be cluster wide or only for index

Indexing is becoming slow, what to look for?

2014-09-05 Thread Thomas
Hi, I have been performing indexing operations in my elasticsearch cluster for some time now. Suddenly, I have been facing some latency while indexing and I'm trying to find the reason for it. Details: I have a custom process which is uploading every interval a number of logs with bulk API.

Re: Indexing is becoming slow, what to look for?

2014-09-05 Thread Michael McCandless
Maybe index throttling is happening (ES would say so in the logs) because your merging is falling behind? Do you throttle IO for merges (it's throttled at paltry 20 MB / sec by default)? What does hot threads report? How about top/iostat? We just got a blog post out about improving indexing

Re: Indexing is becoming slow, what to look for?

2014-09-05 Thread Thomas
Thx Michael, I will read the post in detail and let you know for any findings Thomas. On Friday, 5 September 2014 11:44:42 UTC+3, Thomas wrote: Hi, I have been performing indexing operations in my elasticsearch cluster for some time now. Suddenly, I have been facing some latency while

Re: Indexing is becoming slow, what to look for?

2014-09-05 Thread Thomas
Hi, I wanted to clarify something from the blog post you mentioned. You specify that based on calculations we should give at most ~512 MB indexing buffer per active shard What i wanted to ask is what do we mean with the term active? Do you mean the primary only or not? Thank you again

Re: Indexing is becoming slow, what to look for?

2014-09-05 Thread Nikolas Everett
Active in this contact means currently indexing documents. On Sep 5, 2014 8:17 AM, Thomas thomas.bo...@gmail.com wrote: Hi, I wanted to clarify something from the blog post you mentioned. You specify that based on calculations we should give at most ~512 MB indexing buffer per active

Re: Indexing is becoming slow, what to look for?

2014-09-05 Thread Thomas
Got it thanks On Friday, 5 September 2014 11:44:42 UTC+3, Thomas wrote: Hi, I have been performing indexing operations in my elasticsearch cluster for some time now. Suddenly, I have been facing some latency while indexing and I'm trying to find the reason for it. Details: I have a