Really helpful answer! When you say 'invoke warmers' are you saying to
simply set index.warmer.enabled = true ? Also, in terms of ordering should
warmers be enabled before or after an explicit optimize + refresh in a
scenario where we need the index 100% ready for search before continuing ?
Yes, I mean index.warmer.enabled = true. This is a switch for global
enabling/disabling warmers.
If you have configured warmers at index creation time - see description at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html
and warmers are enabled by the
At the moment, we're able to bulk index data at a rate faster than we
actually need. Indexing is not as important to use as being able to quickly
search for data. Once we start reaching ~30 million documents indexed, we
start to see performance decreasing in ours search queries. What are the
It might be useful to fiddle with the merge configuration
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html
to try to end up with fewer segments. That'll reduce the IO cost of the
underlying Lucene operations that filter your query before the
The 30m docs may have characteristics (volume, term freqs, mappings) so ES
limits are reached within your specific configuration. This is hard to
guess without knowing more facts.
Beside improving merge configuration, you might be able to sacrifice
indexing time by assigning limited daily
Thanks to both of you for the advise. Unfortunately setting daily indexing
times isn't an option for us, however I think I have a good idea of what
we should try next.
On Thursday, July 17, 2014 10:56:31 AM UTC-5, jnortey wrote:
At the moment, we're able to bulk index data at a rate faster
