Re: Bulk load performance

2014-11-19 Thread xaviertrujillo111
Thank you Nick, I tried that but I didn't see a noticeable performance improvement. Also, I tried setting the number of replicas to "0", load the data, then put it back to "5", but this is causing some problems with our health check scripts, because the index is very large, and the shards seems

Re: Bulk load performance

2014-11-19 Thread Nick Canzoneri
On the index settings side, you can dynamically turn off the index refresh_interval and also reduce the number of shard replicas for the duration of the bulk import. Described here: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk On Wed, Nov

Bulk load performance

2014-11-18 Thread xaviertrujillo111
Hello, I'm trying to do a bulk load of ~10M JSON docs (12.8Gb) with some geographical information into an elasticsearch index. With our current params, the loading is taking around 20-25 minutes to run, but we think it should be faster. Are these numbers similar to what other users are getting