So this has me perplexed. I have a bulk data loading job that creates an upsert statement and batches 500 of them in a bulk operation using the _bulk interface.
I send the bulk insert via HTTP (on 9200) and wait for the response before sending the next one, which I do immediately. I do not hit any thread pool limits. I have replicas set to zero and refresh interval set to -1 to make the loading as lightweight as possible. Timing these, they start out pretty fast and run about 2000 documents per second. Four or so HTTP round trips. This lasts for a few minutes and then it starts to slow. Within an hour, it's running about 1200 per second. In another hour, it's down to about 600 per second. Then it seems to flatten-out about 400 per second until the job is done, some 8 million documents later. So my question is - why the slowdown? It's very consistent, seems reasonably linear, and happens 100% of the time. Any clues? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a787d461-f467-4f79-943b-e65e12492783%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
