Which version of ES? This is probably not related to the slowdown, but when using scripts for updating docs, it's best to keep the script constant, and use params for the changing values (all the $vars in your PHP script). This means ES will compile the script once and reuse that, vs paying compilation cost for every update.
Do the node logs say anything about index throttling? Maybe catch and post some hot threads once your'e down to 400 per second? Mike McCandless http://blog.mikemccandless.com On Thu, Nov 20, 2014 at 4:08 PM, Christopher Ambler < const.dogbe...@gmail.com> wrote: > So this has me perplexed. > > I have a bulk data loading job that creates an upsert statement and > batches 500 of them in a bulk operation using the _bulk interface. > > I send the bulk insert via HTTP (on 9200) and wait for the response before > sending the next one, which I do immediately. > > I do not hit any thread pool limits. > > I have replicas set to zero and refresh interval set to -1 to make the > loading as lightweight as possible. > > Timing these, they start out pretty fast and run about 2000 documents per > second. Four or so HTTP round trips. > > This lasts for a few minutes and then it starts to slow. Within an hour, > it's running about 1200 per second. In another hour, it's down to about 600 > per second. Then it seems to flatten-out about 400 per second until the job > is done, some 8 million documents later. > > So my question is - why the slowdown? It's very consistent, seems > reasonably linear, and happens 100% of the time. > > Any clues? > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/a787d461-f467-4f79-943b-e65e12492783%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/a787d461-f467-4f79-943b-e65e12492783%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRcxAuykzqyBe%2BszoouFMerFzE5mJ%2BJnbdASk-zCwk_n7A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
