Turns out it was because the bulk thread pool queue size was too small, any new requests were being rejected.
Is it common to set threadpool.bulk.queue_size to something like 1000 ? On Tuesday, 7 April 2015 11:10:33 UTC+1, Jörg Prante wrote: > > Do you evaluate the bulk request responses? > > Jörg > > On Tue, Apr 7, 2015 at 11:16 AM, mzrth_7810 <afraz...@gmail.com > <javascript:>> wrote: > >> Hey everyone, >> >> I've been trying to maximise my indexing rate. I'm indexing around a >> million documents, using 4 threads. Each thread is indexing at 2500 >> documents per bulk request, so that's 10000 at a time. >> >> I've been playing around with these parameters and found that if I go >> higher and higher, I eventually start getting data-loss. The index ends up >> with less than 1,000,000 documents every time. There are no error in the >> logs, so I'm not sure what's causing this. >> >> Taking these parameters down a notch fixes this problem. >> >> Has anyone seen this issue before? >> Is there anything that can be done about it >> >> Thankyou >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/181a75a6-7a12-421e-9757-a82876b24a15%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/84469eb5-4fa3-480e-951c-712c2a31ff3b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
