Without seeing the code, it is impossible to make helpful statements. 1G is in general a small heap for bulk indexing. 275k documents will work anyway, they should be ready in ~30 seconds. Maybe you see GC starting to kick in. To make guesses about ES, you should run bulk indexing for at least 30-60 min. and not just seconds.
Note, mlockall is not working on Windows. Jörg On Tue, Mar 4, 2014 at 2:58 PM, Ondřej Spilka <spilka.ond...@gmail.com>wrote: > Hi all, > > I'm using JAVA API on ES 1.0.1 to bulk index medium sized docs. > Documents come from 150Mb XML. > Average JSON document is about 500Bytes in 10 propeties, currently testing > on 275.000 documents. Only some key properties are indexed, otherwise > stored in _source. > Bulk index is done in 5000 document blocks. > > While contiguously indexing, the speed of indexing slows linearly down, at > approx 100.000th item it took 5 times longer then at first chunk. > But when I close TrasnportClient after each successful bulk index, the > performance remains the same and indexing is breathlessly excellent. > > What causes such a problem? Is it correct to close TransportClient > connection each time bulk indexing is done? > Seems okay, index is ready and functional. > > 1GB given to Java, bootstrap.mlockall: true, ES_HEAP_SIZE = ES_MIN_MEM > = ES_MAX_MEM = 1GB > Windows 8, i7, 8GB RAM, SSD disk. > > Thanks in advance > > Ondra > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/42e50b9e-3078-462f-b5c5-51b867a34ae9%40googlegroups.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFKbVPPOGFL%3DVFRxfmvo6y31LjKuA9P61d_YejL5q0wgg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
