Jörg, Great, I learned a lot about the process from your responses. Could you elaborate more on your use case, mine I think will be similar to yours where processing/feeding is on one server and I will use transport client, index nodes will be on EC2. So, when I do get to setting up Ec2 nodes, I believe I should be mostly looking for big cores and SSD. For current test, besides running long feeds to guage performance and checking for analyzers, I take it there isn't much else I can do to make significant impact?
On Tuesday, February 4, 2014 3:11:14 AM UTC-5, Jörg Prante wrote: > > SSD will improve overall performance very much, yes. Disk drives are the > slowest part in the chain and this will help. No more low IOPS, so it will > significantly reduce the load on CPU (less IO waits). > > More RAM will not help that much. In fact, more RAM will slow down > persisting, it increases pressure on the memory-to-disk part. ES obviously > does not depend on large RAM for persisting data, some MB suffice, but you > can try and see for yourself. > > 85 MB is not sufficient for testing index segment merging and GC effects, > you should run a bulk indexing feed not for seconds, but for at least 20-30 > minutes, if not for hours. > > Also check if your mapping can be simplified, the less complex analyzers, > the faster ES can index. > > You should also exercise your feed program how long it takes to process > your input without the part of bulk indexing. Then you see a bottom line, > and maybe more space for improvement outside ES. > > In my use case, it helped to move the feed program to another server and > use the TransportClient with a speedup of ~30%. > > I agree that 5.5M/sec is not the end of the line but that heavily depends > on your hard- and software configuration (machine, OS, file systems, JVM). > > Jörg > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8db08c83-c91d-45df-bd28-5fe49f7f32cd%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
