It's normal to see 40-60% deleted docs if you frequently update existing documents. See this recent blog post I wrote for some details: http://www.elasticsearch.org/blog/lucenes-handling-of-deleted-documents/
Mike McCandless http://blog.mikemccandless.com On Sun, Feb 1, 2015 at 3:50 PM, Mark Walkom <markwal...@gmail.com> wrote: > If you're updating single documents often then you should expect high > delete rates, your heap and CPU use seems to be ok though so it's not > anything to be super concerned with (for now). You do have the option of > forcing an optimise (which does a merge and removes deleted docs), but this > is resource intensive so be careful of when you run it, ie it'd be best to > schedule it to run when your users are asleep. > > Your main indices seem to only have 2 shards, with no replicas. You should > really have one shard per node, with one replica to give you some > redundancy for a total of 6 shards, 3 primary, 3 replica. Currently you > will have an imbalance of data, which means some nodes are more overloaded > than others, plus without replicas you cannot survive the loss of a node. > > Without knowing your use, it also looks like something is creating indices > that shouldn't be there - eg phppath, cgi-bin, cfg, phpmyadmin - which > doesn't look like a problem at the moment, but is something you should > check, out if they shouldn't be there that is). > > Finally, if you want better performance, move to Oracle JDK and then > upgrade to the latest release (1.4.3). There is a lot of improvements in > the latest versions, though I cannot comment on the Groovy side of your > question. > > On 2 February 2015 at 06:31, Ricardo Fiel <ricardo.f...@rupeal.com> wrote: > >> Hi, >> >> We've been using ElasticSearch (through qBox) for almost 3 months now, >> but something's very wrong and we need help on this one. I'll try to give >> as much context as possible. >> >> Last month we had a major downtime (almost 24hr). We were using a single >> node with 2GB and after this downtime we upgraded to a 3 node 4GB setup. We >> are online again, but our Documents Deleted values are still way too high >> (40-50% and more) and they never go down, which leaves us uncomfortable. >> We've contacted qBox, which are very heplful, but they always say we need a >> bigger node because of that. It's really not an option, as hosting costs of >> our product skyrocketed with this. >> >> Our product is an invoicing platform. Our text search, filters and all >> invoice listings now get all their data from elastic instead of MySQL, so >> we can't rely (we think) on bulk indexing, as the invoices, items, clients, >> etc.. need to be indexed asap to be shown in listings. Maybe we have a >> design issue and shouldn't be using elastic for so many things, it's >> something we need to understand better. >> >> At the moment, our main concern in making sure we're ok in the long run >> and avoid another huge downtime at all costs. >> >> Can someone help in some way? We'd be very grateful. >> >> Some more info: >> >> - We're still not using round robbin. Could this help? >> - Version 1.2.4. It's on our roadmap to upgrade but we've heard >> stories of worst performance in some cases (groovy-related). >> - I've added some screens with node info from qbox. >> >> Thanks, >> Ricardo >> >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/eca7aa05-fda6-4c21-9fff-dd418d6ae2cb%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-L-C%3DSfNjRN6559%3DGsQ90qY_HsugOmV-i2Di3vOf8r1w%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRcJKMYKg-OX2CkpGezF9VDNYpXOE0mvewFU112j1yXSNg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
