Hello I was a bit surprised to see the number of deleted docs grow so large, but I won't rule out my having something setup wrong. Non-default merge settings are below, by all means let me know if I've done something stupid:
indices.store.throttle.type: none index.merge.policy.reclaim_deletes_weight: 6.0 index.merge.policy.max_merge_at_once: 5 index.merge.policy.segments_per_tier: 5 index.merge.policy.max_merged_segment: 2gb index.merge.scheduler.max_thread_count: 3 I make extensive use of nested documents, and to a smaller degree child docs. Right now things are hovering around 15% deleted after a cleanup on Wednesday. I've also cleaned up my mappings a lot since I saw the 45% deleted number (less redundant data, broke some things off into child docs to maintain separately), but it was up to 30% this last weekend. When I've looked in the past when I saw the 40+% numbers, the segments in the largest tier (2 GB) would sometimes have up to 50+% deleted docs in them, the smaller segments all seemed pretty contained, which I guess makes sense as they didn't stick around for nearly as long. As for where the memory is spent, according to ElasticHQ, right now on one server I have a 20 GB heap (out of 30.5, which I know is above the 50% suggested, just trying to get things to work), I'm using 90% as follows: Field cache: 5.9 GB Filter cache: 4.0 GB (I had reduced this before the last restart, but forgot to make the changes permanent. I do use a lot of filters though, so would like to be able to use the cache). ID cache: 3.5 GB Node stats "Segments: memory_in_bytes": 6.65 GB (I'm not exactly sure how this one contributes to the total heap number). As for the disk-based "doc values", I don't know how I have not come across them thus far, but that sounds quite promising. I'm a little late in the game to be changing everything yet again, but it may be a good idea regardless, and is definitely something I'll read more about and consider going forward. Thank you for bringing it to my attention. Anyway, my current plan, since I'm running in AWS and have the flexibility, is just to add another r3.xlarge node to the cluster over the weekend, try the deleted-doc purge, and then pull the node back out after moving all shards off of it. I'm hoping this will allow me to clean things up with extra horsepower, but not increase costs too much throughout the week. Thanks for you input, it's very much appreciated. On Friday, August 22, 2014 7:14:18 PM UTC-4, Adrien Grand wrote: > > Hi Jonathan, > > The default merge policy is already supposed to merge quite aggressively > segments that contain lots of deleted documents so it is a bit surprising > that you can see that many numbers of deleted documents, even with merge > throttling disabled. > > You mention having memory pressure because of the number of documents in > your index, do you know what causes this memory pressure? In case it is due > to field data maybe you could consider storing field data on disk? (what we > call "doc values") > > > > On Fri, Aug 22, 2014 at 5:27 AM, Jonathan Foy <the...@gmail.com > <javascript:>> wrote: > >> Hello >> >> I'm in the process of putting a two-node Elasticsearch cluster (1.1.2) >> into production, but I'm having a bit of trouble keeping it stable enough >> for comfort. Specifically, I'm trying to figure out the best way to keep >> the number of deleted documents under control. >> >> Both nodes are r3.xlarge EC2 instances (4 cores, 30.5 GB RAM). The ES >> cluster mirrors the primary data store, a MySQL database. Relevant updates >> to the database are caught via triggers which populate a table that's >> monitored by an indexing process. This results in what I'd consider of lot >> of reindexing, any time the primary data is updated. Search and indexing >> performance thus far has been in line with expectations when the number of >> deleted documents is small, but as it grows (up to 30-40%), the amount of >> available RAM becomes limited, ultimately causing memory problems. If I >> optimize/purge deletes then things return to normal, though I usually end >> up having to restart at least one server if not both due to OOM problems >> and shard failures during optimization. When ES becomes the source of all >> searches for the application, I can't really afford this downtime. >> >> What would be the preferred course of action here? I do have a window >> over the weekend where I could work with somewhat reduced capacity; I was >> thinking perhaps I could pull one node out of search rotation, optimize it, >> swap it with the other, optimize it, and then go on my way. However, I >> don't know that I CAN pull one node out of rotation (it seems like the >> search API lets me specify a node, but nothing to say "Node X doesn't need >> any searches"), nor does it appear that I can optimize an index on one node >> without doing the same to the other. >> >> I've tried tweaking the merge settings to favour segments containing >> large numbers of deletions, but it doesn't seem to make enough of a >> difference. I've also disabled merge throttling (I do have SSD-backed >> storage). Is there any safe way to perform regular maintenance on the >> cluster, preferably one node at a time, without causing TOO many problems? >> Am I just trying to do too much with the hardware I have? >> >> Any advice is appreciated. Let me know what info I left out that would >> help. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/65b96db1-0e56-4681-b73d-c21365983199%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/65b96db1-0e56-4681-b73d-c21365983199%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Adrien Grand > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/14ba25f3-db43-4604-b78e-92a2a5213c0f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
