Hello I'm in the process of putting a two-node Elasticsearch cluster (1.1.2) into production, but I'm having a bit of trouble keeping it stable enough for comfort. Specifically, I'm trying to figure out the best way to keep the number of deleted documents under control.
Both nodes are r3.xlarge EC2 instances (4 cores, 30.5 GB RAM). The ES cluster mirrors the primary data store, a MySQL database. Relevant updates to the database are caught via triggers which populate a table that's monitored by an indexing process. This results in what I'd consider of lot of reindexing, any time the primary data is updated. Search and indexing performance thus far has been in line with expectations when the number of deleted documents is small, but as it grows (up to 30-40%), the amount of available RAM becomes limited, ultimately causing memory problems. If I optimize/purge deletes then things return to normal, though I usually end up having to restart at least one server if not both due to OOM problems and shard failures during optimization. When ES becomes the source of all searches for the application, I can't really afford this downtime. What would be the preferred course of action here? I do have a window over the weekend where I could work with somewhat reduced capacity; I was thinking perhaps I could pull one node out of search rotation, optimize it, swap it with the other, optimize it, and then go on my way. However, I don't know that I CAN pull one node out of rotation (it seems like the search API lets me specify a node, but nothing to say "Node X doesn't need any searches"), nor does it appear that I can optimize an index on one node without doing the same to the other. I've tried tweaking the merge settings to favour segments containing large numbers of deletions, but it doesn't seem to make enough of a difference. I've also disabled merge throttling (I do have SSD-backed storage). Is there any safe way to perform regular maintenance on the cluster, preferably one node at a time, without causing TOO many problems? Am I just trying to do too much with the hardware I have? Any advice is appreciated. Let me know what info I left out that would help. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/65b96db1-0e56-4681-b73d-c21365983199%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
