I thought I should revisit this thread in case anyone else is repeating my
mistakes, which it turns out are multiple. On the bright side, I do seem
to have resolved my issues.
tl/dr, optimize was screwing me up, and the merge settings I thought I had
in place were not actually there/active.
Hello
I do agree with both of you that my use of optimize as regular maintenance
isn't the correct way to do things, but it's been the only thing that I've
found that keeps the deleted doc count/memory under control. I very much
want to find something that works to avoid it.
I came to much
Interesting...does the very large max_merged_segment not result in memory
issues when the largest segments are merged? When I run my the cleanup
command (_optimize?only_expunge_deletes) I see a steep spike in memor as
each merge is completing, followed by an immediate drop, presumably as the
On Wed, Dec 3, 2014 at 8:32 AM, Jonathan Foy the...@gmail.com wrote:
Interesting...does the very large max_merged_segment not result in memory
issues when the largest segments are merged? When I run my the cleanup
command (_optimize?only_expunge_deletes) I see a steep spike in memor as
each
Jonathan,
Your current setup doesn't look ideal. As Nikolas pointed out, optimize
should be run under exceptional circumstances, not for regular maintenance.
That's what the merge policy setting are for, and the right settings should
meet your needs, atleast theoretically. That said, I can't say
Jonathan,
Did you find a solution to this? I've been facing pretty much the same
issue since I've added nested documents to my index - delete percentage
goes really high and an explicit optimize leads to an OOM.
Thanks.
On Saturday, August 23, 2014 8:08:32 AM UTC-7, Jonathan Foy wrote:
Hello
Hello
This is something I still struggle with, though not to the degree that I
once did. I've been in production for several months now with limited
issues, though I still don't consider it to be a solved problem for myself,
as it requires regular manual maintenance.
First, I threw more
I've had some issues with high IO exacerbated by lots of deleted docs as
well. I'd get deleted docs in the 30%-40% range on some indexes. We
attacked the problem in two ways:
1. Hardware. More ram and better SSDs really really helped. No consumer
grade SSDs for me.
2. Tweak some merge
I left some comments inline:
On Sat, Aug 23, 2014 at 5:08 PM, Jonathan Foy the...@gmail.com wrote:
I was a bit surprised to see the number of deleted docs grow so large, but
I won't rule out my having something setup wrong. Non-default merge
settings are below, by all means let me know if
Hello
I was a bit surprised to see the number of deleted docs grow so large, but
I won't rule out my having something setup wrong. Non-default merge
settings are below, by all means let me know if I've done something stupid:
indices.store.throttle.type: none
Hi Jonathan,
The default merge policy is already supposed to merge quite aggressively
segments that contain lots of deleted documents so it is a bit surprising
that you can see that many numbers of deleted documents, even with merge
throttling disabled.
You mention having memory pressure because
Hello
I'm in the process of putting a two-node Elasticsearch cluster (1.1.2) into
production, but I'm having a bit of trouble keeping it stable enough for
comfort. Specifically, I'm trying to figure out the best way to keep the
number of deleted documents under control.
Both nodes are
12 matches
Mail list logo
