On Fri, Jan 23, 2015 at 8:42 PM, Drew Kutcharian <d...@venarc.com> wrote:
> Thanks Mike. I’m still a bit unclear on these comments: > > IndexReader requires some RAM for each segment to hold structures like > live docs, terms index, index data structures for doc values fields, and > holds open a number of file descriptors in proportion to how many segments > are in the index. > > There is also a per-indexed-field cost in Lucene; if you have a great many > unique indexed fields that may matter. > > > Aren’t these structures dependent on the size of the “lucene index"? Say > if I have 1 large lucene index vs 10 small lucene indices (considering not > much duplicated data across indices) wouldn’t the total memory used be the > same? I understand that there will be more file descriptors because there > will be more segments. > Yes and no. There are parts of the RAM usage that are just fixed cost per index, per segment, per field, and then there are parts (usually dominating) that correlate to how large the index is. So if you split the same index into 10 indexes, those 10 will use somewhat more RAM. > IndexWriter has a RAM buffer (indices.memory.index_buffer_size in ES) to > hold recently indexed/deleted documents, and periodically opens readers (10 > at a time by default) to do merging, which bumps up RAM usage and file > descriptors while the merge runs. > > > According to the doc at > https://github.com/elasticsearch/elasticsearch/blob/master/docs/reference/modules/indices.asciidoc > seems > like indices.memory.index_buffer_size is the “total” size of the buffer for > all the shards on a node, so not sure how this would matter in case of > having too many shards. I understand that there will be more file > descriptors and a lot more “smaller” merge jobs running. > That's true, so the indexing RAM buffer won't be different if you use 1 vs 10 indices, so it's just the fixed overhead of holding an IndexWriter open for indexing, and IndexReader(s) for searching. Mike McCandless http://blog.mikemccandless.com -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRd43Sujht%2BSGtL-1RTHPF-6KY33v%3D8n%2BehDpSyQn7CAxw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
