100 shards per index on a 10 node cluster only "feel" like being fast, because the file structures sizes are very small in the beginning and most of the memory is not allocated. But they are not. You will run into trouble much later when indexes grow and segment merges get bigger, or indexes must be recovered, or caches are used in filters/aggregations, or when tuning for faster search.
10 or 20 shards per index on 10 nodes are really more than sufficient in the vast majority of cases. Jörg On Wed, Jan 21, 2015 at 5:47 AM, Hajime <placeofnomemor...@gmail.com> wrote: > I have 440 cores in my cluster and I find that having 100 shards are much > more faster in searching compare to 20 shards. > Does having many shards per index too costly, even I have so much cores? > > > > On Wed, Jan 21, 2015 at 12:12 PM, Mark Walkom <markwal...@gmail.com> > wrote: > >> Is that 100 shards per index? If to it's too many and you should drop >> that to 10-20 per index. >> >> Check your hot threads, it might show something. >> >> On 21 January 2015 at 15:09, Hajime <placeofnomemor...@gmail.com> wrote: >> >>> Hi, >>> >>> I have a 10 indexes with 100 shards using 10 nodes which have 40 cores >>> each. >>> >>> I have a problem when indexing 1000~2000/doc or 2M~4M per sec in new >>> index, searching to the other old indexes happen to be very slow. >>> >>> How can I prevent from indexing to interfere the searching activity? >>> >>> >>> Additional info: >>> /_cat/thread_pool is almost 0 in every column. >>> load average / cpu usage / memory usage /reading usage is low and stable. >>> I use bulk api of rest client not transport client. >>> My version is 1.4.0 >>> >>> thanks, >>> >>> hajime >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAHm3Zso%2Bu0H_uG3Dw33Ubk7OxJYU8CLdypmASPZJiwto5CM_0w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-7TG5sxuCxnEjyWRWTb%2BUs9SXcREtSe-seT068L9wGmg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAHm3ZsopXSg6k_-iygZMRSrkx5x5c-u8bbLtrk_uxYp24Og%2B-A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGtJcXG4vT-fbftX0S5y41yHCsDwuycmNNGSpZ0ZKPqWg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
