Hi Chris, Usually, the problem is not that much in terms of indices but shards, which are the physical units of data storage (an index being a logical view over several shards).
Something to beware of is that shards typically have some constant overhead (disk space, file descriptors, memory usage) that does not depend on the amount of data that they store. Although it would be ok to have up to a few tens of shards per nodes, you should avoid to have eg. thousands of shards per node. if you plan on always adding a filter for a specific application in your search requests, then splitting by application makes sense since this will make the filter useless at search time, you will just need to query the application-specific index. On the other hand if you don't filter by application, then splitting data by yourself into smaller indices would be pretty equivalent to storing everything in a single index with a higher number of shards. You might want to check out the following resources that talk about capacity planning: - http://www.elasticsearch.org/videos/big-data-search-and-analytics/ - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/capacity-planning.html On Fri, Aug 22, 2014 at 9:08 PM, Chris Neal <chris.n...@derbysoft.net> wrote: > Hi all, > > As the subject says, I'm wondering about index size vs. number of indexes. > > I'm indexing many application log files, currently with an index by day > for all logs, which will make a very large index. For just a few > applications in Development, the index is 55GB a day (across 2 servers). > In prod with all applications, it will be "much more than that". 1TB a > day maybe? > > I'm wondering if there is value in splitting the indexes by day and by > application, which would produce more indexes per day, but they would be > smaller, vs. value in having a single, mammoth index by day alone. > > Is it just a resource question? If I have enough RAM/disk/CPU to support > a "mammoth" index, then I'm fine? Or are there other reasons to (or to > not) split up indexes? > > Very much appreciate your time. > Chris > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAND3DphfsYx0LW0M-yvLWGauRSzVWG0etaBkiTrN7zVafq7tMA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5i7AAnasMYZgR83aTXvELan%3DkR6OLvGYKfs9d5Subi4A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
