Re: scalability and creating 1 index per user

2014-03-11 Thread Otis Gospodnetic
Hi, I know some users of SPM for Elasticsearch have clusters with many thousands indexes (and growing), each with 5+ shards. They've been using SPM for man months now, plus they are our clients, so I've had the chance to see their servers and metrics and can tell you that I don't see any othe

Re: scalability and creating 1 index per user

2014-02-25 Thread Mark Walkom
20K is a lot of indexes, probably too many as ES will need to maintain state about each of those in memory which could mean you have nothing left for caching indexed data! You might want to look at http://www.elasticsearch.org/blog/customizing-your-document-routing/instead, that way you can reduce

Re: scalability and creating 1 index per user

2014-02-25 Thread Nikolas Everett
On Tue, Feb 25, 2014 at 4:46 PM, ESUser wrote: > Hi All, > I am exploring elastic search to create one index per user instead of one > big index for all the users. Each index would be about 6G. > I am wondering if anyone has tried it and how would it scale? > > I couldn't find that elastic search

scalability and creating 1 index per user

2014-02-25 Thread ESUser
Hi All, I am exploring elastic search to create one index per user instead of one big index for all the users. Each index would be about 6G. I am wondering if anyone has tried it and how would it scale? I couldn't find that elastic search has limit on maximum number of indices. Is it safe/recomm