Chris, As far as I know, AOL is using Solr with lots of cores. What I don't know is how they are handling shutting down of idle cores, which is something you'll need to do if your machine can't handle all cores being open and their data structures being populated at all times. I know I had to do that same for Simpy. :)
Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: Chris Cornell <srchn...@gmail.com> > To: solr-user@lucene.apache.org > Sent: Sunday, May 17, 2009 10:11:10 PM > Subject: Re: multicore for 20k users? > > On Sun, May 17, 2009 at 8:38 PM, Otis Gospodnetic > wrote: > > > > Chris, > > > > Yes, disk space is cheap, and with so little overlap you won't gain much by > putting everything in a single index. Plus, when each user has a separate > index, it's easy to to split users and distribute over multiple machines if > you > ever need to do that, it's easy and fast to completely reindex one user's > data > without affecting other users, etc. > > > > Several years ago I built Simpy at http://www.simpy.com/ that way (but > pre-Solr, so it uses Lucene directly) and never regretted it. There are way > more than 20K users there with many searches per second and with constant > indexing. Each user has an index for bookmarks and an index for notes. Each > group has its own index, shared by all group members. The main bookmark > search > is another index. People search is yet another index. And so on. Single > server. > > > > Thankyou very much for your insight and experience, sounds like we > shouldn't be thinking about "prematurely optimizing" this. > > Has someone actually used multicore this way, though? With thousands of them? > > Independently of advice in that regard, I guess our next step is to > explore and create some "dummy" scenarios/tests to try and stress > multicore (search latency is not as much of a factor as memory usage > is). I'll report back on any conclusion we come to. > > Thanks! > Chris
