+1. We are probably already breaching this with 256 shards, something we are still investigating.
On the question of other limits, there's also the limit on how many threads your OS can support, we have in the past hit 65k threads and errored out because of 64 shards per machine, not sure as yet if this is configurable (needless to say a scalability problem nevertheless.. I know there are some nascent ideas floating about with doing away with the thread per connection model, but none concrete afaik). We are currently migrating to 4.6.1 from 4.4, planning to attack this once that's done. The mess you've got to clear up with the 4.4 SolrCloud after its screwed makes you think twice before attempting to test it again :-) On 18 Feb 2014 06:12, "Shawn Heisey" <[email protected]> wrote: > While working on SOLR-5741, I noticed that HttpShardHandlerFactory did > not have an option to configure the max total connections. I *think* > that this value ultimately gets configured at 10000. Do we need to make > it configurable? Does this get used for SolrCloud as well as the old > distributed search? > > > http://wiki.apache.org/solr/SolrConfigXml#Configuration_of_Shard_Handlers_for_Distributed_searches > > The default of 10000 is well beyond what mere mortals will require, but > if there are 1000 shards and the search application manages to crank up > the simultaneous requests to 11, that won't be enough. > > There are other things that would need changes to handle that scenario, > like maxThreads and the HTTP header size on the container. > > If it doesn't already exist, we probably need a reference guide section > that covers what needs to be changed from defaults when you're going for > extreme scalability. > > Thanks, > Shawn > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
