Hi Chris, Thanks for the link to Patrick's github (looks like some good stuff in there).
One thing to try (and this isn't the final word on this, but is helpful) is to go into the tree view in the Cloud panel and find out which node is hosting the Overseer (/overseer_elect/leader). When restarting your cluster, make sure you restart this node last. We've seen instances where you end up restarting the overseer node each time as you restart the cluster, which causes all kinds of craziness. I'll bet you'll see better results by doing this, but let us know either way. Also, at 600 cores per machine, I had to reduce the JVM's thread stack size to -Xss256k as there are an extreme number of threads allocated when starting up that many cores. Cheers, Timothy Potter Sr. Software Engineer, LucidWorks www.lucidworks.com ________________________________________ From: Chris W <chris1980....@gmail.com> Sent: Thursday, March 20, 2014 4:31 PM To: solr-user@lucene.apache.org Subject: Re: Limit on # of collections -SolrCloud The replication factor is two. I have equally sharded all collections across all nodes. We have a 6 node cluster setup. 300* 6 shards and 2 replicas per shard. I have almost 600 cores per machine Also one fact is that my zk timeout is in the order of 2-3 minutes. I see zk responses very slow and a lot of outstanding requests (found that out thanks to https://github.com/phunt/) On Thu, Mar 20, 2014 at 2:53 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hours sounds too long indeed. We recently had a client with several > thousand collections, but restart wasn't taking hours... > > Otis > Solr & ElasticSearch Support > http://sematext.com/ > On Mar 20, 2014 5:49 PM, "Erick Erickson" <erickerick...@gmail.com> wrote: > > > How many total replicas are we talking here? > > As in how many shards and, for each shard, > > how many replicas? I'm not asking for a long list > > here, just if you have a bazillion replicas in aggregate. > > > > Hours is surprising. > > > > Best, > > Erick > > > > On Thu, Mar 20, 2014 at 2:17 PM, Chris W <chris1980....@gmail.com> > wrote: > > > Thanks, Shalin. Making clusterstate.json on a collection basis sounds > > > awesome. > > > > > > I am not having problems with #2 . #3 is a major time hog in my > > > environment. I have over 300 +collections and restarting the entire > > cluster > > > takes in the order of hours. (2-3 hour). Can you explain more about > the > > > leaderVoteWait setting? > > > > > > > > > > > > > > > On Thu, Mar 20, 2014 at 1:28 PM, Shalin Shekhar Mangar < > > > shalinman...@gmail.com> wrote: > > > > > >> There are no arbitrary limits on the number of collections but yes > > >> there are practical limits. For example, the cluster state can become > > >> a bottleneck. There is a lot of work happening on finding and > > >> addressing these problems. See > > >> https://issues.apache.org/jira/browse/SOLR-5381 > > >> > > >> Boot up time is because of: > > >> 1) Core discovery, schema/config parsing etc > > >> 2) Transaction log replay on startup > > >> 3) Wait time for enough replicas to become available before leader > > >> election happens > > >> > > >> You can't do much about 1 right now I think. For #2, you can keep your > > >> transaction logs smaller by a hard commit before shutdown. For #3 > > >> there is a leaderVoteWait settings but I'd rather not touch that > > >> unless it becomes a problem. > > >> > > >> On Fri, Mar 21, 2014 at 1:39 AM, Chris W <chris1980....@gmail.com> > > wrote: > > >> > Hi there > > >> > > > >> > Is there a limit on the # of collections solrcloud can support? Can > > >> > zk/solrcloud handle 1000s of collections? > > >> > > > >> > Also i see that the bootup time of solrcloud increases with increase > > in # > > >> > of cores. I do not have any expensive warm up queries. How do i > > speedup > > >> > solr startup? > > >> > > > >> > -- > > >> > Best > > >> > -- > > >> > C > > >> > > >> > > >> > > >> -- > > >> Regards, > > >> Shalin Shekhar Mangar. > > >> > > > > > > > > > > > > -- > > > Best > > > -- > > > C > > > -- Best -- C
