Aleksey, It was a less than ideal situation. because we did not have a choice. We had external systems/scripts to manage this. A new custom implementation is being built on SolrCloud which would have taken care of most of hose issues.
SolrReplication is a hidden once you move to cloud. But it will continue in the same way if you have a stand-lone deployment. On Mon, Jun 10, 2013 at 1:20 AM, Aleksey <bitterc...@gmail.com> wrote: > Thanks Paul. Just a little clarification: > > You mention that you migrate data using built-in replication, but if > you map and route users yourself, doesn't that mean that you also need > to manage replication yourself? Your routing logic needs to be aware > of how to map both replicas for each user, and if one hosts goes down, > then it needs to distribute traffic that it was receiving over other > hosts. Same thing for adding more hosts. > I did a couple of quick searches and found mostly older wikis that say > solr replication will change in the future. Would you be able to point > me to the right one? > > > - > > On Fri, Jun 7, 2013 at 8:34 PM, Noble Paul നോബിള് नोब्ळ् > <noble.p...@gmail.com> wrote: > > We set it up like this > > + individual solr instances are setup > > + external mapping/routing to allocate users to instances. This > information > > can be stored in an external data store > > + all cores are created as transient and loadonstart as false > > + cores come online on demand > > + as and when users data get bigger (or hosts are hot)they are migrated > > between less hit hosts using in built replication > > > > Keep in mind we had the schema for all users. Currently there is no way > to > > upload a new schema to solr. > > On Jun 8, 2013 1:15 AM, "Aleksey" <bitterc...@gmail.com> wrote: > > > >> > Aleksey: What would you say is the average core size for your use > case - > >> > thousands or millions of rows? And how sharded would each of your > >> > collections be, if at all? > >> > >> Average core/collection size wouldn't even be thousands, hundreds more > >> like. And the largest would be half a million or so but that's a > >> pathological case. I don't need sharding and queries than fan out to > >> different machines. If fact I'd like to avoid that so I don't have to > >> collate the results. > >> > >> > >> > The Wiki page was built not for Cloud Solr. > >> > > >> > We have done such a deployment where less than a tenth of cores were > >> active > >> > at any given point in time. though there were tens of million indices > >> they > >> > were split among a large no:of hosts. > >> > > >> > If you don't insist of Cloud deployment it is possible. I'm not sure > if > >> it > >> > is possible with cloud > >> > >> By Cloud you mean specifically SolrCloud? I don't have to have it if I > >> can do without it. Bottom line is I want a bunch of small cores to be > >> distributed over a fleet, each core completely fitting on one server. > >> Would you be willing to provide a little more details on your setup? > >> In particular, how are you managing the cores? > >> How do you route requests to proper server? > >> If you scale the fleet up and down, does reshuffling of the cores > >> happen automatically or is it an involved manual process? > >> > >> Thanks, > >> > >> Aleksey > >> > -- ----------------------------------------------------- Noble Paul
