[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490666#comment-13490666 ]
Erick Erickson commented on SOLR-1028: -------------------------------------- I think the main thrust of these changes is compatible with SolrCloud, but the more eyes the merrier. Here's my reasoning: > the idea of rapidly opening/closing cores, and limiting the number of > concurrently-open cores will have to be handled locally, which is what the > bulk of these changes actually are. One bit of noise here will be that I > refactored the ZK-case to the local-case, so it may look worse than it is. > After I looked a bit more at the ZK instance, I noticed that the SolrCloud > stuff _also_ has a new coreDescriptorProvider (I'll reconcile those two as > part of SOLR-1306, BTW. There's no reason to have two). So whether the > descriptor comes from ZK or comes from "some other place" _should_ (tm) be > transparent on the level of these changes. > I think the biggest question for me about how ZK interacts with all this is > mostly how opening/closing cores is _supposed_ to work during indexing. The > whole notion of distributed indexing across a zillion rapidly opening/closing > cores on a single machine really seems like something that shouldn't be > happening during indexing at all. Or at least a way for users to shoot > themselves in the foot. Imagine that you have 10K cores/machine, each with 3 > replicas and you're randomly sending updates to those cores. Further imagine > that your concurrently open core limit is 100. Throughput would be horrible. > I suppose the right solution is that whoever is setting this up (and I assume > they're pretty sophisticated) needs to index to a single core at a time until > all the updates were sent, then go on to the _next_ core. Or pay the price > speed-wise. > The other bit I'm not clear about the ZK end is how we keep, say, 10K > coreDescriptors in ZK with the 1M limit as has been mentioned. But again I > don't think that is incompatible at all with these changes. > I don't think all the JIRA's associated with SOLR-1293 need to be addressed. > Some of them appear to be already done or have yet to be proved to be > helpful. But since they're all local to the Solr instance anyway, I suspect > they'll be the same whether in SolrCloud or not. > If we go to a model where ZK runs transparently even in the "normal" case, > then as long as the CoreDescriptorProvider is pluggable in that situation, I > think we're good to go. All that said, it would be a Good Thing if anyone can poke holes in my hand-waving before I back myself into a corner. Note that if anyone looks at this, they should look at SOLR-1306 in conjunction with this JIRA. Between the two of them the bulk of the changes I'm thinking about are handled. > Automatic core loading unloading for multicore > ---------------------------------------------- > > Key: SOLR-1028 > URL: https://issues.apache.org/jira/browse/SOLR-1028 > Project: Solr > Issue Type: New Feature > Components: multicore > Affects Versions: 4.0, 5.0 > Reporter: Noble Paul > Assignee: Erick Erickson > Fix For: 4.1, 5.0 > > Attachments: SOLR-1028.patch, SOLR-1028.patch > > > usecase: I have many small cores (say one per user) on a single Solr box . > All the cores are not be always needed . But when I need it I should be able > to directly issue a search request and the core must be STARTED automatically > and the request must be served. > This also requires that I must have an upper limit on the no:of cores that > should be loaded at any given point in time. If the limit is crossed the > CoreContainer must unload a core (preferably the least recently used core) > There must be a choice of specifying some cores as fixed. These cores must > never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org