[
https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490666#comment-13490666
]
Erick Erickson commented on SOLR-1028:
--------------------------------------
I think the main thrust of these changes is compatible with SolrCloud, but the
more eyes the merrier. Here's my reasoning:
> the idea of rapidly opening/closing cores, and limiting the number of
> concurrently-open cores will have to be handled locally, which is what the
> bulk of these changes actually are. One bit of noise here will be that I
> refactored the ZK-case to the local-case, so it may look worse than it is.
> After I looked a bit more at the ZK instance, I noticed that the SolrCloud
> stuff _also_ has a new coreDescriptorProvider (I'll reconcile those two as
> part of SOLR-1306, BTW. There's no reason to have two). So whether the
> descriptor comes from ZK or comes from "some other place" _should_ (tm) be
> transparent on the level of these changes.
> I think the biggest question for me about how ZK interacts with all this is
> mostly how opening/closing cores is _supposed_ to work during indexing. The
> whole notion of distributed indexing across a zillion rapidly opening/closing
> cores on a single machine really seems like something that shouldn't be
> happening during indexing at all. Or at least a way for users to shoot
> themselves in the foot. Imagine that you have 10K cores/machine, each with 3
> replicas and you're randomly sending updates to those cores. Further imagine
> that your concurrently open core limit is 100. Throughput would be horrible.
> I suppose the right solution is that whoever is setting this up (and I assume
> they're pretty sophisticated) needs to index to a single core at a time until
> all the updates were sent, then go on to the _next_ core. Or pay the price
> speed-wise.
> The other bit I'm not clear about the ZK end is how we keep, say, 10K
> coreDescriptors in ZK with the 1M limit as has been mentioned. But again I
> don't think that is incompatible at all with these changes.
> I don't think all the JIRA's associated with SOLR-1293 need to be addressed.
> Some of them appear to be already done or have yet to be proved to be
> helpful. But since they're all local to the Solr instance anyway, I suspect
> they'll be the same whether in SolrCloud or not.
> If we go to a model where ZK runs transparently even in the "normal" case,
> then as long as the CoreDescriptorProvider is pluggable in that situation, I
> think we're good to go.
All that said, it would be a Good Thing if anyone can poke holes in my
hand-waving before I back myself into a corner. Note that if anyone looks at
this, they should look at SOLR-1306 in conjunction with this JIRA. Between the
two of them the bulk of the changes I'm thinking about are handled.
> Automatic core loading unloading for multicore
> ----------------------------------------------
>
> Key: SOLR-1028
> URL: https://issues.apache.org/jira/browse/SOLR-1028
> Project: Solr
> Issue Type: New Feature
> Components: multicore
> Affects Versions: 4.0, 5.0
> Reporter: Noble Paul
> Assignee: Erick Erickson
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-1028.patch, SOLR-1028.patch
>
>
> usecase: I have many small cores (say one per user) on a single Solr box .
> All the cores are not be always needed . But when I need it I should be able
> to directly issue a search request and the core must be STARTED automatically
> and the request must be served.
> This also requires that I must have an upper limit on the no:of cores that
> should be loaded at any given point in time. If the limit is crossed the
> CoreContainer must unload a core (preferably the least recently used core)
> There must be a choice of specifying some cores as fixed. These cores must
> never be unloaded
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
