[
https://issues.apache.org/jira/browse/SOLR-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628782#comment-14628782
]
Erick Erickson commented on SOLR-5146:
--------------------------------------
Scott:
Interesting. Indexing and querying are such different beasts in SolrCloud that
I'm afraid that things would get out of hand. The point is, though, that I
really have no clue except this would have to be scoped out pretty thoroughly
in order to flush out all the crannies before starting the work, it could go
down a rat-hole really quickly.
Unloading the caches would probably be viable for the indexing side. The
problem being that when an update goes out, _all_ the replicas must get it thus
would need to be loaded. But if it was just a case of the caches being absent,
that might be OK.
Querying is a bit more problematical. Assuming replicas then just because one
query went to one replica in a shard doesn't mean the next one would. I can see
this leading to perf issues. Not to mention long-loading cores (even if it's
just reloading the caches) going past ZK timeouts, the leader marking the node
as down and forcing it into recovery etc.
And the scale is interesting. I agree that the bulk of the memory is in the
various caches. That said, though, people are talking 10,000+ cores. The
overhead adds up there.
In SolrCloud mode, having this many cores really is about having this many
collections I think. The first thing that would be needed to work out is
whether this even makes sense in SolrCloud.
The model "lots of cores" was designed for is, say, e-mail searching where each
user has her own core. The assumption here is that the pattern is userA logs
on, searches for a few minutes then goes away. Not sure that model even
translates well into SolrCloud.
> Figure out what it would take for lazily-loaded cores to play nice with
> SolrCloud
> ---------------------------------------------------------------------------------
>
> Key: SOLR-5146
> URL: https://issues.apache.org/jira/browse/SOLR-5146
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Affects Versions: 4.5, Trunk
> Reporter: Erick Erickson
> Assignee: Shalin Shekhar Mangar
> Fix For: Trunk
>
>
> The whole lazy-load core thing was implemented with non-SolrCloud use-cases
> in mind. There are several user-list threads that ask about using lazy cores
> with SolrCloud, especially in multi-tenant use-cases.
> This is a marker JIRA to investigate what it would take to make lazy-load
> cores play nice with SolrCloud. It's especially interesting how this all
> works with shards, replicas, leader election, recovery, etc.
> NOTE: This is pretty much totally unexplored territory. It may be that a few
> trivial modifications are all that's needed. OTOH, It may be that we'd have
> to rip apart SolrCloud to handle this case. Until someone dives into the
> code, we don't know.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
