[
https://issues.apache.org/jira/browse/SOLR-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023078#comment-17023078
]
David Smiley commented on SOLR-5146:
------------------------------------
I'm going to start work on this as it's a critical feature to support thousands
of cores per node. Shawn rightfully points out it's necessary to also load
cores super-fast, though I think separate issues should exist. SOLR-14040 for
schema sharing is one, and there are smaller ones that have been contributed.
In a SolrCloud world I think we'll identify the need to do more. Also out of
scope on this specific Jira issue is directly addressing SolrCloud scale of #
of collections/shards or whatever.
The title says "lazy cores" which suggests only load-on-startup=false and is a
rather easy case but I think the scope is actually full "transient" core
compatibility, thus on-demand load/*unload* and implies a cache. I want to
edit the title to not not have the word "lazy". I expect I'll use the new
"Resource Management API" SOLR-13579 for the node-level cache, though I haven't
yet dug into the details of that.
I tried out the transient core cache in SolrCloud for the heck of it (not
expecting it to work) and sure enough, it led to an error. Debugging that
further might be a good early step to get a sense of the challenges ahead.
Like Erick, I'm hopeful that there's not much technical barrier preventing this
feature from "just working" in spite of SolrCloud. SolrCloud will think all
the cores on the node are live; the trick is that some are asleep and can be
awoken easily, but that distinction needs to be invisible to SolrCloud. I
sympathize with Scott's theory that maybe we don't do transient cores and
instead flush cashes and perhaps other things. I've thought of that a great
deal. It's promising, but you won't quite save as much memory as actually
closing the core. If I get stuck with SolrCloud transient core difficulties, I
may look at such an alternative. And ultimately we can do both; there aren't
mutually exclusive!
I'm rather unsatisfied with the implementation of the existing transient core
cache. It's weird to me that it's pluggable and has a rather large API surface
area for something conceptually straight-forward. I suppose the details are
complicated, and I'll have to dig to appreciate those complexities.
> Figure out what it would take for lazily-loaded cores to play nice with
> SolrCloud
> ---------------------------------------------------------------------------------
>
> Key: SOLR-5146
> URL: https://issues.apache.org/jira/browse/SOLR-5146
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Affects Versions: 4.5, 6.0
> Reporter: Erick Erickson
> Assignee: David Smiley
> Priority: Major
>
> The whole lazy-load core thing was implemented with non-SolrCloud use-cases
> in mind. There are several user-list threads that ask about using lazy cores
> with SolrCloud, especially in multi-tenant use-cases.
> This is a marker JIRA to investigate what it would take to make lazy-load
> cores play nice with SolrCloud. It's especially interesting how this all
> works with shards, replicas, leader election, recovery, etc.
> NOTE: This is pretty much totally unexplored territory. It may be that a few
> trivial modifications are all that's needed. OTOH, It may be that we'd have
> to rip apart SolrCloud to handle this case. Until someone dives into the
> code, we don't know.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
