[ 
https://issues.apache.org/jira/browse/SOLR-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023078#comment-17023078
 ] 

David Smiley commented on SOLR-5146:
------------------------------------

I'm going to start work on this as it's a critical feature to support thousands 
of cores per node.  Shawn rightfully points out it's necessary to also load 
cores super-fast, though I think separate issues should exist.  SOLR-14040 for 
schema sharing is one, and there are smaller ones that have been contributed.  
In a SolrCloud world I think we'll identify the need to do more.  Also out of 
scope on this specific Jira issue is directly addressing SolrCloud scale of # 
of collections/shards or whatever.

The title says "lazy cores" which suggests only load-on-startup=false and is a 
rather easy case but I think the scope is actually full "transient" core 
compatibility, thus on-demand load/*unload* and implies a cache.  I want to 
edit the title to not not have the word "lazy".  I expect I'll use the new 
"Resource Management API" SOLR-13579 for the node-level cache, though I haven't 
yet dug into the details of that.

I tried out the transient core cache in SolrCloud for the heck of it (not 
expecting it to work) and sure enough, it led to an error.  Debugging that 
further might be a good early step to get a sense of the challenges ahead.  
Like Erick, I'm hopeful that there's not much technical barrier preventing this 
feature from "just working" in spite of SolrCloud.  SolrCloud will think all 
the cores on the node are live; the trick is that some are asleep and can be 
awoken easily, but that distinction needs to be invisible to SolrCloud.  I 
sympathize with Scott's theory that maybe we don't do transient cores and 
instead flush cashes and perhaps other things.  I've thought of that a great 
deal.  It's promising, but you won't quite save as much memory as actually 
closing the core.  If I get stuck with SolrCloud transient core difficulties, I 
may look at such an alternative.  And ultimately we can do both; there aren't 
mutually exclusive!

I'm rather unsatisfied with the implementation of the existing transient core 
cache.  It's weird to me that it's pluggable and has a rather large API surface 
area for something conceptually straight-forward.  I suppose the details are 
complicated, and I'll have to dig to appreciate those complexities.

 

 

> Figure out what it would take for lazily-loaded cores to play nice with 
> SolrCloud
> ---------------------------------------------------------------------------------
>
>                 Key: SOLR-5146
>                 URL: https://issues.apache.org/jira/browse/SOLR-5146
>             Project: Solr
>          Issue Type: Improvement
>          Components: SolrCloud
>    Affects Versions: 4.5, 6.0
>            Reporter: Erick Erickson
>            Assignee: David Smiley
>            Priority: Major
>
> The whole lazy-load core thing was implemented with non-SolrCloud use-cases 
> in mind. There are several user-list threads that ask about using lazy cores 
> with SolrCloud, especially in multi-tenant use-cases.
> This is a marker JIRA to investigate what it would take to make lazy-load 
> cores play nice with SolrCloud. It's especially interesting how this all 
> works with shards, replicas, leader election, recovery, etc.
> NOTE: This is pretty much totally unexplored territory. It may be that a few 
> trivial modifications are all that's needed. OTOH, It may be that we'd have 
> to rip apart SolrCloud to handle this case. Until someone dives into the 
> code, we don't know.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

Reply via email to