[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552233#comment-13552233 ] Yonik Seeley commented on SOLR-1028: I started a really quick review of the code, but the first method I visited (CoreContainer.getCore()) seems to have a race condition? Trying to open the same core more than once concurrently seems like it could get pretty messy, and it doesn't seem like it's protected against. All one would have to do is send in more than one request targeted toward the same core around the same time to trigger it. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[
https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552268#comment-13552268
]
Erick Erickson commented on SOLR-1028:
--
You're right, thanks! I'm a bit reluctant to extend the synchronized block
around the entire following try block just because loading the core can be
quite a lengthy process.
I _have_ to go away until early evening, but I think I can do something about
this tonight. Do you think the risk/reward ratio is OK for 4.1? This will only
manifest itself with transient cores, which is really fully supported in 4.2
not 4.1 (yes, that _is_ a cop-out). But deadlocks are _so_ easy to introduce
when in a rush
I've raised another JIRA, SOLR-4300 and called it a blocker for now, feel
free to change that depending...
This seems really trappy, I'm wondering if for 4.2 I should encapsulate this
whole lazy and transient and permanent distinction into a class that
handles all the messy details, it's too easy to miss one of these places in the
current code. There are several other places in CoreContainer that synchronize
on cores and it's not obvious whether all of them should worry about
transient cores or not. E.g. renaming cores.
But what I'm thinking of is a pending set of cores, and the code would look
something like this: (just above the try). Note this is just a sketch, I
haven't checked it out much at all just wondering if it makes sense.
// there'd be a new static Map of cores declared, call it pending
boolean isPending = false;
synchronized(pending) {
if (pending.get(name) == null) {
pending.put(name);
isPending = true;
}
}
while (isPending) {
synchronized (pending) {
if (pending.get(name) == null) { // other thread has completed or we
wouldn't be here
look in the normal and lazy core maps and return the core,
synchronize your lookups. Bump the refcount (open)
can refactor out looking into both lists as above in this method into
a common method, it's used in several places.
throw an error if it's not in one of the cores? return null? Fall
through and try again? Throwing an error seems best
}
}
sleep(100);
Not sure how to bail out here though, this could spin forever. 1 minute?
5 minutes? Never?
}
try {
core = create(desc); // This should throw an error if it fails.
core.open();
if (desc.isTransient()) {
registerLazyCore(name, core, false);// This is a transient core
} else {
register(name, core, false); // This is a permanent, although
deferred-load core
}
} catch (Exception ex) {
throw recordAndThrow(name, Unable to create core + name, ex);
} finally {
synchronized(pending) {
pending.remove(name);
}
}
return core;
Automatic core loading unloading for multicore
--
Key: SOLR-1028
URL: https://issues.apache.org/jira/browse/SOLR-1028
Project: Solr
Issue Type: New Feature
Components: multicore
Affects Versions: 4.0, 5.0
Reporter: Noble Paul
Assignee: Erick Erickson
Fix For: 4.1, 5.0
Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch,
SOLR-1028_testnoise.patch
usecase: I have many small cores (say one per user) on a single Solr box .
All the cores are not be always needed . But when I need it I should be able
to directly issue a search request and the core must be STARTED automatically
and the request must be served.
This also requires that I must have an upper limit on the no:of cores that
should be loaded at any given point in time. If the limit is crossed the
CoreContainer must unload a core (preferably the least recently used core)
There must be a choice of specifying some cores as fixed. These cores must
never be unloaded
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551991#comment-13551991 ] Steve Rowe commented on SOLR-1028: -- Erick, can this issue be resolved? Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533110#comment-13533110 ] Shawn Heisey commented on SOLR-1028: The new swappable default of false needs a prominent mention in the .txt files in sections about upgrading from previous versions where it didn't exist. While working on some other issues on branch_4x, I was nearly bitten by this. Just before I was about to start a full rebuild, I happened to glance at my solr.xml file and noticed the new default had been added. If I hadn't done that, I would have gone through hours of rebuild only to have the core swap at the end fail. If swappable is false and you do a SolrJ CoreAdmin#swap, will it throw an exception? I believe it should, for error handling in SolrJ code. I will eventually test this myself, but I won't be able to get to it right now. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533117#comment-13533117 ] Shawn Heisey commented on SOLR-1028: Further thought -- I believe that the solr.xml file was rewritten when I did a swap between seven pairs of cores sequentially on my last rebuild yesterday. If the first swap rewrote the solr.xml with swappable=false, then I am wondering how the other six swaps were able to complete. Perhaps it's not really being enforced? Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533124#comment-13533124 ] Erick Erickson commented on SOLR-1028: -- I picked a really bad name there, I'll change it. Swappable has nothing to do with swapping cores via core admin and everything to do with whether there's a limited number of cores that are loaded at once. swappable here means that the system can automatically load/unload cores as needed to handle the case where there are lots of cores and the installation can tolerate the pain of having cores load on demand and be slow for the first few queries. There should be no change in behavior as far as the core admin handler swap commands no matter what the value of swappable in solr.xml is, it's actually an unrelated concept despite the name, but I can sure see why it would be confused Maybe cacheable? Sorry for the confusion. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533137#comment-13533137 ] Shawn Heisey commented on SOLR-1028: Erick, thank you for the clarification. Mild panic unjustified. :) The mild panic however did uncover a possible problem for you to investigate. If loadOnStartup=true and swappable=true, none of the cores get loaded on startup. Once one is accessed, it does get loaded. This is what the wiki currently says about this combination: loadOnStartup=true swappable=true: There are some cores you want loaded when the server first starts up, but that you'll allow to be swapped out. It's wasteful to specify more cores like this than your swappableCacheSize value. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533162#comment-13533162 ] Erick Erickson commented on SOLR-1028: -- Gh. See SOLR-4201. Really stupid coding error. If you're feeling really brave, change the code as per the second comment on that JIRA and give it a whirl. I should have something up tomorrow. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13530379#comment-13530379 ] Commit Tag Bot commented on SOLR-1028: -- [branch_4x commit] Yonik Seeley http://svn.apache.org/viewvc?view=revisionrevision=1420992 SOLR-2592: SOLR-1028: SOLR-3922: SOLR-3911: sync trunk with 4x Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13528011#comment-13528011 ] Erick Erickson commented on SOLR-1028: -- Has anyone with a machine that testLazyCores used to fail on seen failures since I committed my attempt at a fix? I committed SOLR-4149 last week, don't quite know if it's had enough time to really say it's fixed, unless we're seeing more failures FWIW, if it's what I think it was, it was a test artifact rather than the underlying code. I can hope anyway. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511417#comment-13511417 ] Erick Erickson commented on SOLR-1028: -- When merging into 4.x, also include r: 1417907 for SOLR-4149 Dawid: What did I see somewhere? Something about that which has been seen cannot be unseen Iron man meets the butler Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511419#comment-13511419 ] Mark Miller commented on SOLR-1028: --- Ha - I've got to hack that image into my jenkins instance. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511422#comment-13511422 ] Dawid Weiss commented on SOLR-1028: --- I think jenkins itself should detect whether it's running under a hypervisor or on bare metal and adjust accordingly... I'd file a JIRA feature request with them, LOL :) Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: jenkins.jpg, SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510570#comment-13510570 ] Erick Erickson commented on SOLR-1028: -- right, I haven't been able to devote any serious time to this this week, but I should get a chance to work on it tomorrow. Might be a race condition. Last time I tried to reproduce I couldn't. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510748#comment-13510748 ] Robert Muir commented on SOLR-1028: --- I can't reproduce these failures, even with master seed. The problem seems to be testCachingLimit. The rest of it is noise, because the try/finally in testCachingLimit doesn't close the cores it opens. This causes the endTrackingSearchers() to fire additional class-level failures which just add to noise. Is there any way we can take the endTrackingSearchers/zkClients and instead implement it as a test rule, plugging it into the class rules chain, and so it won't do this if the test already failed? Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510752#comment-13510752 ] Mark Miller commented on SOLR-1028: --- bq. This causes the endTrackingSearchers() to fire additional class-level failures which just add to noise. Yeah, that is always really annoying. You almost always have to dig for the true failure because so many failures can keep cores from being closed. I suppose many of those cases can be fixed one by one, but it would be awesome to get the real error and not always bury it under endTrackingSearchers failures. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510754#comment-13510754 ] Robert Muir commented on SOLR-1028: --- Right, i dont think tests should have to have such tricky try/finally/stuff. Back in the day when all these checks were piled into LuceneTestCase, there was a boolean 'testsFailed' which these things checked to avoid the noise. But now its cleaner with these test rules chains, I just don't know how to factor out these trackers into test rules, and placed in the chain in such a way that they work like the other checks and don't fire if the test already failed. Maybe [~dweiss] knows? Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510783#comment-13510783 ] Mark Miller commented on SOLR-1028: --- bq. I think maybe some tests depend on the sleeps done here as a side-effect? ugg...that would be a bummer, but it's certainly possible. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510786#comment-13510786 ] Dawid Weiss commented on SOLR-1028: --- It all depends where things went wrong. If it's a test that failed and you're at the after-test or after-suite level then I think it'd be possible to tell that a previous failure has occurred. In fact, it's currently being done in the code -- TestRuleMarkFailure.java contains code to intercept the status of the wrapped code. You need to be careful about a few things (assumptions are propagated as exceptions, for example) but otherwise it's pretty straightforward. One problem with the current use of this rule is that it is always called *last* to detect failures from other rules. It's a chicken-and-egg problem that doesn't have clean solutions... or so I tend to think -- I'd need to take a look at the runner again and thing if maybe this flag could be part of the randomized context. The problem in general is that you could create a test rule that intercepts exceptions (or failures) from tests and cancels them out (simply by eating up the thrown exception). So it's not just the fact of throwing an exception from a test that matters -- it's crossing the boundary from the suite class back to the runner. If this sounds complex and twisted to you it's because it kind of is... bq. You almost always have to dig for the true failure because so many failures can keep cores from being closed One thing that should help here is that the exceptions are always reported in the order in which they were captured and chained. So the first exception in the failure report is the one that happened first... I realize it's still a lot of digging, sorry. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510931#comment-13510931 ] Mark Miller commented on SOLR-1028: --- Looking at the logs of the latest failure I found a bit (this is one of only 2 or 3 types of fails I've seen though) It appears that it's trying to create the index for a new collection under the existing collection1's data dir - I don't know if this is due to a race or what, but it's obviously not supposed to be happening... oasc.CoreContainer.recordAndThrow SEVERE Unable to create core: collectionLazy5 org.apache.solr.common.SolrException: Cannot create Caused by: java.io.IOException: Cannot create directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\data\index [junit4:junit4] 2at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:171) Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510935#comment-13510935 ] Mark Miller commented on SOLR-1028: --- FYI, I've seen this on 3 machines now - my fullmetal jenkins (linux), policeman jenkins (windows), and just recently my 11 inch low powered mac book pro. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[
https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510951#comment-13510951
]
Mark Miller commented on SOLR-1028:
---
{noformat}[junit4:junit4] 2 2265 T632 oasc.SolrCore.initIndex WARNING
[collection1] Solr index directory
'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\data\index'
doesn't exist. Creating new index...
[junit4:junit4] 2 2265 T633 oasc.SolrCore.initIndex WARNING
[collectionLazy5] Solr index directory
'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test-files\solr\collection1\data\index'
doesn't exist. Creating new index...
{noformat}
So this may be the race - if collection1 creates it's index dir fast enough,
collectionLazy5 prob does not blow up - but it's probably still borked and
pointing to collection1's instance dir - I'm guessing the test is just not
catching this problem.
Automatic core loading unloading for multicore
--
Key: SOLR-1028
URL: https://issues.apache.org/jira/browse/SOLR-1028
Project: Solr
Issue Type: New Feature
Components: multicore
Affects Versions: 4.0, 5.0
Reporter: Noble Paul
Assignee: Erick Erickson
Fix For: 4.1, 5.0
Attachments: SOLR-1028.patch, SOLR-1028.patch,
SOLR-1028_testnoise.patch
usecase: I have many small cores (say one per user) on a single Solr box .
All the cores are not be always needed . But when I need it I should be able
to directly issue a search request and the core must be STARTED automatically
and the request must be served.
This also requires that I must have an upper limit on the no:of cores that
should be loaded at any given point in time. If the limit is crossed the
CoreContainer must unload a core (preferably the least recently used core)
There must be a choice of specifying some cores as fixed. These cores must
never be unloaded
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511080#comment-13511080 ] Erick Erickson commented on SOLR-1028: -- Gh! Thanks a million for this detail. I think I know what's going on. SOLR-4149 is an attempt to fix this. Can someone apply 4149 and give it a try or should I just commit 4149? Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511083#comment-13511083 ] Mark Miller commented on SOLR-1028: --- It doesn't happen enough on my mac to easily reproduce - and my mac takes a long time for each test run. I'd just commit and see how it works out. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch, SOLR-1028_testnoise.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509759#comment-13509759 ] Yonik Seeley commented on SOLR-1028: TestLazyCores seems to have failed a number of times since it was added here. http://svn.apache.org/viewvc?rev=1403710view=rev Anyone have any clues? Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490666#comment-13490666 ] Erick Erickson commented on SOLR-1028: -- I think the main thrust of these changes is compatible with SolrCloud, but the more eyes the merrier. Here's my reasoning: the idea of rapidly opening/closing cores, and limiting the number of concurrently-open cores will have to be handled locally, which is what the bulk of these changes actually are. One bit of noise here will be that I refactored the ZK-case to the local-case, so it may look worse than it is. After I looked a bit more at the ZK instance, I noticed that the SolrCloud stuff _also_ has a new coreDescriptorProvider (I'll reconcile those two as part of SOLR-1306, BTW. There's no reason to have two). So whether the descriptor comes from ZK or comes from some other place _should_ (tm) be transparent on the level of these changes. I think the biggest question for me about how ZK interacts with all this is mostly how opening/closing cores is _supposed_ to work during indexing. The whole notion of distributed indexing across a zillion rapidly opening/closing cores on a single machine really seems like something that shouldn't be happening during indexing at all. Or at least a way for users to shoot themselves in the foot. Imagine that you have 10K cores/machine, each with 3 replicas and you're randomly sending updates to those cores. Further imagine that your concurrently open core limit is 100. Throughput would be horrible. I suppose the right solution is that whoever is setting this up (and I assume they're pretty sophisticated) needs to index to a single core at a time until all the updates were sent, then go on to the _next_ core. Or pay the price speed-wise. The other bit I'm not clear about the ZK end is how we keep, say, 10K coreDescriptors in ZK with the 1M limit as has been mentioned. But again I don't think that is incompatible at all with these changes. I don't think all the JIRA's associated with SOLR-1293 need to be addressed. Some of them appear to be already done or have yet to be proved to be helpful. But since they're all local to the Solr instance anyway, I suspect they'll be the same whether in SolrCloud or not. If we go to a model where ZK runs transparently even in the normal case, then as long as the CoreDescriptorProvider is pluggable in that situation, I think we're good to go. All that said, it would be a Good Thing if anyone can poke holes in my hand-waving before I back myself into a corner. Note that if anyone looks at this, they should look at SOLR-1306 in conjunction with this JIRA. Between the two of them the bulk of the changes I'm thinking about are handled. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[
https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487324#comment-13487324
]
Hoss Man commented on SOLR-1028:
Erick: note the following recent jenkins failure...
{noformat}
java.lang.AssertionError: reload exception doesn't refer to prolog 프롤로그에서는 콘텐츠가
허용되지 않습니다.
at
__randomizedtesting.SeedInfo.seed([F661F6E616E4E41:959CD4968591C045]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.solr.core.CoreContainerCoreInitFailuresTest.testFlowBadFromStart(CoreContainerCoreInitFailuresTest.java:254)
ant test -Dtestcase=CoreContainerCoreInitFailuresTest
-Dtests.method=testFlowBadFromStart -Dtests.seed=F661F6E616E4E41
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ko -Dtests.timezone=GB
-Dtests.file.encoding=UTF-8
{noformat}
The problem seems to be that when you cleaned up what exceptions get thrown,
you also changed what logic is used in the test to verify that the expected
error was generated so that it relies on a specific (english) string prolog,
and when we run in some locals that specific string is not included in the
wraped XML exceptions - hence the previous usage of
e.getSystemId().indexOf(solrconfig.xml) in that test since that is a
garunteed property of the exception that is in our control.
Automatic core loading unloading for multicore
--
Key: SOLR-1028
URL: https://issues.apache.org/jira/browse/SOLR-1028
Project: Solr
Issue Type: New Feature
Components: multicore
Affects Versions: 4.0, 5.0
Reporter: Noble Paul
Assignee: Erick Erickson
Fix For: 4.1, 5.0
Attachments: SOLR-1028.patch, SOLR-1028.patch
usecase: I have many small cores (say one per user) on a single Solr box .
All the cores are not be always needed . But when I need it I should be able
to directly issue a search request and the core must be STARTED automatically
and the request must be served.
This also requires that I must have an upper limit on the no:of cores that
should be loaded at any given point in time. If the limit is crossed the
CoreContainer must unload a core (preferably the least recently used core)
There must be a choice of specifying some cores as fixed. These cores must
never be unloaded
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487385#comment-13487385 ] Erick Erickson commented on SOLR-1028: -- Hoss: Thanks for pointing me in the right direction here. How I made that test pass kinda sucked as per our chat, this apparently is a Java7 thing, which is why I can't seem to reproduce. Anyway, checked in a fix r: 1403936 that I'll back-port along with this issue. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487418#comment-13487418 ] Mark Miller commented on SOLR-1028: --- bq. The ugly thing I see in SolrCloud is the single huge file stored in zk . It's not suitable for tens of thousands of cores. We will easily run into multiple megabytes Yeah, this is my main concern as well - though it remains to be seen if a few megabytes is an issue on a gigbit or better network. We actually did things differently in the past, but that came with it's own issues. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487420#comment-13487420 ] Mark Miller commented on SOLR-1028: --- bq. As you said, though, I suspect we'll be hashing this all out in the how to bring this all together discussion, wherever we have it. I guess my point is that we should hash things out first. If you are going in a non solrcloud compatible direction with all of these massive core issues, I think that is a mistake. You are adding a bunch of code and complication that will just make it harder to pull away from the old style solr and into a cleaner new world. You may be creating features we will have to continue supporting or rip away from users shortly after they are added. Both things may not be a good idea. Developing along the old Solr model and just hoping everything will converge in the end seems like the wrong approach. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Affects Versions: 4.0, 5.0 Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1, 5.0 Attachments: SOLR-1028.patch, SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485987#comment-13485987 ] Erick Erickson commented on SOLR-1028: -- Yeah, I've punted entirely on the zookeeper case so far, and I've been wondering about how the two will come together. That said, I suspect SolrCloud will make use of most of this infrastructure when they do come together, the use-case is how to provide a way to automatically load/unload cores when you have a bunch of them resident but only want to have a subset live at any one time. The use-case I'm working on here is on the order of 10,000 cores with, maybe, 100-200 active at any one time. My _really_ fuzzy conception is that if we can get SolrCloud to take the place of the CoreDescriptorProvider mechanism in this kind of case, it'll just work. How all that plays with replicas/leaders/ZK the ZK election process and all of that is...er...not clear to me... FWIW, I hacked the code as an experiment to make it used in all the SolrCloud tests (without the hack, anything that's ZK sensitive doesn't use this code path) and was pleasantly surprised by how few of them failed, so I think reconciling the two ways of doing things won't be horrible. Always room for surprises of course. As you said, though, I suspect we'll be hashing this all out in the how to bring this all together discussion, wherever we have it. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1 Attachments: SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486005#comment-13486005 ] Noble Paul commented on SOLR-1028: -- I agree with Mark on this. We should not make a feature completely incompatible with SolrCloud . The ugly thing I see in SolrCloud is the single huge file stored in zk . It's not suitable for tens of thousands of cores. We will easily run into multiple megabytes Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1 Attachments: SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485679#comment-13485679 ] Erick Erickson commented on SOLR-1028: -- Here are the comments I _meant_ to include. 1 implementa loadOnStartup (default true) as a core... attribute. Causes the core to be loaded when Solr starts, just like it does now. 2 implements swappable (default false) as a core... attribute. Allows Solr to automatically unload the core if it runs out of room in a (new) LRU core list. 3 implements a swappableCacheSize attribute for cores... (default unlimited) for how large the list of swappable cores can grow to before they're closed and swapped out. 4 Wraps a bunch of exceptions in CoreContainer into a SolrException (as per comments in that code). 5 persists these two new core attributes. 6 persists the swappableCacheSize attribute of cores if set. All the tests run, so it'll be interesting to see what the build machine thinks. In all its forms. The cost of this functionality in the usual case (i.e. for current users) is, I believe, only a single check in CoreContainer.getCore when the core name isn't already found in the usual list. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1 Attachments: SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore
[ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485683#comment-13485683 ] Mark Miller commented on SOLR-1028: --- Kind of an aside to this specific issue, but in relation to all of these massive number of SolrCore issues: I'm not sure what I think about all this multi-core work if it's not going to jive with SolrCloud. Trying to develop Solr as two different systems is too costly in the long term IMO. My feeling has been that we would push away from the std mode and always run in SolrCloud mode eventually - it's just a matter of waiting until it's ready. And so introducing a lot of new work and code that is not compatible with SolrCloud makes me think. A lot of the ugly warts that are left now are around because we are straddling two different ways of doing things. The path that makes it all much nicer involves narrowing down to one way of doing things I think. Not quite the right place for the discussion here, but probably one that we should have. Automatic core loading unloading for multicore -- Key: SOLR-1028 URL: https://issues.apache.org/jira/browse/SOLR-1028 Project: Solr Issue Type: New Feature Components: multicore Reporter: Noble Paul Assignee: Erick Erickson Fix For: 4.1 Attachments: SOLR-1028.patch usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served. This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core) There must be a choice of specifying some cores as fixed. These cores must never be unloaded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
