[jira] [Commented] (SOLR-2593) A new core admin action 'split' for splitting index
[ https://issues.apache.org/jira/browse/SOLR-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483335#comment-13483335 ] Deepak Kumar commented on SOLR-2593: I have a situation which demands 2 core merging, re-create data partitions, split install in 2(or more) cores, seems like this place has got somewhat things closer in that area, basically the case is that there are 2 cores on same schema roughly of 55G and 35G(and growing) each and data keeps on getting pushed continuously on 35G core, we can't allow it to get filled infinitely so essentially over a period of time(offline period/maintenance period) we regenrate(by re-indexing to a fresh core) both the cores with the desired set of data keyed on some unique key, discard the old oversized cores and install the fresh ones, re-indexing is a kind of pain and eventually it'll create the same set of documents but the older core will loose too older docs due to size constraint and the smaller core would be further shrinked as it'll probably be holding lesser documents due to docs getting shifted to bigger one, this can be considered as a sliding time window based core, so the basic steps in demand could be: 1.) Merge N cores to 1 big core(high cost). 2.) Scan through all the documents of the big core and create N(num of cores that were merged initially) new cores till allowed size by the side. 3.) Hot swap the main cores with the fresh ones. 4.) Discard the old cores probably after backing it up. Above 1 may be omitted if we can directly scan through documents of N cores and keep on pushing the new docs over to target cores. A new core admin action 'split' for splitting index --- Key: SOLR-2593 URL: https://issues.apache.org/jira/browse/SOLR-2593 Project: Solr Issue Type: New Feature Reporter: Noble Paul Fix For: 4.1 If an index is too large/hot it would be desirable to split it out to another core . This core may eventually be replicated out to another host. There can be to be multiple strategies * random split of x or x% * fq=user:johndoe example : action=splitsplit=20percentnewcore=my_new_index or action=splitfq=user:johndoenewcore=john_doe_index -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2593) A new core admin action 'split' for splitting index
[ https://issues.apache.org/jira/browse/SOLR-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207452#comment-13207452 ] Andrzej Bialecki commented on SOLR-2593: - Jason, see LUCENE-2632 for a possible way to implement this at the Lucene level. Splitting into arbitrary parts so far required multiple passes over input data, using the approach of tee/filter codecs it's possible to do this in one pass over the input data. A new core admin action 'split' for splitting index --- Key: SOLR-2593 URL: https://issues.apache.org/jira/browse/SOLR-2593 Project: Solr Issue Type: New Feature Reporter: Noble Paul Fix For: 4.0 If an index is too large/hot it would be desirable to split it out to another core . This core may eventually be replicated out to another host. There can be to be multiple strategies * random split of x or x% * fq=user:johndoe example : action=splitsplit=20percentnewcore=my_new_index or action=splitfq=user:johndoenewcore=john_doe_index -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2593) A new core admin action 'split' for splitting index
[ https://issues.apache.org/jira/browse/SOLR-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153533#comment-13153533 ] Terrance A. Snyder commented on SOLR-2593: -- @Noble Paul - do you have more information on this, we have a unique requirement that would greatly benefit from being able to take a 'slice' of data a user has modified and persist it in such a fashion. A new core admin action 'split' for splitting index --- Key: SOLR-2593 URL: https://issues.apache.org/jira/browse/SOLR-2593 Project: Solr Issue Type: New Feature Reporter: Noble Paul Fix For: 4.0 If an index is too large/hot it would be desirable to split it out to another core . This core may eventually be replicated out to another host. There can be to be multiple strategies * random split of x or x% * fq=user:johndoe example : action=splitsplit=20percentnewcore=my_new_index or action=splitfq=user:johndoenewcore=john_doe_index -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
