[jira] [Comment Edited] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601143#comment-13601143 ] Mark Miller edited comment on SOLR-3755 at 3/13/13 2:01 PM: Anshum caught me up in chat - I am actually the one that is confused - because slice state stuff has already been committed. I thought I was looking at pre shard splitting trunk code. The real problem here is how the slice state is being handled in relation to clusterstate.json updates - you can lose inactive slices from the clusterstate which will cause havoc. was (Author: markrmil...@gmail.com): Anshum caught me up in chat - I am actually the one that is confused - because slice state stuff has already been committed. I thought I was looking at pre shard splitting trunk code. The real problem here is how the slice state is being handled in relation to clusterstate.json updates - you can close inactive slices from the clusterstate which will cause havoc. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755-combined.patch, SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13446054#comment-13446054 ] Yonik Seeley edited comment on SOLR-3755 at 9/1/12 10:04 PM: - So we need to have new cores up and running, and then install the new indexes in them. We could either do it like replication and use a new index directory (and use a property file to redirect to that latest index), or we could try and make sure that there is no open writer on the new core and then split directly into the normal core index directory. edit: we could also simply use the existing writers on the new cores to merge in the new indexes (we just need to ensure that they are empty) was (Author: ysee...@gmail.com): So we need to have new cores up and running, and then install the new indexes in them. We could either do it like replication and use a new index directory (and use a property file to redirect to that latest index), or we could try and make sure that there is no open writer on the new core and then split directly into the normal core index directory. shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13445371#comment-13445371 ] Uwe Schindler edited comment on SOLR-3755 at 8/31/12 9:43 AM: -- Hi Yonik, looks nice, similar to oal.index.MultiPassIndexSplitter / PKIndexSplitter in misc module just using a HashPartitioned LiveDocs. I am just confused, why does it not use FixedBitSet? The length is fixed and no (int) casts needed. +1 otherwise was (Author: thetaphi): Hi Yonik, looks nice, similar to oal.index.IndexSplitter in misc module just using a HashPartitioned LiveDocs. I am just confused, why does it not use FixedBitSet? The length is fixed and no (int) casts needed. +1 otherwise shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley Attachments: SOLR-3755.patch, SOLR-3755.patch We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440698#comment-13440698 ] Yonik Seeley edited comment on SOLR-3755 at 8/29/12 6:51 AM: - We need to associate hash ranges with shards and allow overlapping shards (i.e. 1-10, 1-5,6-10) General Strategy for splitting w/ no service interruptions: - Bring up 2 new cores on the same node, covering the new hash ranges - Both cores should go into recovery mode (i.e. leader should start forwarding updates) - leaders either need to consider these new smaller shards as replicas, or they need to forward to the leader for the new smaller shard - searches should no longer go across all shards, but should just span the complete hash range - leader does a hard commit and splits the index - Smaller indexes are installed on the new cores - Overseer should create new replicas for new shards - Mark old shard as “retired” – some mechanism to shut it down (after there is an acceptable amount of coverage of the new shards via replicas) Future: allow splitting even with “custom” shards was (Author: ysee...@gmail.com): We need to associate hash ranges with shards and allow overlapping shards (i.e. 1-10, 1-5,6-10) General Strategy for splitting w/ no service interruptions: - Bring up 2 new cores on the same node, covering the new hash ranges - Both cores should go into recovery mode (i.e. leader should start forwarding updates) - leader does a hard commit and splits the index - Smaller indexes are installed on the new cores - Overseer should create new replicas for new shards - Mark old shard as “retired” – some mechanism to shut it down (after there is an acceptable amount of coverage of the new shards via replicas) Future: allow splitting even with “custom” shards shard splitting --- Key: SOLR-3755 URL: https://issues.apache.org/jira/browse/SOLR-3755 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Yonik Seeley We can currently easily add replicas to handle increases in query volume, but we should also add a way to add additional shards dynamically by splitting existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
