[ https://issues.apache.org/jira/browse/SOLR-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264323#comment-15264323 ]
Hrishikesh Gadre commented on SOLR-9038: ---------------------------------------- [~dsmiley] Thanks for the comments :) >>I think so. Not doing so might be a pain, and it's not evident to me it's >>important to worry about it. If we are going to allow the "backup" operation to use this snapshot commit in future, then I think we need to make sure that that snapshot commit is preserved during collection configuration changes. If the snapshot commit is created on all replicas for a shard, then it probably is OK to delete one or more replicas. But I am not sure how would we handle the case when a shard containing a one or more snapshot commits is deleted. >>Perhaps a snapshot commit needs to block for all replicas to not be in >>recovery first? That seems much easier than trying to get replicas in >>recovery to somehow get IndexCommit data which I think is kinda impossible / >>infeasible. However, another bad situation is when there are already >>successful snapshot commits, and then for whatever reason a replica goes into >>recovery – full recovery, and thus only grabs the latest commit (which might >>not even be a snapshot commit. So perhaps recovering replicas need to ask to >>replicate not just the latest commit but all snapshot commits as well. Seems >>pretty doable. One would hope that the commits would share lots of big >>segments, but they might not. I don't think this scenario would block an >>initial release. Possible but too bad. I agree that requiring replicas to transfer snapshot commits during recovery may not be a good idea since in case of large collections it will increase the size of data transferred over the network. I am also not very sure if we should block for all replicas to be "active" before creating a snapshot since on a large cluster it would more likely that one or more replicas would be "down" or "recovering". I do have an alternative design in mind, but just want to make sure that we are on the same page regarding overall semantics before diving into details :) Thoughts? > Ability to create/delete/list snapshots for a solr collection > ------------------------------------------------------------- > > Key: SOLR-9038 > URL: https://issues.apache.org/jira/browse/SOLR-9038 > Project: Solr > Issue Type: New Feature > Components: SolrCloud > Reporter: Hrishikesh Gadre > > Currently work is under-way to implement backup/restore API for Solr cloud > (SOLR-5750). SOLR-5750 is about providing an ability to "copy" index files > and collection metadata to a configurable location. > In addition to this, we should also provide a facility to create "named" > snapshots for Solr collection. Here by "snapshot" I mean configuring the > underlying Lucene IndexDeletionPolicy to not delete a specific commit point > (e.g. using PersistentSnapshotIndexDeletionPolicy). This should not be > confused with SOLR-5340 which implements core level "backup" functionality. > The primary motivation of this feature is to decouple recording/preserving a > known consistent state of a collection from actually "copying" the relevant > files to a physically separate location. This decoupling have number of > advantages > - We can use specialized data-copying tools for transferring Solr index > files. e.g. in Hadoop environment, typically > [distcp|https://hadoop.apache.org/docs/r1.2.1/distcp2.html] tool is used to > copy files from one location to other. This tool provides various options to > configure degree of parallelism, bandwidth usage as well as integration with > different types and versions of file systems (e.g. AWS S3, Azure Blob store > etc.) > - This separation of concern would also help Solr to focus on the key > functionality (i.e. querying and indexing) while delegating the copy > operation to the tools built for that purpose. > - Users can decide if/when to copy the data files as against creating a > snapshot. e.g. a user may want to create a snapshot of a collection before > making an experimental change (e.g. updating/deleting docs, schema change > etc.). If the experiment is successful, he can delete the snapshot (without > having to copy the files). If the experiment is failed, then he can copy the > files associated with the snapshot and restore. > Note that Apache Blur project is also providing a similar feature > [BLUR-132|https://issues.apache.org/jira/browse/BLUR-132] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org