[
https://issues.apache.org/jira/browse/SOLR-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264323#comment-15264323
]
Hrishikesh Gadre commented on SOLR-9038:
----------------------------------------
[~dsmiley] Thanks for the comments :)
>>I think so. Not doing so might be a pain, and it's not evident to me it's
>>important to worry about it.
If we are going to allow the "backup" operation to use this snapshot commit in
future, then I think we need to make sure that that snapshot commit is
preserved during collection configuration changes. If the snapshot commit is
created on all replicas for a shard, then it probably is OK to delete one or
more replicas. But I am not sure how would we handle the case when a shard
containing a one or more snapshot commits is deleted.
>>Perhaps a snapshot commit needs to block for all replicas to not be in
>>recovery first? That seems much easier than trying to get replicas in
>>recovery to somehow get IndexCommit data which I think is kinda impossible /
>>infeasible. However, another bad situation is when there are already
>>successful snapshot commits, and then for whatever reason a replica goes into
>>recovery – full recovery, and thus only grabs the latest commit (which might
>>not even be a snapshot commit. So perhaps recovering replicas need to ask to
>>replicate not just the latest commit but all snapshot commits as well. Seems
>>pretty doable. One would hope that the commits would share lots of big
>>segments, but they might not. I don't think this scenario would block an
>>initial release. Possible but too bad.
I agree that requiring replicas to transfer snapshot commits during recovery
may not be a good idea since in case of large collections it will increase the
size of data transferred over the network. I am also not very sure if we should
block for all replicas to be "active" before creating a snapshot since on a
large cluster it would more likely that one or more replicas would be "down" or
"recovering".
I do have an alternative design in mind, but just want to make sure that we are
on the same page regarding overall semantics before diving into details :)
Thoughts?
> Ability to create/delete/list snapshots for a solr collection
> -------------------------------------------------------------
>
> Key: SOLR-9038
> URL: https://issues.apache.org/jira/browse/SOLR-9038
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
> Reporter: Hrishikesh Gadre
>
> Currently work is under-way to implement backup/restore API for Solr cloud
> (SOLR-5750). SOLR-5750 is about providing an ability to "copy" index files
> and collection metadata to a configurable location.
> In addition to this, we should also provide a facility to create "named"
> snapshots for Solr collection. Here by "snapshot" I mean configuring the
> underlying Lucene IndexDeletionPolicy to not delete a specific commit point
> (e.g. using PersistentSnapshotIndexDeletionPolicy). This should not be
> confused with SOLR-5340 which implements core level "backup" functionality.
> The primary motivation of this feature is to decouple recording/preserving a
> known consistent state of a collection from actually "copying" the relevant
> files to a physically separate location. This decoupling have number of
> advantages
> - We can use specialized data-copying tools for transferring Solr index
> files. e.g. in Hadoop environment, typically
> [distcp|https://hadoop.apache.org/docs/r1.2.1/distcp2.html] tool is used to
> copy files from one location to other. This tool provides various options to
> configure degree of parallelism, bandwidth usage as well as integration with
> different types and versions of file systems (e.g. AWS S3, Azure Blob store
> etc.)
> - This separation of concern would also help Solr to focus on the key
> functionality (i.e. querying and indexing) while delegating the copy
> operation to the tools built for that purpose.
> - Users can decide if/when to copy the data files as against creating a
> snapshot. e.g. a user may want to create a snapshot of a collection before
> making an experimental change (e.g. updating/deleting docs, schema change
> etc.). If the experiment is successful, he can delete the snapshot (without
> having to copy the files). If the experiment is failed, then he can copy the
> files associated with the snapshot and restore.
> Note that Apache Blur project is also providing a similar feature
> [BLUR-132|https://issues.apache.org/jira/browse/BLUR-132]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
