Re: Continuous async replication

[Andrew Selden]

Correct, this would create new indices. 

On Apr 21, 2014, at 2:50 PM, Mohit Anchlia  wrote:

> If I understand this correct with this pattern evey restore is a new index?
>  
> curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
> "indices": "index_1,index_2",
> "ignore_unavailable": "true",
> "include_global_state": false,
> "rename_pattern": "index_(.+)",
> "rename_replacement": "restored_index_$1"
> }'
> 
> On Mon, Apr 21, 2014 at 2:32 PM, Andrew Selden 
>  wrote:
> Hi,
> 
> How many indices you create is really up to you. Typically you would make a 
> snapshot of an index once and then schedule incremental snapshots at regular 
> intervals. 
> 
> Andrew
> 
> On Apr 21, 2014, at 2:24 PM, Mohit Anchlia  wrote:
> 
>> Thanks for the information. To keep the indexes in near real time sync I'll 
>> end up creating thousands of indexes every day alone with this method, 
>> correct?
>> 
>> On Mon, Apr 21, 2014 at 1:37 PM, Andrew Selden 
>>  wrote:
>> Hi,
>> 
>> I can see how the documentation may not be as clear as it should be on this 
>> point. Let me try to explain. If you refer back to [1], you will see an 
>> example restore command:
>> $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
>> 
>> 
>> "indices": "index_1,index_2",
>> 
>> 
>> "ignore_unavailable": "true",
>> 
>> 
>> "include_global_state": false,
>> 
>> 
>> "rename_pattern": "index_(.+)",
>> 
>> 
>> "rename_replacement": "restored_index_$1"
>> 
>> 
>> }'
>> The important parts to notice are: rename_pattern and rename_replacement. 
>> What this means is that, for any index in the repository named with the 
>> prefix 'index_', it will be restored to the cluster under the new name 
>> 'restored_index_$1', where $1 is the name of the original index. So 
>> "index_1" gets restored to "restored_index_1", meanwhile you can keep 
>> index_1 open and servicing requests. 
>> 
>> The basic operational workflow is that you would perform the restore 
>> operation with a rename. Once the restore is complete, do some sanity 
>> checking to confirm that you have in fact restored what you intended to. 
>> Finally, to make "restored_index_1" live and take "index_1" offline without 
>> impacting production service, you would use an atomic rename alias command 
>> [2]. 
>> 
>> This assumes you have been using aliases to manage your indices. For 
>> example, typically you would set up an alias to point to a 'logical' index 
>> to mask one or more 'physical' indices. This is highly recommended as it 
>> gives you the flexibility to do things like hot-swap restored/live indices 
>> as in the above example.
>> 
>> Hope that helps.
>> 
>> Andrew
>> 
>> 1. 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>> 2. 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html
>> 
>> On Apr 21, 2014, at 12:36 PM, Mohit Anchlia  wrote:
>> 
>>> This document seem to suggest that to do the restore elasticsearch need to 
>>> close the index, which means index is not available during that time while 
>>> restore is occurring?
>>> 
>>> On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden 
>>>  wrote:
>>> Hi,
>>> 
>>> Snapshot/restore is fine to run on an active, production system. The 
>>> indices are still available for normal indexing and searching operations. 
>>> You can throttle the snapshot operation if you are concerned about 
>>> impacting production quality of service [1].
>>> 
>>> I would suggest avoiding using scroll as a backup strategy. Keeping a 
>>> scroll open for a long time can be taxing on an actively used production 
>>> cluster. You are much better off using the snapshot/restore functionality. 
>>> 
>>> Andrew
>>> 
>>> 1. 
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>>> 
>>> 
>>> On Apr 21, 2014, at 9:23 AM, Mohit Anchlia  wrote:
>>> 
 Our data centers are a actively used and need to be highly available. Does 
 snapshot and restore happen in the background and indexes are still 
 available to the clients for read/write?
  
 Can Scroll be used to read indexes continuously and insert/update them?
 
 On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen  
 wrote:
 hey,
 
 snapshot and restore might be an option, if you are willing to accept a 
 lag a few minutes behind. another option might be to index/update/delete 
 all your data into two different clusters, so you dont need replication at 
 all?
 
 
 --Alex
 
 
 On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia  
 wrote:
 As I understand there is currently no feature that does async replication 
 between 2 clusters or even within the same cluster, but we have a need to 
 write one. What would be the best way to do it in elasticsearch? I was 
 thinking of leveraging Scroll for this.
 
 -- 
 You received this me

Re: Continuous async replication

[Mohit Anchlia]

If I understand this correct with this pattern evey restore is a new index?

curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
"indices": "index_1,index_2",
"ignore_unavailable": "true",
"include_global_state": false,
"rename_pattern": "index_(.+)",
"rename_replacement": "restored_index_$1"
}'

On Mon, Apr 21, 2014 at 2:32 PM, Andrew Selden <
andrew.sel...@elasticsearch.com> wrote:

> Hi,
>
> How many indices you create is really up to you. Typically you would make
> a snapshot of an index once and then schedule incremental snapshots at
> regular intervals.
>
> Andrew
>
> On Apr 21, 2014, at 2:24 PM, Mohit Anchlia  wrote:
>
> Thanks for the information. To keep the indexes in near real time sync
> I'll end up creating thousands of indexes every day alone with this method,
> correct?
>
> On Mon, Apr 21, 2014 at 1:37 PM, Andrew Selden <
> andrew.sel...@elasticsearch.com> wrote:
>
>> Hi,
>>
>> I can see how the documentation may not be as clear as it should be on
>> this point. Let me try to explain. If you refer back to [1], you will see
>> an example restore command:
>>
>> $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
>>
>> "indices": "index_1,index_2",
>>
>> "ignore_unavailable": "true",
>>
>> "include_global_state": false,
>>
>> "rename_pattern": "index_(.+)",
>>
>> "rename_replacement": "restored_index_$1"
>> }'
>>
>> The important parts to notice are: rename_pattern and rename_replacement.
>> What this means is that, for any index in the repository named with the
>> prefix ‘index_’, it will be restored to the cluster under the new name
>> ‘restored_index_$1’, where $1 is the name of the original index. So
>> “index_1" gets restored to “restored_index_1", meanwhile you can keep
>> index_1 open and servicing requests.
>>
>> The basic operational workflow is that you would perform the restore
>> operation with a rename. Once the restore is complete, do some sanity
>> checking to confirm that you have in fact restored what you intended to.
>> Finally, to make “restored_index_1” live and take “index_1” offline without
>> impacting production service, you would use an atomic rename alias command
>> [2].
>>
>> This assumes you have been using aliases to manage your indices. For
>> example, typically you would set up an alias to point to a ‘logical’ index
>> to mask one or more ‘physical’ indices. This is highly recommended as it
>> gives you the flexibility to do things like hot-swap restored/live indices
>> as in the above example.
>>
>> Hope that helps.
>>
>> Andrew
>>
>> 1.
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>> 2.
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html
>>
>> On Apr 21, 2014, at 12:36 PM, Mohit Anchlia 
>> wrote:
>>
>> This document seem to suggest that to do the restore elasticsearch need
>> to close the index, which means index is not available during that time
>> while restore is occurring?
>>
>> On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden <
>> andrew.sel...@elasticsearch.com> wrote:
>>
>>> Hi,
>>>
>>> Snapshot/restore is fine to run on an active, production system. The
>>> indices are still available for normal indexing and searching operations.
>>> You can throttle the snapshot operation if you are concerned about
>>> impacting production quality of service [1].
>>>
>>> I would suggest avoiding using scroll as a backup strategy. Keeping a
>>> scroll open for a long time can be taxing on an actively used production
>>> cluster. You are much better off using the snapshot/restore functionality.
>>>
>>> Andrew
>>>
>>> 1.
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>>>
>>>
>>> On Apr 21, 2014, at 9:23 AM, Mohit Anchlia 
>>> wrote:
>>>
>>> Our data centers are a actively used and need to be highly available.
>>> Does snapshot and restore happen in the background and indexes are still
>>> available to the clients for read/write?
>>>
>>> Can Scroll be used to read indexes continuously and insert/update them?
>>>
>>> On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen wrote:
>>>
 hey,

 snapshot and restore might be an option, if you are willing to accept a
 lag a few minutes behind. another option might be to index/update/delete
 all your data into two different clusters, so you dont need replication at
 all?


 --Alex


 On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia >>> > wrote:

> As I understand there is currently no feature that does async
> replication between 2 clusters or even within the same cluster, but we 
> have
> a need to write one. What would be the best way to do it in elasticsearch?
> I was thinking of leveraging Scroll for this.
>
> --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from th

Re: Continuous async replication

[Andrew Selden]

Hi,

How many indices you create is really up to you. Typically you would make a 
snapshot of an index once and then schedule incremental snapshots at regular 
intervals. 

Andrew

On Apr 21, 2014, at 2:24 PM, Mohit Anchlia  wrote:

> Thanks for the information. To keep the indexes in near real time sync I'll 
> end up creating thousands of indexes every day alone with this method, 
> correct?
> 
> On Mon, Apr 21, 2014 at 1:37 PM, Andrew Selden 
>  wrote:
> Hi,
> 
> I can see how the documentation may not be as clear as it should be on this 
> point. Let me try to explain. If you refer back to [1], you will see an 
> example restore command:
> $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
> 
> "indices": "index_1,index_2",
> 
> "ignore_unavailable": "true",
> 
> "include_global_state": false,
> 
> "rename_pattern": "index_(.+)",
> 
> "rename_replacement": "restored_index_$1"
> 
> }'
> The important parts to notice are: rename_pattern and rename_replacement. 
> What this means is that, for any index in the repository named with the 
> prefix 'index_', it will be restored to the cluster under the new name 
> 'restored_index_$1', where $1 is the name of the original index. So "index_1" 
> gets restored to "restored_index_1", meanwhile you can keep index_1 open and 
> servicing requests. 
> 
> The basic operational workflow is that you would perform the restore 
> operation with a rename. Once the restore is complete, do some sanity 
> checking to confirm that you have in fact restored what you intended to. 
> Finally, to make "restored_index_1" live and take "index_1" offline without 
> impacting production service, you would use an atomic rename alias command 
> [2]. 
> 
> This assumes you have been using aliases to manage your indices. For example, 
> typically you would set up an alias to point to a 'logical' index to mask one 
> or more 'physical' indices. This is highly recommended as it gives you the 
> flexibility to do things like hot-swap restored/live indices as in the above 
> example.
> 
> Hope that helps.
> 
> Andrew
> 
> 1. 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
> 2. 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html
> 
> On Apr 21, 2014, at 12:36 PM, Mohit Anchlia  wrote:
> 
>> This document seem to suggest that to do the restore elasticsearch need to 
>> close the index, which means index is not available during that time while 
>> restore is occurring?
>> 
>> On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden 
>>  wrote:
>> Hi,
>> 
>> Snapshot/restore is fine to run on an active, production system. The indices 
>> are still available for normal indexing and searching operations. You can 
>> throttle the snapshot operation if you are concerned about impacting 
>> production quality of service [1].
>> 
>> I would suggest avoiding using scroll as a backup strategy. Keeping a scroll 
>> open for a long time can be taxing on an actively used production cluster. 
>> You are much better off using the snapshot/restore functionality. 
>> 
>> Andrew
>> 
>> 1. 
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>> 
>> 
>> On Apr 21, 2014, at 9:23 AM, Mohit Anchlia  wrote:
>> 
>>> Our data centers are a actively used and need to be highly available. Does 
>>> snapshot and restore happen in the background and indexes are still 
>>> available to the clients for read/write?
>>>  
>>> Can Scroll be used to read indexes continuously and insert/update them?
>>> 
>>> On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen  
>>> wrote:
>>> hey,
>>> 
>>> snapshot and restore might be an option, if you are willing to accept a lag 
>>> a few minutes behind. another option might be to index/update/delete all 
>>> your data into two different clusters, so you dont need replication at all?
>>> 
>>> 
>>> --Alex
>>> 
>>> 
>>> On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia  
>>> wrote:
>>> As I understand there is currently no feature that does async replication 
>>> between 2 clusters or even within the same cluster, but we have a need to 
>>> write one. What would be the best way to do it in elasticsearch? I was 
>>> thinking of leveraging Scroll for this.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>> 
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

Re: Continuous async replication

[Mohit Anchlia]

Thanks for the information. To keep the indexes in near real time sync I'll
end up creating thousands of indexes every day alone with this method,
correct?

On Mon, Apr 21, 2014 at 1:37 PM, Andrew Selden <
andrew.sel...@elasticsearch.com> wrote:

> Hi,
>
> I can see how the documentation may not be as clear as it should be on
> this point. Let me try to explain. If you refer back to [1], you will see
> an example restore command:
>
> $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
> "indices": "index_1,index_2",
> "ignore_unavailable": "true",
> "include_global_state": false,
> "rename_pattern": "index_(.+)",
> "rename_replacement": "restored_index_$1"
> }'
>
> The important parts to notice are: rename_pattern and rename_replacement.
> What this means is that, for any index in the repository named with the
> prefix ‘index_’, it will be restored to the cluster under the new name
> ‘restored_index_$1’, where $1 is the name of the original index. So
> “index_1" gets restored to “restored_index_1", meanwhile you can keep
> index_1 open and servicing requests.
>
> The basic operational workflow is that you would perform the restore
> operation with a rename. Once the restore is complete, do some sanity
> checking to confirm that you have in fact restored what you intended to.
> Finally, to make “restored_index_1” live and take “index_1” offline without
> impacting production service, you would use an atomic rename alias command
> [2].
>
> This assumes you have been using aliases to manage your indices. For
> example, typically you would set up an alias to point to a ‘logical’ index
> to mask one or more ‘physical’ indices. This is highly recommended as it
> gives you the flexibility to do things like hot-swap restored/live indices
> as in the above example.
>
> Hope that helps.
>
> Andrew
>
> 1.
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
> 2.
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html
>
> On Apr 21, 2014, at 12:36 PM, Mohit Anchlia 
> wrote:
>
> This document seem to suggest that to do the restore elasticsearch need to
> close the index, which means index is not available during that time while
> restore is occurring?
>
> On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden <
> andrew.sel...@elasticsearch.com> wrote:
>
>> Hi,
>>
>> Snapshot/restore is fine to run on an active, production system. The
>> indices are still available for normal indexing and searching operations.
>> You can throttle the snapshot operation if you are concerned about
>> impacting production quality of service [1].
>>
>> I would suggest avoiding using scroll as a backup strategy. Keeping a
>> scroll open for a long time can be taxing on an actively used production
>> cluster. You are much better off using the snapshot/restore functionality.
>>
>> Andrew
>>
>> 1.
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>>
>>
>> On Apr 21, 2014, at 9:23 AM, Mohit Anchlia 
>> wrote:
>>
>> Our data centers are a actively used and need to be highly available.
>> Does snapshot and restore happen in the background and indexes are still
>> available to the clients for read/write?
>>
>> Can Scroll be used to read indexes continuously and insert/update them?
>>
>> On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen wrote:
>>
>>> hey,
>>>
>>> snapshot and restore might be an option, if you are willing to accept a
>>> lag a few minutes behind. another option might be to index/update/delete
>>> all your data into two different clusters, so you dont need replication at
>>> all?
>>>
>>>
>>> --Alex
>>>
>>>
>>> On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia 
>>> wrote:
>>>
 As I understand there is currently no feature that does async
 replication between 2 clusters or even within the same cluster, but we have
 a need to write one. What would be the best way to do it in elasticsearch?
 I was thinking of leveraging Scroll for this.

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion 

Re: Continuous async replication

[Andrew Selden]

Hi,

I can see how the documentation may not be as clear as it should be on this 
point. Let me try to explain. If you refer back to [1], you will see an example 
restore command:
$ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
"indices": "index_1,index_2",
"ignore_unavailable": "true",
"include_global_state": false,
"rename_pattern": "index_(.+)",
"rename_replacement": "restored_index_$1"
}'
The important parts to notice are: rename_pattern and rename_replacement. What 
this means is that, for any index in the repository named with the prefix 
'index_', it will be restored to the cluster under the new name 
'restored_index_$1', where $1 is the name of the original index. So "index_1" 
gets restored to "restored_index_1", meanwhile you can keep index_1 open and 
servicing requests. 

The basic operational workflow is that you would perform the restore operation 
with a rename. Once the restore is complete, do some sanity checking to confirm 
that you have in fact restored what you intended to. Finally, to make 
"restored_index_1" live and take "index_1" offline without impacting production 
service, you would use an atomic rename alias command [2]. 

This assumes you have been using aliases to manage your indices. For example, 
typically you would set up an alias to point to a 'logical' index to mask one 
or more 'physical' indices. This is highly recommended as it gives you the 
flexibility to do things like hot-swap restored/live indices as in the above 
example.

Hope that helps.

Andrew

1. 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
2. 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html

On Apr 21, 2014, at 12:36 PM, Mohit Anchlia  wrote:

> This document seem to suggest that to do the restore elasticsearch need to 
> close the index, which means index is not available during that time while 
> restore is occurring?
> 
> On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden 
>  wrote:
> Hi,
> 
> Snapshot/restore is fine to run on an active, production system. The indices 
> are still available for normal indexing and searching operations. You can 
> throttle the snapshot operation if you are concerned about impacting 
> production quality of service [1].
> 
> I would suggest avoiding using scroll as a backup strategy. Keeping a scroll 
> open for a long time can be taxing on an actively used production cluster. 
> You are much better off using the snapshot/restore functionality. 
> 
> Andrew
> 
> 1. 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
> 
> 
> On Apr 21, 2014, at 9:23 AM, Mohit Anchlia  wrote:
> 
>> Our data centers are a actively used and need to be highly available. Does 
>> snapshot and restore happen in the background and indexes are still 
>> available to the clients for read/write?
>>  
>> Can Scroll be used to read indexes continuously and insert/update them?
>> 
>> On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen  wrote:
>> hey,
>> 
>> snapshot and restore might be an option, if you are willing to accept a lag 
>> a few minutes behind. another option might be to index/update/delete all 
>> your data into two different clusters, so you dont need replication at all?
>> 
>> 
>> --Alex
>> 
>> 
>> On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia  
>> wrote:
>> As I understand there is currently no feature that does async replication 
>> between 2 clusters or even within the same cluster, but we have a need to 
>> write one. What would be the best way to do it in elasticsearch? I was 
>> thinking of leveraging Scroll for this.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsear

Re: Continuous async replication

[Mohit Anchlia]

This document seem to suggest that to do the restore elasticsearch need to
close the index, which means index is not available during that time while
restore is occurring?

On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden <
andrew.sel...@elasticsearch.com> wrote:

> Hi,
>
> Snapshot/restore is fine to run on an active, production system. The
> indices are still available for normal indexing and searching operations.
> You can throttle the snapshot operation if you are concerned about
> impacting production quality of service [1].
>
> I would suggest avoiding using scroll as a backup strategy. Keeping a
> scroll open for a long time can be taxing on an actively used production
> cluster. You are much better off using the snapshot/restore functionality.
>
> Andrew
>
> 1.
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>
>
> On Apr 21, 2014, at 9:23 AM, Mohit Anchlia  wrote:
>
> Our data centers are a actively used and need to be highly available. Does
> snapshot and restore happen in the background and indexes are still
> available to the clients for read/write?
>
> Can Scroll be used to read indexes continuously and insert/update them?
>
> On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen wrote:
>
>> hey,
>>
>> snapshot and restore might be an option, if you are willing to accept a
>> lag a few minutes behind. another option might be to index/update/delete
>> all your data into two different clusters, so you dont need replication at
>> all?
>>
>>
>> --Alex
>>
>>
>> On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia 
>> wrote:
>>
>>> As I understand there is currently no feature that does async
>>> replication between 2 clusters or even within the same cluster, but we have
>>> a need to write one. What would be the best way to do it in elasticsearch?
>>> I was thinking of leveraging Scroll for this.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpqVdoSWv552vh3ZOL7siBiMUCGRUV5NcBwAFoWPisVLQ%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/B852C0E8-A2BE-4C23-B14A-EE2593A4D7ED%40elasticsearch.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr14cNv05GAjoxtXVtUY4-uPzaxt200tp_spDNXvO_tyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Continuous async replication

[Andrew Selden]

Hi,

Snapshot/restore is fine to run on an active, production system. The indices 
are still available for normal indexing and searching operations. You can 
throttle the snapshot operation if you are concerned about impacting production 
quality of service [1].

I would suggest avoiding using scroll as a backup strategy. Keeping a scroll 
open for a long time can be taxing on an actively used production cluster. You 
are much better off using the snapshot/restore functionality. 

Andrew

1. 
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html


On Apr 21, 2014, at 9:23 AM, Mohit Anchlia  wrote:

> Our data centers are a actively used and need to be highly available. Does 
> snapshot and restore happen in the background and indexes are still available 
> to the clients for read/write?
>  
> Can Scroll be used to read indexes continuously and insert/update them?
> 
> On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen  wrote:
> hey,
> 
> snapshot and restore might be an option, if you are willing to accept a lag a 
> few minutes behind. another option might be to index/update/delete all your 
> data into two different clusters, so you dont need replication at all?
> 
> 
> --Alex
> 
> 
> On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia  
> wrote:
> As I understand there is currently no feature that does async replication 
> between 2 clusters or even within the same cluster, but we have a need to 
> write one. What would be the best way to do it in elasticsearch? I was 
> thinking of leveraging Scroll for this.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpqVdoSWv552vh3ZOL7siBiMUCGRUV5NcBwAFoWPisVLQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/B852C0E8-A2BE-4C23-B14A-EE2593A4D7ED%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.

Re: Continuous async replication

[Mohit Anchlia]

Our data centers are a actively used and need to be highly available. Does
snapshot and restore happen in the background and indexes are still
available to the clients for read/write?

Can Scroll be used to read indexes continuously and insert/update them?

On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen  wrote:

> hey,
>
> snapshot and restore might be an option, if you are willing to accept a
> lag a few minutes behind. another option might be to index/update/delete
> all your data into two different clusters, so you dont need replication at
> all?
>
>
> --Alex
>
>
> On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia wrote:
>
>> As I understand there is currently no feature that does async replication
>> between 2 clusters or even within the same cluster, but we have a need to
>> write one. What would be the best way to do it in elasticsearch? I was
>> thinking of leveraging Scroll for this.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpqVdoSWv552vh3ZOL7siBiMUCGRUV5NcBwAFoWPisVLQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Continuous async replication

[Alexander Reelsen]

hey,

snapshot and restore might be an option, if you are willing to accept a lag
a few minutes behind. another option might be to index/update/delete all
your data into two different clusters, so you dont need replication at all?


--Alex


On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia wrote:

> As I understand there is currently no feature that does async replication
> between 2 clusters or even within the same cluster, but we have a need to
> write one. What would be the best way to do it in elasticsearch? I was
> thinking of leveraging Scroll for this.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Continuous async replication

[Mohit Anchlia]

As I understand there is currently no feature that does async replication
between 2 clusters or even within the same cluster, but we have a need to
write one. What would be the best way to do it in elasticsearch? I was
thinking of leveraging Scroll for this.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.