We double checked and actually there is an issue in the elasticsearch core documentation about this. This verify option is available whatever the storage you are using.
But it's a URL attribute. So you should do something like: $ curl -XPUT 'http://localhost:9200/_snapshot/amos0?verify=false' -d '{ "type":"s3", "settings": { "region": "ap-southeast-1", "bucket": "prod-es-backup", "base_path": "elasticsearch/dev/snapshots0" } }' Thanks for reporting this and for the tests BTW. -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr <https://twitter.com/elasticsearchfr> | @scrutmydocs <https://twitter.com/scrutmydocs> > Le 28 janv. 2015 à 08:25, David Pilato <da...@pilato.fr> a écrit : > > Perfect. Thanks. > > David > > Le 28 janv. 2015 à 03:55, Amos S <amos.shap...@gmail.com > <mailto:amos.shap...@gmail.com>> a écrit : > >> I opened an issue for AWS plugin project on github, I hope this is what you >> were referring to. Here is the issue: >> https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/167 >> <https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/167> >> About the "type missing" error - it turned out to be my mistake in trying to >> copy the output of the "GET" verbatim to the input of the PUT, it turned out >> that I had to peel off a couple of "{}"'s. Once I did that, the order of the >> "type" attribute in relation to the rest didn't matter and the PUT >> succeeded. See the update I gave in a previous message. >> I ended up making a new copy of the bucket. I'm now trying to restore it to >> the test node. >> Thanks for your help. >> >> --Amos >> >> On Wednesday, 28 January 2015 13:18:24 UTC+11, David Pilato wrote: >> Could you open an issue in AWS plugin project (and may be in azure and gce) >> to support verify option as well? >> >> BTW, I think we should try to support have type after settings or to clearly >> document it needs to be on the first line. Could you open an issue for this >> in elasticsearch? >> >> Coming back to your issue, I'm afraid you need to wait for a next cloud >> plugin release or patch it yourself (and send a PR) or make your repo >> writable. >> >> Best >> >> -- >> David ;-) >> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >> >> Le 27 janv. 2015 à 23:30, Amos S <amos.s...@gmail.com <javascript:>> a écrit >> : >> >>> OK, following previous responses by you about the "type is missing" error, >>> I corrected the JSON payload I send to the PUT and got another error: >>> >>> $ curl -XPUT 'http://localhost:9200/_snapshot/amos0' >>> <http://localhost:9200/_snapshot/amos0'> -d '{ >>> "type":"s3", >>> "settings": { >>> "region": "ap-southeast-1", >>> "bucket": "prod-es-backup", >>> "base_path": "elasticsearch/dev/snapshots0", >>> "verify": "false" >>> } >>> }' >>> {"error":"RepositoryVerificationException[[amos0] path >>> [elasticsearch][dev][snapshots0] is not accessible on master node]; nested: >>> IOException[Unable to upload object >>> elasticsearch/dev/snapshots0/tests-ng9f5N6tTm6HZhrGF9s8aQ-master]; nested: >>> AmazonS3Exception[Access Denied (Service: Amazon S3; Status Code: 403; >>> Error Code: AccessDenied; Request ID: 7F3DE23BB617FFF4)]; ","status":500} >>> >>> I think this confirms your suspicion that the repo creation process tries >>> to verify the repository by uploading a test object onto it and also that >>> it ignores the "verify: false" setting. >>> >>> I can either allow this role to write only to this specific prefix or just >>> make a copy of the bucket and allow access to the copy. I'll try the later. >>> >>> Thanks, >>> >>> --Amos >>> >>> On Wednesday, 28 January 2015 09:20:25 UTC+11, Amos S wrote: >>> Thanks David, >>> >>> It seems that the "verify: false" setting is specific to the "fs" type and >>> not recognised by the "s3" type. >>> I tried it anyway and got the same worrying "type is missing" error: >>> >>> $ curl -XPUT 'http://localhost:9200/_snapshot/amos0 >>> <http://localhost:9200/_snapshot/amos0>' -d '{ >>> "s3dev0": { >>> "settings": { >>> "base_path": "elasticsearch/dev/snapshots0", >>> "bucket": "prod-es-backup", >>> "region": "ap-southeast-1", >>> "verify": "false" >>> }, >>> "type": "s3" >>> } >>> } >>> ' >>> {"error":"ActionRequestValidationException[Validation Failed: 1: type is >>> missing;]","status":400} >>> >>> I think I first have to address the "type is missing" issue. I suspect >>> ElasticSearch doesn't recognise the "s3" type. >>> >>> On Tuesday, 27 January 2015 17:13:53 UTC+11, David Pilato wrote: >>> Could you try to set verify to false? >>> >>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_repositories >>> >>> <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_repositories> >>> >>> Not sure if it works but would love to know. >>> >>> -- >>> David ;-) >>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>> >>> Le 27 janv. 2015 à 07:09, Amos S <amos.s...@gmail.com <>> a écrit : >>> >>>> Thanks David, >>>> >>>> That would explain it. >>>> >>>> Is there a way to skip the validation? >>>> >>>> On Tuesday, 27 January 2015 16:52:11 UTC+11, David Pilato wrote: >>>> IIRC when you create a repository we first try to validate it by writing a >>>> sample file in it. >>>> >>>> As you set it to read only, I guess it could be the cause. >>>> >>>> -- >>>> David ;-) >>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>>> >>>> Le 27 janv. 2015 à 06:20, Amos S <amos.s...@gmail.com <>> a écrit : >>>> >>>>> Hello, >>>>> >>>>> For some investigation work, I'm trying to restore specific indices from >>>>> our production ES cluster to a single one-off node. >>>>> >>>>> We run a cluster of ES 1.4.2 on EC2, the data is stored locally on each >>>>> EC2 instance with snapshots stored on an S3 bucket. >>>>> >>>>> I've setup a one-off EC2 instance and am trying to restore a single index >>>>> from snapshot into that new instance. >>>>> >>>>> The instance has its own cluster name and node name, and I've setup a >>>>> read-only S3 role for it so it doesn't accidentally overwrite our backup. >>>>> >>>>> Trying to follow instructions I found in various locations on the web, I >>>>> think the next step for me is to configure the S3 snapshot bucket as a >>>>> repository on the new instance, is that correct? >>>>> >>>>> So I did the following to find the S3 snapshot repository configuration >>>>> in the production environment: >>>>> >>>>> $ curl -XGET http://production-cluster:9200/_snapshot/ >>>>> <http://production-cluster:9200/_snapshot/> | python -mjson.tool >>>>> { >>>>> "s3prod0": { >>>>> "settings": { >>>>> "base_path": "elasticsearch/prod/snapshots0", >>>>> "bucket": "prod-es-backup", >>>>> "region": "ap-southeast-1" >>>>> }, >>>>> "type": "s3" >>>>> } >>>>> } >>>>> >>>>> I then tried to feed this into my new node's configuration: >>>>> >>>>> $ curl -XPUT 'http://localhost:9200/_snapshot/amos0' >>>>> <http://localhost:9200/_snapshot/amos0'> -d ' >>>>> { >>>>> "type": "s3", >>>>> "s3prod0": { >>>>> "settings": { >>>>> "base_path": "elasticsearch/prod/snapshots0", >>>>> "bucket": "prod-es-backup", >>>>> "region": "ap-southeast-1" >>>>> } >>>>> } >>>>> } >>>>> ' >>>>> {"error":"RepositoryException[[amos0] failed to create repository]; >>>>> nested: CreationException[Guice creation errors:\n\n1) Error injecting >>>>> constructor, org.elasticsearch.repositories.RepositoryException: [amos0] >>>>> No bucket defined for s3 gateway\n at >>>>> org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n >>>>> while locating org.elasticsearch.repositories.s3.S3Repository\n while >>>>> locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: >>>>> RepositoryException[[amos0] No bucket defined for s3 gateway]; >>>>> ","status":500} >>>>> >>>>> What am I missing here? >>>>> >>>>> Thanks. >>>>> >>>>> --Amos >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to elasticsearc...@googlegroups.com <>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/e3576b32-02c2-4e05-92de-2a1e5de285d2%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/elasticsearch/e3576b32-02c2-4e05-92de-2a1e5de285d2%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to elasticsearc...@googlegroups.com <>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/facc1f6f-1626-47be-bd18-28d02a76380e%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/elasticsearch/facc1f6f-1626-47be-bd18-28d02a76380e%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to elasticsearc...@googlegroups.com <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/40cc5b7f-5d3b-4abf-9473-9a77f8bdf215%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/40cc5b7f-5d3b-4abf-9473-9a77f8bdf215%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearch+unsubscr...@googlegroups.com >> <mailto:elasticsearch+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/68e2bc5c-3417-4116-a4bc-f8fb7f9a178e%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/68e2bc5c-3417-4116-a4bc-f8fb7f9a178e%40googlegroups.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/400FE638-485D-4EB5-A4FB-A5C57DA3CD15%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
