[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852425#comment-16852425
]
Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:40 PM:
---------------------------------------------------------------
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
<backup>
<repository name="S3"
class="org.apache.solr.core.backup.repository.HdfsBackupRepository"
default="false">
<str name="location">s3a://my-bucket/solr_811_backups</str>
<str name="solr.hdfs.home">s3a://my-bucket/solr_811_backups</str>
</repository>
</backup>{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP&name=testS3A&collection=test&repository=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}} {code}
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-object --bucket my-bucket --key solr_811_backups{code}
Output of the backup
{code:java}
aws s3 ls s3:/my-bucket/solr_811_backups/testS3A/
PRE snapshot.shard1/
PRE zk_backup/
2019-05-30 16:31:48 194 backup.properties{code}
was (Author: varunthacker):
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
<backup>
<repository name="S3"
class="org.apache.solr.core.backup.repository.HdfsBackupRepository"
default="false">
<str name="location">s3a://my-bucket/solr_811_backups</str>
<str name="solr.hdfs.home">s3a://my-bucket/solr_811_backups</str>
</repository>
</backup>{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP&name=testS3A&collection=test&repository=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}}{code}
Related code -
[https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-object --bucket my-bucket --key solr_811_backups{code}
Output of the backup
{code:java}
aws s3 ls s3:/my-bucket/solr_811_backups/testS3A/
PRE snapshot.shard1/
PRE zk_backup/
2019-05-30 16:31:48 194 backup.properties{code}
> S3BackupRepository
> ------------------
>
> Key: SOLR-9952
> URL: https://issues.apache.org/jira/browse/SOLR-9952
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Backup/Restore
> Reporter: Mikhail Khludnev
> Priority: Major
> Attachments:
> 0001-SOLR-9952-Added-dependencies-for-hadoop-amazon-integ.patch,
> 0002-SOLR-9952-Added-integration-test-for-checking-backup.patch, Running Solr
> on S3.pdf, core-site.xml.template
>
>
> I'd like to have a backup repository implementation allows to snapshot to AWS
> S3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
