[jira] [Updated] (SOLR-11616) Backup failing on a constantly changing index with NoSuchFileException
[
https://issues.apache.org/jira/browse/SOLR-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand updated SOLR-11616:
Fix Version/s: master (8.0)
> Backup failing on a constantly changing index with NoSuchFileException
> --
>
> Key: SOLR-11616
> URL: https://issues.apache.org/jira/browse/SOLR-11616
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
> Fix For: 7.2, master (8.0)
>
> Attachments: SOLR-11616.patch, SOLR-11616.patch, solr-6.3.log,
> solr-7.1.log
>
>
> As reported by several users on SOLR-9120 , Solr backups fail with
> NoSuchFileException on a constantly changing index.
> Users linked SOLR-9120 to the root cause as the stack trace is the same , but
> the fix proposed there won't fix backups to stop failing.
> We need to implement a similar fix in {{SnapShooter#createSnapshot}} to fix
> the problem
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11616) Backup failing on a constantly changing index with NoSuchFileException
[
https://issues.apache.org/jira/browse/SOLR-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-11616:
-
Attachment: SOLR-11616.patch
Patch with a test case which will fail all the time if we don't fold the
SnapShooter changes.
For the test I am invoking the backup-core directly . It makes the BACKUPCORE
API available to standalone solr users. SolrCloud users could always access it .
I think it's not a big deal. It's a much cleaner API and we maybe this pushes
us to implement SOLR-9239 sooner
> Backup failing on a constantly changing index with NoSuchFileException
> --
>
> Key: SOLR-11616
> URL: https://issues.apache.org/jira/browse/SOLR-11616
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
> Attachments: SOLR-11616.patch, SOLR-11616.patch, solr-6.3.log,
> solr-7.1.log
>
>
> As reported by several users on SOLR-9120 , Solr backups fail with
> NoSuchFileException on a constantly changing index.
> Users linked SOLR-9120 to the root cause as the stack trace is the same , but
> the fix proposed there won't fix backups to stop failing.
> We need to implement a similar fix in {{SnapShooter#createSnapshot}} to fix
> the problem
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11616) Backup failing on a constantly changing index with NoSuchFileException
[
https://issues.apache.org/jira/browse/SOLR-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-11616:
-
Attachment: solr-7.1.log
Okay I kept trying and after a few attempts in reproduced on Solr 7.1 as well .
I played around with the soft commit time and reduced it to 100ms and then ran
into the error
{code}
{
responseHeader: {
status: 500,
QTime: 23019
},
failure: {
172.16.0.83:8983_solr:
"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://172.16.0.83:8983/solr: Failed to backup
core=gettingstarted_shard2_replica_n6 because
java.nio.file.NoSuchFileException:
/Users/varunthacker/solr-7.1.0/example/cloud/node1/solr/gettingstarted_shard2_replica_n6/data/index/_i9_Lucene50_0.tip"
},
Operation backup caused exception::
"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
Could not backup all replicas",
exception: {
msg: "Could not backup all replicas",
rspCode: 500
},
error: {
metadata: [
"error-class",
"org.apache.solr.common.SolrException",
"root-error-class",
"org.apache.solr.common.SolrException"
],
msg: "Could not backup all replicas",
trace: "org.apache.solr.common.SolrException: Could not backup all replicas at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:306)
at
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:243)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:221)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745) at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507) at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534) at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.base/java.lang.Thread.run(Thread.java:844) ",
code: 500
}
}
{code}
> Backup failing on a constantly changing index with NoSuchFileException
> --
>
> Key: SOLR-11616
> URL: https://issues.apache.org/jira/browse/SOLR-11616
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
> Attachments: SOLR-11616.patch, solr-6.3.log, solr-7.1.log
>
>
> As reported by several users on SOLR-912
[jira] [Updated] (SOLR-11616) Backup failing on a constantly changing index with NoSuchFileException
[
https://issues.apache.org/jira/browse/SOLR-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-11616:
-
Attachment: solr-6.3.log
I was able to reproduce it very easily on Solr 6.3
Just kept continuous index running with commits and open searchers and then ran
the backup command.
Response from the API call
{code}
500
636
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://172.16.0.83:8983/solr: Failed to backup
core=gettingstarted_shard1_replica1 because java.nio.file.NoSuchFileException:
/Users/varunthacker/Downloads/search-downloads/solr-6.3.0/example/cloud/node1/solr/gettingstarted_shard1_replica1/data/index/segments_32
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
Could not backup all replicas
Could not backup all replicas
500
org.apache.solr.common.SolrException
org.apache.solr.common.SolrException
Could not backup all replicas
org.apache.solr.common.SolrException: Could not backup all
replicas at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:287)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:218)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441) at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518) at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244) at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
500
{code}
I wasn't able to reproduce this on Solr 7.1 . As soon as I issue a backup
command I see updates come to a grinding halt i.e the backup took 2 minutes and
i only saw 4 update requests come in . Before the after the backup docs were
being added every 200 milliseconds. So something was blocking updates during
the backup command and hence the call succeeded? Very strange . I'll dig into
this more
> Backup failing on a constantly changing index with NoSuchFileException
> --
>
> Key: SOLR-11616
> URL: https://issues.apache.org/jira/browse/SOLR-11616
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
> Attachments: SOLR-11616.patch, solr-6.3.log
>
>
> As reported by several users on SOLR-9120 , Solr backups fail with
> NoSuchFileException on a constantly changing index.
> Users linked SOLR-9120 to the root cause as the stack trace is the same , but
> the fix propose
[jira] [Updated] (SOLR-11616) Backup failing on a constantly changing index with NoSuchFileException
[
https://issues.apache.org/jira/browse/SOLR-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-11616:
-
Attachment: SOLR-11616.patch
Completely untested patch copying the solution that was used in SOLR-9120 .
I will test it out later in the week but wanted to put the fix out there
> Backup failing on a constantly changing index with NoSuchFileException
> --
>
> Key: SOLR-11616
> URL: https://issues.apache.org/jira/browse/SOLR-11616
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
> Attachments: SOLR-11616.patch
>
>
> As reported by several users on SOLR-9120 , Solr backups fail with
> NoSuchFileException on a constantly changing index.
> Users linked SOLR-9120 to the root cause as the stack trace is the same , but
> the fix proposed there won't fix backups to stop failing.
> We need to implement a similar fix in {{SnapShooter#createSnapshot}} to fix
> the problem
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
