[
https://issues.apache.org/jira/browse/SOLR-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324726#comment-14324726
]
Shalin Shekhar Mangar commented on SOLR-7118:
---------------------------------------------
Okay, here's what's happening at
http://jenkins.thetaphi.de/job/Lucene-Solr-5.0-Linux/151/consoleText
# collection has numShards=1 and replicationFactor=2
# collection1 has stateFormat=2 so CloudSolrClient will cache cluster state and
update when the server returns a stale state error
# ChaosMonkey kills the leader when state version = 7
# The leader publishes itself as down and unsets the leader flag. Overseer
writes new state(s) and new state version = 9
# A Query request returns stale state error. CloudSolrClient evicts cached
state and retries after refreshing+caching the state from ZK
{code}
[junit4] 2> 1646950 T8921
oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection
collection1 failed due to (510)
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://127.0.0.1:38608/collection1: STATE STALE:
collection1:7valid : false, retry? 0
[junit4] 2> 1646951 T8921
oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to
collection(s) collection1 after stale state error from server.
[junit4] 2> 1646954 T8903 C3533 P38608 oasc.SolrCore.execute [collection1]
webapp= path=/select
params={q=to+come&_stateVer_=collection1:9&wt=javabin&version=2} hits=0
status=0 QTime=1
{code}
# An indexing request comes in but there is no leader in the cached state.
Every subsequent indexing request will continue to fail until the cache entry
expires i.e. 60 seconds
{code}
[junit4] 2> 1647581 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection
collection1 failed due to (510) org.apache.solr.common.SolrException: Not
enough nodes to handle the request, retry? 0
[junit4] 2> 1647582 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to
collection(s) collection1 after stale state error from server.
[junit4] 2> 1647583 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection
collection1 failed due to (510) org.apache.solr.common.SolrException: Not
enough nodes to handle the request, retry? 1
[junit4] 2> 1647583 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to
collection(s) collection1 after stale state error from server.
[junit4] 2> 1647584 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection
collection1 failed due to (510) org.apache.solr.common.SolrException: Not
enough nodes to handle the request, retry? 2
[junit4] 2> 1647584 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to
collection(s) collection1 after stale state error from server.
[junit4] 2> 1647585 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection
collection1 failed due to (510) org.apache.solr.common.SolrException: Not
enough nodes to handle the request, retry? 3
[junit4] 2> 1647585 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to
collection(s) collection1 after stale state error from server.
[junit4] 2> 1647586 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection
collection1 failed due to (510) org.apache.solr.common.SolrException: Not
enough nodes to handle the request, retry? 4
[junit4] 2> 1647587 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to
collection(s) collection1 after stale state error from server.
[junit4] 2> 1647588 T8920
oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection
collection1 failed due to (510) org.apache.solr.common.SolrException: Not
enough nodes to handle the request, retry? 5
[junit4] 2> REQUEST FAILED for id=0-107
[junit4] 2> org.apache.solr.client.solrj.SolrServerException:
org.apache.solr.common.SolrException: Not enough nodes to handle the request
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:873)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:738)
[junit4] 2> at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
[junit4] 2> at
org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:174)
[junit4] 2> at
org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158)
[junit4] 2> at
org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103)
[junit4] 2> Caused by: org.apache.solr.common.SolrException: Not enough
nodes to handle the request
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1002)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:795)
[junit4] 2> ... 10 more
[junit4] 2> ROOT CAUSE for id=0-107
[junit4] 2> org.apache.solr.common.SolrException: Not enough nodes to
handle the request
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1002)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:795)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866)
[junit4] 2> at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:738)
[junit4] 2> at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
[junit4] 2> at
org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:174)
[junit4] 2> at
org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158)
[junit4] 2> at
org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103)
{code}
> ChaosMonkeyNothingIsSafeTest fails with too many update fails
> -------------------------------------------------------------
>
> Key: SOLR-7118
> URL: https://issues.apache.org/jira/browse/SOLR-7118
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud, Tests
> Affects Versions: 5.0
> Reporter: Shalin Shekhar Mangar
> Fix For: Trunk, 5.1
>
> Attachments: SOLR-7118.patch
>
>
> There are frequent failures on both trunk and branch_5x with the following
> message:
> {code}
> java.lang.AssertionError: There were too many update fails - we expect it can
> happen, but shouldn't easily
> at
> __randomizedtesting.SeedInfo.seed([786DB0FD42626C16:F98B3EE5353D0C2A]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertFalse(Assert.java:68)
> at
> org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:224)
> at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:878)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
