[
https://issues.apache.org/jira/browse/KUDU-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884277#comment-17884277
]
Bakai Ádám commented on KUDU-3605:
----------------------------------
I had to recreate my dev environment, and the same test now fails with a
different error:
{code:java}
There was 1 failure:
1)
testExternallyProvidedSubjectRefreshedExternally(org.apache.kudu.client.TestSecurity)
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout:
KuduRpc(method=ListTabletServers, tablet=null, attempt=24,
TimeoutTracker(timeout=30000, elapsed=28366), Traces: [0ms] refreshing cache
from master, [30ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.62:46867, [105ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.1.11.60:40051, [107ms] Sub RPC ConnectToMaster: sending RPC
to server master-127.1.11.61:46759, [144ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.62:46867: Network error: Failed to connect
to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [148ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [431ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [436ms] delaying RPC due to: Service
unavailable: Master config
(127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759, [455ms] refreshing cache from master, [456ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [458ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[460ms] Sub RPC ConnectToMaster: received response from server
master-127.1.11.62:46867: Network error: Failed to connect to peer
master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [460ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [463ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [465ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [466ms] delaying RPC due to: Service
unavailable: Master config
(127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759, [475ms] refreshing cache from master, [476ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [477ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[477ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [480ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [481ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [482ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.62:46867: Network error: Failed to connect
to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [482ms] delaying RPC due to: Service unavailable: Master
config (127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867, [495ms] refreshing cache from master, [496ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [497ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[497ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [500ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [500ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.62:46867: Network error: Failed to connect to peer
master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [502ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [502ms] delaying RPC due to: Service
unavailable: Master config
(127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867, [515ms] refreshing cache from master, [515ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [516ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[517ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [520ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [520ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [521ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.62:46867: Network error: Failed to connect
to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [523ms] delaying RPC due to: Service unavailable: Master
config (127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867, [555ms] refreshing cache from master, [556ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [556ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[557ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [560ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [561ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.62:46867: Network error: Failed to connect to peer
master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [562ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [563ms] delaying RPC due to: Service
unavailable: Master config
(127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867, [595ms] refreshing cache from master, [595ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [596ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[597ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [599ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.62:46867: Network error: Failed to connect to peer
master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [599ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [601ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [601ms] delaying RPC due to: Service
unavailable: Master config
(127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759, [635ms] refreshing cache from master, [635ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [637ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[637ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [639ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [640ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [641ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.62:46867: Network error: Failed to connect
to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [642ms] delaying RPC due to: Service unavailable: Master
config (127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867, [895ms] refreshing cache from master, [895ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [896ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[897ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [899ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.62:46867: Network error: Failed to connect to peer
master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [900ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [900ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [902ms] delaying RPC due to: Service
unavailable: Master config
(127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759, [995ms] refreshing cache from master, [995ms] Sub
RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867, [996ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.60:40051,
[997ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.61:46759, [1000ms] Sub RPC ConnectToMaster: received response
from server master-127.1.11.61:46759: Network error: Failed to connect to peer
master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [1000ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [1001ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.62:46867: Network error: Failed to connect
to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [1003ms] delaying RPC due to: Service unavailable: Master
config (127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867, [1495ms] refreshing cache from master, [1495ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867,
[1496ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.60:40051, [1497ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.1.11.61:46759, [1499ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.61:46759: Network error: Failed to connect
to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [1500ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [1500ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.62:46867: Network error: Failed to connect
to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [1500ms] delaying RPC due to: Service unavailable: Master
config (127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867, [1695ms] refreshing cache from master, [1695ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867,
[1696ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.60:40051, [1697ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.1.11.61:46759, [1699ms] Sub RPC ConnectToMaster: received
response from server master-127.1.11.61:46759: Network error: Failed to connect
to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection refused:
/127.1.11.61:46759, [1699ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.62:46867: Network error: Failed to connect to peer
master-127.1.11.62:46867(127.1.11.62:46867): Connection refused:
/127.1.11.62:46867, [1700ms] Sub RPC ConnectToMaster: received response from
server master-127.1.11.60:40051: OK, [1701ms] delaying RPC due to: Service
unavailable: Master config
(127.1.11.62:46867,127.1.11.60:40051,127.1.11.61:46759) has no leader.
Exceptions received: org.apache.kudu.client.RecoverableException: Failed to
connect to peer master-127.1.11.62:46867(127.1.11.62:46867): Connection
refused: /127.1.11.62:46867,org.apache.kudu.client.RecoverableException: Failed
to connect to peer master-127.1.11.61:46759(127.1.11.61:46759): Connection
refused: /127.1.11.61:46759, [4655ms] refreshing cache from master, [4656ms]
Sub RPC ConnectToMaster: sending RPC to server master-127.1.11.62:46867,
[4656ms] Sub RPC ConnectToMaster: sending RPC to server
master-127.1.11.60:40051, [4657ms] Sub RPC ConnectToMaster: sending RPC to
server master-127.1.11.61:46759, [4661ms] too many traces: truncated at 100
traces, [4661ms] too many traces: truncated at 100 traces,
deferred=Deferred@1563905897(state=PENDING, result=null, callback=(continuation
of Deferred@247697810 after retry RPC after error@435860221) -> (continuation
of Deferred@1219186591 after retry RPC after error@796925412) -> (continuation
of Deferred@1037911687 after retry RPC after error@1912366747) -> (continuation
of Deferred@1673074828 after retry RPC after error@1283016226) -> (continuation
of Deferred@466077780 after retry RPC after error@152939785) -> (continuation
of Deferred@583776734 after retry RPC after error@612738035) -> (continuation
of Deferred@27609695 after retry RPC after error@1706347143) -> (continuation
of Deferred@449747622 after retry RPC after error@324872247) -> (continuation
of Deferred@939622321 after retry RPC after error@2074241732) -> (continuation
of Deferred@358077628 after retry RPC after error@1865501361) -> (continuation
of Deferred@1560173585 after retry RPC after error@878901504) -> (continuation
of Deferred@532359428 after retry RPC after error@2059050112) -> (continuation
of Deferred@1418348818 after retry RPC after error@2005074042) -> (continuation
of Deferred@825685127 after retry RPC after error@524693693) -> (continuation
of Deferred@419571870 after retry RPC after error@1995270932) -> (continuation
of Deferred@1895377794 after retry RPC after error@1578385144) -> (continuation
of Deferred@1457171324 after retry RPC after error@1274204024) -> (continuation
of Deferred@1928222911 after retry RPC after error@233987044) -> (continuation
of Deferred@242258029 after retry RPC after error@1394874324) -> (continuation
of Deferred@1401915055 after retry RPC after error@1279121985) -> (continuation
of Deferred@1178912851 after retry RPC after error@892202488) -> (continuation
of Deferred@448315148 after retry RPC after error@947229128) -> (continuation
of Deferred@1477693637 after retry RPC after error@42705938),
errback=(continuation of Deferred@247697810 after retry RPC after
error@435860221) -> (continuation of Deferred@1219186591 after retry RPC after
error@796925412) -> (continuation of Deferred@1037911687 after retry RPC after
error@1912366747) -> (continuation of Deferred@1673074828 after retry RPC after
error@1283016226) -> (continuation of Deferred@466077780 after retry RPC after
error@152939785) -> (continuation of Deferred@583776734 after retry RPC after
error@612738035) -> (continuation of Deferred@27609695 after retry RPC after
error@1706347143) -> (continuation of Deferred@449747622 after retry RPC after
error@324872247) -> (continuation of Deferred@939622321 after retry RPC after
error@2074241732) -> (continuation of Deferred@358077628 after retry RPC after
error@1865501361) -> (continuation of Deferred@1560173585 after retry RPC after
error@878901504) -> (continuation of Deferred@532359428 after retry RPC after
error@2059050112) -> (continuation of Deferred@1418348818 after retry RPC after
error@2005074042) -> (continuation of Deferred@825685127 after retry RPC after
error@524693693) -> (continuation of Deferred@419571870 after retry RPC after
error@1995270932) -> (continuation of Deferred@1895377794 after retry RPC after
error@1578385144) -> (continuation of Deferred@1457171324 after retry RPC after
error@1274204024) -> (continuation of Deferred@1928222911 after retry RPC after
error@233987044) -> (continuation of Deferred@242258029 after retry RPC after
error@1394874324) -> (continuation of Deferred@1401915055 after retry RPC after
error@1279121985) -> (continuation of Deferred@1178912851 after retry RPC after
error@892202488) -> (continuation of Deferred@448315148 after retry RPC after
error@947229128) -> (continuation of Deferred@1477693637 after retry RPC after
error@42705938)))
at
org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at
org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at
org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:158)
at
org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:454)
... 11 trimmed
Suppressed: org.apache.kudu.client.KuduException$OriginalException:
Original asynchronous stack trace
at
org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at
org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at
org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
{code}
It looks like it just can not find the leader but according to the log lines
before, there is a leader.
Here is the test run:
http://dist-test.cloudera.org/job?job_id=root.1727182047.420135
> org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally
> is flaky
> ---------------------------------------------------------------------------------------------
>
> Key: KUDU-3605
> URL: https://issues.apache.org/jira/browse/KUDU-3605
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Bakai Ádám
> Assignee: Bakai Ádám
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
