[
https://issues.apache.org/jira/browse/KUDU-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397522#comment-16397522
]
Todd Lipcon commented on KUDU-2342:
-----------------------------------
Reconstructing the timeline a bit:
- 07:20:54.751998: peer e3fdd8 fell behind the retention and "can never be
caught up"
- 07:20:54.766460: peer f7376c added as a NON_VOTER
- 07:20:55.268965: tablet copy starts to f7376c
- 07:21:34.559736: tablet copy ends
- 07:21:34.779841: logs held by the tablet copy session are GCed
- 07:21:34.790443: the new NON_VOTER peer is already unable to be caught up
because the logs just got GCed (*hmm, interesting*)
- 07:21:34.790797: nevertheless, the leader issues a config change to promote
f7376c to VOTER
Now we have 2/4 VOTER replicas which can never be caught up -- the original bad
one, and the one we just promoted. Hence we can't make progress.
It seems there are two serious issues at play here:
- why did we not retain the logs between the tablet copy session finishing and
catching up the peer? perhaps because the non-voter isn't included in the log
retention calculations and was more than 80 segments behind?
- why did we promote a non-voter that wasn't relatively up to date or in a
"good" state?
> Insert into Lineitem table with 1340 tablets on 129 node cluster failed with
> "Failed to write batch "
> -----------------------------------------------------------------------------------------------------
>
> Key: KUDU-2342
> URL: https://issues.apache.org/jira/browse/KUDU-2342
> Project: Kudu
> Issue Type: Bug
> Components: tablet
> Affects Versions: 1.7.0
> Reporter: Mostafa Mokhtar
> Assignee: Alexey Serbin
> Priority: Critical
> Labels: scalability
> Attachments: Impala query profile.txt, tablet-info.html
>
>
> While loading TPCH 30TB on 129 node cluster via Impala, write operation
> failed with :
> Query Status: Kudu error(s) reported, first error: Timed out: Failed to
> write batch of 38590 ops to tablet b8431200388d486995a4426c88bc06a2 after 1
> attempt(s): Failed to write to server: a260dca5a9c846e99cb621881a7b86b8
> (vc1515.halxg.cloudera.com:7050): Write RPC to X.X.X.X:7050 timed out after
> 180.000s (SENT)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
