[ https://issues.apache.org/jira/browse/KUDU-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Henke updated KUDU-2549: ------------------------------ Labels: trivial (was: ) > Kudu kinit renewal thread's exponential backoff may need an upper bound > ----------------------------------------------------------------------- > > Key: KUDU-2549 > URL: https://issues.apache.org/jira/browse/KUDU-2549 > Project: Kudu > Issue Type: Improvement > Components: security > Affects Versions: 1.7.0 > Reporter: Michael Ho > Priority: Critical > Labels: trivial > > An Impala instance (which recently adopted the Kudu Kerberos implementation) > happened to run into a temporary DNS outage. The user set up Kerberos to have > a very short Kerberos ticket lifetime (30 minutes). For the couple of hours > in which the DNS was done, the renewal thread quickly racked up many renewal > failure, leading to a very long backoff time (up to *5 hours* eventually). > Even after the DNS has recovered, the Impala process still fails to > communicate with other nodes due to the expired TGT. The renewal thread > didn't wake up in some cases for more than 3 hours after the DNS recovered. > This seems to provide a rather bad user experience so it may be worth > considering having a configurable upper bound on exponential backoff when > ticket renewal fails. At a minimum, may help to log the backoff time to help > diagnose the issue. > {noformat} > W0822 23:15:35.960669 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm '---redacted---' > W0822 23:19:21.016465 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm '---redacted---' > W0822 23:25:48.059895 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm '---redacted---' > W0822 23:38:14.100435 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm '---redacted---' > W0822 23:59:26.152209 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm '---redacted---' > W0823 00:42:28.194363 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm ''---redacted---' > W0823 01:58:41.240950 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm ''---redacted---' > W0823 03:28:54.285295 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm ''---redacted---' > W0823 08:42:57.335754 10964 init.cc:188] Kerberos reacquire error: : Runtime > error: Reacquire error: unable to login from keytab: Cannot contact any KDC > for realm '---redacted---' > I0823 13:58:11.337008 10964 init.cc:283] Successfully reacquired a new > kerberos TGT > I0823 14:08:46.362918 10964 init.cc:283] Successfully reacquired a new > kerberos TGT > {noformat} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)