----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65268/#review197148 -----------------------------------------------------------
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/TransactionManager.java Line 221 (original), 227 (patched) <https://reviews.apache.org/r/65268/#comment277273> Try execurting transaction. If it fails due to SentryUserException, propagate the exception immediately. Otherwise retry transaction multiple times trying not to exceed the total time limit. sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/TransactionManager.java Line 224 (original), 240 (patched) <https://reviews.apache.org/r/65268/#comment277274> I sould suggest something along the following lines: ` private class ExponentialBackoff { @SuppressWarnings("squid:S00112") <T> T execute(Callable<T> arg) throws Exception { Exception ex = null; int retryNum = 0; // Maximumm total sleep time long sleepTime = maxRetryDurationMills / 32; // Maximum sleep time between retries final long maxSleep = maxRetryDurationMills / 4; final long deadline = System.currentTimeMillis() + maxRetryDurationMills; do { try { return arg.call(); } catch (SentryUserException e) { // throw the sentry exception without retry LOGGER.warn("Transaction manager encountered non-retriable exception", e); throw e; } catch (Exception e) { ex = e; LOGGER.warn("Transaction execution encountered exception", e); long remainingTime = deadline - System.currentTimeMillis(); if (remainingTime <= 0) { break; } if (sleepTime < maxSleep) { // Introduce some randomness in the backoff time. int fuzz = random.nextInt((int) sleepTime / 2); sleepTime *= 3; sleepTime /= 2; sleepTime += fuzz; } sleepTime = Math.min(sleepTime, Math.min(maxSleep, remainingTime)); LOGGER.warn("Retrying transaction {} times, sleeping for {} ms", ++retryNum, sleepTime); retryCount.inc(); Thread.sleep(sleepTime); } } while (true); assert (ex != null); String message = "Retried for " + maxRetryDurationMills + " milliseconds. Giving up " + ex.getMessage(); LOGGER.error(message, ex); throw new Exception(message, ex); } } ` - Alexander Kolbasov On Feb. 7, 2018, 6:04 p.m., kalyan kumar kalvagadda wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65268/ > ----------------------------------------------------------- > > (Updated Feb. 7, 2018, 6:04 p.m.) > > > Review request for sentry, Arjun Mishra, Na Li, Sergio Pena, and Vadim > Spector. > > > Bugs: SENTRY-1904 > https://issues.apache.org/jira/browse/SENTRY-1904 > > > Repository: sentry > > > Description > ------- > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. > > New patch that is attached adds upper bounds on below > > 1.Interval between the retry attempts which increases exponentially. > 2.Total time a transaction could spend in retries. > 3.Removed retry based on count. > > > With out these limits we would not have a control on how long a transaction > could be be active. > > > Diffs > ----- > > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/persistent/TransactionManager.java > f4ff962a67f8a5c23cc5c8daa7bcb861d2e6b6a5 > > sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/ServiceConstants.java > 7e02874b4be6a7109108809b1f404fe971b7b8e2 > > sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestHMSFollowerSentryStoreIntegration.java > 91c90f9d302f4feb3a8b3d06541f43541c87bf0f > > sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/persistent/TestSentryStore.java > b4100278392986c161625a366212c6fef66ec0a9 > > > Diff: https://reviews.apache.org/r/65268/diff/7/ > > > Testing > ------- > > Made sure all the tests pass. > > > Thanks, > > kalyan kumar kalvagadda > >
