Folks, While running load tests on Aurora I see the following. What does it mean? This <https://pastebin.com/cy0CXgMR> is the new code I added (we discussed this a few weeks ago) to decline incoming update API calls when Aurora is slowing down(not triggered in this case as this is a get call). The following exceptions were seen earlier w/o my code change and appear unrelated to the change. I also saw these during previous outages and want to understand what is going on.
Mohit. ------- Exception ----- Dec 20 16:57:46 node64 aurora-scheduler[78639]: W1220 16:57:46.061 [qtp337673094-129204, LoggingInterceptor:106] Uncaught exception while handling getJobUpdateDetails(null, JobUpdateQuery(role:test-uber-xxx-2, key:null, jobKey:null, user:null, updateStatuses:[], offset:0, limit:0)) org.apache.ibatis.exceptions.PersistenceException: Dec 20 16:57:46 node64 aurora-scheduler[78639]: ### Error rolling back transaction. Cause: java.sql.SQLException: Error accessing PooledConnection. Connection is invalid. Dec 20 16:57:46 node64 aurora-scheduler[78639]: ### Cause: java.sql.SQLException: Error accessing PooledConnection. Connection is invalid. Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.ibatis.session.defaults.DefaultSqlSession.rollback(DefaultSqlSession.java:244) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.ibatis.session.SqlSessionManager.rollback(SqlSessionManager.java:315) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:116) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.storage.log.LogStorage.read(LogStorage.java:562) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:113) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobUpdateDetails(ReadOnlySchedulerImpl.java:327) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobUpdateDetails(SchedulerThriftInterface.java:1133) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.ThrottlingInterceptor.invoke(ThrottlingInterceptor.java:122) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:68) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:98) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobUpdateDetails.getResult(ReadOnlyScheduler.java:1156) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobUpdateDetails.getResult(ReadOnlyScheduler.java:1141) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.http.api.TContentAwareServlet.doPost(TContentAwareServlet.java:154) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ------- AopModule now includes ThrottlingInterceptor <https://pastebin.com/cy0CXgMR> at the end ----- /** * Binding module for AOP-style decorations of the thrift API. */ public class AopModule extends AbstractModule { private static final Matcher<? super Class<?>> THRIFT_IFACE_MATCHER = Matchers.subclassesOf(AnnotatedAuroraAdmin.class) .and(Matchers.annotatedWith(DecoratedThrift.class)); @Override protected void configure() { // Layer ordering: // APIVersion -> Log -> StatsExporter -> SchedulerThriftInterface // It's important for this interceptor to be registered first to ensure it's at the 'top' of // the stack and the standard message is always applied. bindThriftDecorator(new ServerInfoInterceptor()); bindThriftDecorator(new LoggingInterceptor()); bindThriftDecorator(new ThriftStatsExporterInterceptor()); bindThriftDecorator(new ThrottlingInterceptor()); } private void bindThriftDecorator(MethodInterceptor interceptor) { bindThriftDecorator(binder(), THRIFT_IFACE_MATCHER, interceptor); } @VisibleForTesting static void bindThriftDecorator( Binder binder, Matcher<? super Class<?>> classMatcher, MethodInterceptor interceptor) { binder.bindInterceptor( classMatcher, Matchers.returns(Matchers.subclassesOf(Response.class)), interceptor); binder.requestInjection(interceptor); } }
