[ https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842130#comment-16842130 ]
Sam Tunnicliffe commented on CASSANDRA-15041: --------------------------------------------- Hey [~eperott]. I agree that leaking the details about the failure back to the user and spamming the logs are not really desirable and probably quite confusing, so it would be great to fix them both. I'm not totally convinced about changing the response type as an auth failure signifies incorrect credentials or insufficient privileges and so a client should not assume that retrying the request is a valid strategy. On the other hand, a server error indicates an issue outside the client's control and suggests that if the same request is sent again, it may be successful. We could definitely improve the text of the error response though to give a nicer message when this does occur. {quote}4.0: TC2 fail {quote} This appears to be due to a conflict in the Caffeine cache (earlier branches use Guava) between {{refreshAfterWrite}} and {{expireAfterWrite}}. Basically. the refresh keeps trying and failing, which prevents the expiration happening. I haven't been through the changelog to see exactly what the issue was, but upgrading to Caffeine 2.7.0 seems to fix this. > UncheckedExecutionException if authentication/authorization query fails > ----------------------------------------------------------------------- > > Key: CASSANDRA-15041 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15041 > Project: Cassandra > Issue Type: Bug > Components: Feature/Authorization > Reporter: Per Otterström > Priority: Normal > > If cache update for permissions/credentials/roles fails with > UnavailableException this comes back to client as UncheckedExecutionException. > Stack trace on server side: > {noformat} > ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537 > ErrorMessage.java:384 - Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level QUORUM > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) > [apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) > [apache-cassandra-3.11.4.jar:3.11.4] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_181] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > [apache-cassandra-3.11.4.jar:3.11.4] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) > [apache-cassandra-3.11.4.jar:3.11.4] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] > Caused by: com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level QUORUM > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > ... 26 common frames omitted > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level QUORUM > at > org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:283) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > ... 40 common frames omitted > Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot > achieve consistency level QUORUM > at > org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508) > ~[apache-cassandra-3.11.4.jar:3.11.4] > ... 47 common frames omitted > {noformat} > Also, if {{x_validity_in_ms}} > {{x_update_interval_in_ms}}, then the > background update thread will fail in a similar way: > {noformat} > ERROR [PermissionsCacheRefresh:1] 2019-03-04 16:30:43,541 > CassandraDaemon.java:228 - Exception in thread > Thread[PermissionsCacheRefresh:1,5,main] > java.lang.RuntimeException: > org.apache.cassandra.exceptions.UnavailableException: Cannot achieve > consistency level QUORUM > at > org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraRoleManager.isSuper(CassandraRoleManager.java:307) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.AuthCache$1.lambda$reload$0(AuthCache.java:180) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_181] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[na:1.8.0_181] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_181] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) > [apache-cassandra-3.11.4.jar:3.11.4] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181] > Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot > achieve consistency level QUORUM > at > org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526) > ~[apache-cassandra-3.11.4.jar:3.11.4] > at > org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508) > ~[apache-cassandra-3.11.4.jar:3.11.4] > ... 11 common frames omitted > {noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org