[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873909#comment-16873909
]
Per Otterström commented on CASSANDRA-15041:
Thanks for taking the time to review.
bq. Whilst the new dtests for handling unavailability do what they claim,
they’re not doing so in the way one might expect.
Yeah, had to spend some time to get this sorted in my head. There are certainly
a few surprises in how the caches affect the result, and one another. What's
more, behavior is a bit different between versions since we're caching the
super-user flag in 4.0, while in pre-4.0 we're not. For this reason, a stale
entry in the permissions cache on pre-4.0 will result in a query to pull up the
role's super-user flag even if the role cache is still up to date. The
resulting error message is a bit confusing, but this is more intuitive in 4.0 I
think. Also, I believe this makes it hard to use a non-super-user role in the
tests and still get clean code and consistent results across versions, so
didn't change that. Open to suggestions if you have some ideas...
I've added some comments around assumptions and test strategy in the dtests.
One thing I discovered while working on this is that the assert_exception
helper (and friends) don't verify error message properly. I fixed this locally
while verifying, but will create a separate ticket for this since it will
affect a few places where the helpers are used.
bq. So I just have these nits
Fixed those. Also, since I removed the stack trace dump on error level when
background updates fail (pre-4.0), I've re-added the trace log we used to have
in the AuthCache.
bq. I’ve been running the tests with the HIRES circle configuration
Thanks. Tried to make it work on free service, but I give up...
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Assignee: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868818#comment-16868818
]
Sam Tunnicliffe commented on CASSANDRA-15041:
-
Thanks for this [~eperott], there are definitely valuable improvements here.
Whilst the new dtests for handling unavailability do what they claim, they’re
not doing so in the way one *might* expect. First of all, because they use a
superuser, table/keyspace level permissions will never be read from system
tables and so won’t be vulnerable to unavailablity. The reason the tests are
currently passing is that the check that the user has access to the local DC is
resulting in a failed read by {{CassandraNetworkAuthorizer}}. So the tests are
definitely still valid as they are, but it might be worth adding some
commentary just to highlight this and maybe use a non-superuser role to
exercise the permissions cache too. Potentially, adding some logging to the new
catch blocks in {{CassandraRoleManager}} might be useful, similar to that in
{{CassandraAuthorizer::authorize}}. {{CassandraNetworkAuthorizer}} is
especially vulnerable here, due to CASSANDRA-15089, which I will commit as soon
as I get chance.
Otherwise, this all looks pretty good to me, modulo a typo in
{{ClientState::canLogin}} (only in 2.2/3.0/3.11).
I’ve been running the tests with the HIRES circle configuration and everything
looks good - all failures are unrelated and/or fixed already by other tickets.
So I just have these nits:
* typo in {{ClientState::canLogin}} (2.2/3.0/3.11)
* maybe add some minimal logging in {{CassandraRoleManager::isSuper/canLogin}}
or make the exception messages distinguishable
* document the potential sources of the availability failures that the dtests
are checking and/or use a non-superuser in tests
One thing that I had missed up to now was that when Guava was replaced with
Caffeine in CASSANDRA-10855, the ability to do async reloading in {{AuthCache}}
was lost. We should definitely fix that, as having those reads done in a thread
servicing client reads is not a good idea. I’ve opened CASSANDRA-15177 for
that.
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Assignee: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866919#comment-16866919
]
Sam Tunnicliffe commented on CASSANDRA-15041:
-
Thanks, I should get finished with this soon. I've been running the branches
through CircleCI with the HIRES settings and the only failures are for
unrelated things (one which has already been fixed in the dtests & one known
issue with Thrift tests on 3.11 and lower).
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Assignee: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_181]
> at
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866829#comment-16866829
]
Per Otterström commented on CASSANDRA-15041:
bq. Also, had a setback with one of the goals of this ticket - to make the
background cache reload thread mute if it fails. Turns out the error message is
buried deep down in the Guava LoadingCache. Only option I see for pre-4.0 is to
mute this in the logback config.
Discovered that this was a false assumption based on comments in the Guava
Cache API. The log message is actually coming from the
{{DebuggableThreadPoolExecutor}} which is invoking the
{{DefaultUncaughtExceptionHandler}} which in turn is installed by
{{CassandraDaemon}}. Added a small patch on top of 2.2, 3.0, and 3.11 branches
to override this behavior. Also update dtest branch to verify that a failing
background update will actually be silent.
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Assignee: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864627#comment-16864627
]
Per Otterström commented on CASSANDRA-15041:
Since dtests on CircleCI are unreliable on the free service, I've run failing
j8_dtests-no-vnodes locally and compared with each base branch. There is no
regression.
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Assignee: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_181]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860306#comment-16860306
]
Per Otterström commented on CASSANDRA-15041:
Had to make some adjustments while implementing this.
When we fail to perform authorization it is not always possible to convert
whatever-exception-we-get into an {{UnavailableException}} since the
{{UnavailableException}} constructor requires a bunch of parameters (CL plus
required and live nodes). I didn't feel comfortable to change this to achieve
our goals here, so I went with the other proposal to convert this into an
{{UnauthorizedException}} instead. But I'm happy to discuss options. Worth
considering, since IAuthorizer is a public plug-in interface it should define a
generic behavior. And, for example, it would be somewhat awkward for an
{{LDAPAuthorizer}} to throw an {{UnavailableException}} if it fails to contact
the LDAP server, so the {{UnauthorizedException}} may be a better fit anyway.
A side effect of signaling {{UnauthorizedException}} instead of
{{UnavailableException}} is that the issue with the stale entries from the
Caffeine cache don'ẗ show any more. This is because the driver will not retry
on {{UnauthrizedException}}, and the Caffeine issue only shows if it get
repeated queries on failing keys. But IMO we should still see to this. I
created CASSANDRA-15153 for this.
Also, had a setback with one of the goals of this ticket - to make the
background cache reload thread mute if it fails. Turns out the error message is
buried deep down in the Guava {{LoadingCache}}. Only option I see for pre-4.0
is to mute this in the logback config.
PR for [dtest|https://github.com/apache/cassandra-dtest/pull/52].
The patches for Cassandra differs a bit on 2.2/3.0 vs. 3.11 vs. trunk. Not sure
what's the best way to provide these patches to simplify review and merge into
upstream repo. Below are links to the individual branches on my github clone
without merge commits, is that OK? Lots if dtests are timing out since I only
have the free service, but will try to run failing tests locally
||Patch||CI||
|[15041-cassandra-2.2|https://github.com/eperott/cassandra/tree/15041-cassandra-2.2]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-cassandra-2.2]|
|[15041-cassandra-3.0|https://github.com/eperott/cassandra/tree/15041-cassandra-3.0]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-cassandra-3.0]|
|[15041-cassandra-3.11|https://github.com/eperott/cassandra/tree/15041-cassandra-3.11]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-cassandra-3.11]|
|[15041-trunk|https://github.com/eperott/cassandra/tree/15041-trunk]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-trunk]|
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Assignee: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850725#comment-16850725
]
Sam Tunnicliffe commented on CASSANDRA-15041:
-
[~eperott] sure, your proposal does sound entirely reasonable and consistent so
let's got for that. I can definitely review when you're ready with a patch.
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_181]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849436#comment-16849436
]
Per Otterström commented on CASSANDRA-15041:
I've been giving this some more thought. My proposal is:
* Indicate {{AuthenticationException}} to client if system_auth is
inaccessible during authentication
This is in line with existing behavior in 2.2, 3.0 and partially 3.11.
* Indicate {{UnavailableException}} to client if system_auth is inaccessible
during authorization
This should wrap whatever {{RequestExecutionException}} that was thrown by the
authorization backend (perhaps this is what you meant to say from the
beginning).
* In both cases above should the error message clearly state that issue is
related to Authenticatoin/Authorization
The underlying cause should also be included in the message as a hint for
troubleshooting.
* Create a separate ticket for upgrading the Caffein dependency.
[~samt], would be great to get your thoughts on this. Also, would you be able
to review this if I work on a patch?
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843347#comment-16843347
]
Per Otterström commented on CASSANDRA-15041:
{quote}I'm not totally convinced about changing the response type as an auth
failure signifies incorrect credentials or insufficient privileges and so a
client should not assume that retrying the request is a valid strategy. On the
other hand, a server error indicates an issue outside the client's control and
suggests that if the same request is sent again, it may be successful.
{quote}
True. If we were to expose the internal RequestExecutionException, then clients
might retry and succeed. Less headache for operators.
But this all depends on the retry strategy of the client/driver and whether the
actual client request is idempotent. This could also result in a
ReadTimeoutException when the client actually is doing an INSERT, which might
add to the confusion.
About authentication, both 2.2 and 3.0 convert the internal
RequestExecutionException to an AuthenticationException. 3.11 will do the same
when the credentials cache is enabled, but otherwise it will not. 4.0 will
never convert to AuthenticationException. The behavior in 2.2/3.0 seem quite
deliberate.
IMO we should stick with the legacy behavior during authentication and update
3.11/4.0 accordingly. I'm not sure about authorization behavior -
RequestExecutionException seem more practical, while an UnauthorizedException
seem more correct.
I've added a few more
[dtests|https://github.com/apache/cassandra-dtest/compare/master...eperott:cassandra-15041]
to cover for authentication as well as valid cases for responding from cached
entries. (ATM tests expect AuthenticationException during authentication, and
UnavailableException during authorization.)
{quote}We could definitely improve the text of the error response though to
give a nicer message when this does occur.
{quote}
Right, so I've incorporated my suggestions in the dtests. The idea is to give
an indication of why the request was rejected, and give a hint of the cause for
this.
{quote}but upgrading to Caffeine 2.7.0 seems to fix this
{quote}
Indeed it does! For better traceability, and considering that the caffeine
library is used on many places in the code base, I suggest to follow up on this
issue in a separate ticket.
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Priority: Normal
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842130#comment-16842130
]
Sam Tunnicliffe commented on CASSANDRA-15041:
-
Hey [~eperott].
I agree that leaking the details about the failure back to the user and
spamming the logs are not really desirable and probably quite confusing, so it
would be great to fix them both. I'm not totally convinced about changing the
response type as an auth failure signifies incorrect credentials or
insufficient privileges and so a client should not assume that retrying the
request is a valid strategy. On the other hand, a server error indicates an
issue outside the client's control and suggests that if the same request is
sent again, it may be successful. We could definitely improve the text of the
error response though to give a nicer message when this does occur.
{quote}4.0: TC2 fail
{quote}
This appears to be due to a conflict in the Caffeine cache (earlier branches
use Guava) between {{refreshAfterWrite}} and {{expireAfterWrite}}. Basically.
the refresh keeps trying and failing, which prevents the expiration happening.
I haven't been through the changelog to see exactly what the issue was, but
upgrading to Caffeine 2.7.0 seems to fix this.
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Priority: Normal
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.4.jar:3.11.4]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if authentication/authorization query fails
[
https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841705#comment-16841705
]
Per Otterström commented on CASSANDRA-15041:
So, this is mostly a cosmetic issue in the sense that there is no way to
resolve the underlying problem - that not enough replicas are available to read
from the system_auth tables. Still, this generates error messages that cause
operators to jump, and Cassandra isn't behaving well towards clients when this
happens.
I've updated the auth_test.py to use consistent cache settings and created a
few new test cases to reproduce this issue in the scenario where not enough
replicas are available. They revile unwanted behavior with small variation on
different releases. In short:
* TC1 will trigger Cassandra to perform a background update of a cached
credentials/roles/permissions entries.
* TC2 will trigger authorization when cached entries have passed both
update-interval and validity (blocking update)
* TC3 will trigger authorization when cache is disabled.
Link to dtest
[patch|https://github.com/apache/cassandra-dtest/compare/master...eperott:cassandra-15041].
I'm expecting Cassandra to fail gracefully on TC1, possibly with a warning, but
no stack trace.
I'm expecting TC2 and TC3 to reject the request with exception to indicate
not-authorized|unavailable|timeout, not sure which makes most sense. In any
case, TC2 and TC3 should fail the same way and there should be no errors or
stack traces in the log.
Results on different releases:
* 4.0: TC2 fail
* 3.11: TC1, TC2, TC3 and existing test_login fail
* 3.0: TC1, TC2 and TC3 fail
* 2.2: TC1, TC2 and TC3 fail
4.0 behaves generally better since a similar ticket as this one was fixed in
CASSANDRA-13113. The reason TC2 fails is that the request actually will be
authorized, even though the cached entries should have timed out. From what I
can tell the cache is handing out stale entries.
The reason test_login fail on 3.11 branch is that we're caching credentials
since 3.4.
All new test cases fail on 3.11, 3.0 and 2.2 as reported in this ticket.
So far I've made no attempt to work on a fix for this. Before we dive into
that, I'd like to get some feedback on the dtests and my findings above.
TC2 and TC3 currently expect an UnavailableException and a message similar to
"Cannot achieve consistency level QUORUM" at the client side, simply because
this is the behavior in 4.0 branch. I feel this might confuse users a bit as
the exception and message is related to the internal lookup on the
system_auth.* table, rather than the actual query sent to the cluster. Would it
make more sense to throw back an UnauthorizedException?
[~beobal] and [~ifesdjeen], you were both much involved in CASSANDRA-13113.
What are your thoughts on this?
> UncheckedExecutionException if authentication/authorization query fails
> ---
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Per Otterström
>Priority: Normal
>
> If cache update for permissions/credentials/roles fails with
> UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537
> ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot achieve
> consistency level QUORUM
> at
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:439)
> ~[apache-cassandra-3.11.4.jar:3.11.4]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368)
>
