[
https://issues.apache.org/jira/browse/CASSANDRA-20423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17933647#comment-17933647
]
Dmitry Konstantinov commented on CASSANDRA-20423:
-------------------------------------------------
Root cause analysis:
* The issue is not reproduced locally, so I have analysed the related source
code to find a cause.
* There is only one place in source code which throws an exception with "You
do not have access to this datacenter" text. It is
org.apache.cassandra.service.ClientState#validateLogin
* It throws the exception when the following check returns false:
{code:java}
networkPermissionsCache.get(this.getPrimaryRole()).canAccess(Datacenters.thisDatacenter())
{code}
networkPermissionsCache values are populated by INetworkAuthorizer
implementation
* According to the test code we use
org.apache.cassandra.auth.AuthTestUtils.LocalCassandraNetworkAuthorizer
* org.apache.cassandra.auth.DCPermissions#canAccess is implemented by 3
implementations:
** org.apache.cassandra.auth.DCPermissions#ALL - always return true, so we
cannot get false result here
** org.apache.cassandra.auth.DCPermissions.SubsetPermissions - we do not
configure any explicit permissions, so
org.apache.cassandra.auth.DCPermissions.Builder#build should return all(), so
we cannot get false result here
** org.apache.cassandra.auth.DCPermissions#NONE - it is used in
org.apache.cassandra.auth.CassandraNetworkAuthorizer#authorize:
{code:java}
public DCPermissions authorize(RoleResource role)
{
if (!Roles.canLogin(role))
{
return DCPermissions.none();
}
{code}
so, it looks like our case - if a role does not have a login flag set we can
return such error.
* The failing test sets the flag but we retrieve the flag from a cache
(networkPermissionsCache)
* The test methods shares a common Cassandra instance and they use the same
user across them, f we check other test methods we can find that
org.apache.cassandra.auth.GrantAndRevokeTest#testSpecificGrantsOnSystemKeyspaces
{code:java}
executeNet("CREATE ROLE '" + user + "'");
{code}
without LOGIN = true flag.
* So, we may have a case when a role is updated by
testSpecificGrantsOnSystemKeyspaces, then dropped by tearDown() method
{code:java}
executeNet("DROP ROLE " + user);
{code}
but roles cache is not updated yet, so when another test method is executing -
it can uses an old version of the role without LOGIN = true flag and get the
error.
* let's check the hypothesis:
** increase roles validity from default 2 sec to 10 seconds
** run junit tests in a specific order to establish a CQL connection first
with a valid user, then fill the role cache with such user name but without
login grants and finally execute our failing test:
{code:java}
org.apache.cassandra.auth.GrantAndRevokeTest,testGrantedAllTables
org.apache.cassandra.auth.GrantAndRevokeTest,testSpecificGrantsOnSystemKeyspaces
org.apache.cassandra.auth.GrantAndRevokeTest,testGrantOnAllKeyspaces
{code}
and evict caches in testSpecificGrantsOnSystemKeyspaces to force the required
invalid content loading:
{code:java}
@Test
public void testSpecificGrantsOnSystemKeyspaces() throws Throwable
{
Roles.cache.invalidate();
AuthenticatedUser.networkPermissionsCache.invalidate();
{code}
* by applying all these tricks I was able get the failed test on my laptop
quite frequently:
{code}
java.lang.AssertionError: Expected error message to contain 'User user has no
MODIFY permission on <table user_keyspace.t1> or any of its parents', but got
'You do not have access to this datacenter (datacenter1)'
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.cassandra.cql3.CQLTester.assertMessageContains(CQLTester.java:2472)
at
org.apache.cassandra.cql3.CQLTester.assertInvalidThrowMessage(CQLTester.java:2411)
at
org.apache.cassandra.cql3.CQLTester.assertUnauthorizedQuery(CQLTester.java:2484)
at
org.apache.cassandra.auth.GrantAndRevokeTest.testGrantOnAllKeyspaces(GrantAndRevokeTest.java:543)
{code}
-----
There are several options to fix the issue:
# invalidate all security caches between runs using
AuthCacheService.instance.invalidateCaches();
# use unique user names for each test method
# do a polling check to wait for a cache refresh as we had added in
CASSANDRA-17173 (note: as of now we do not have it in testGrantOnAllKeyspaces
method added recently in CASSANDRA-20090)
> Test failure: org.apache.cassandra.auth.GrantAndRevokeTest
> ----------------------------------------------------------
>
> Key: CASSANDRA-20423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20423
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Test/unit
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 5.x
>
> Attachments: Cassandra-trunk_2058_GrantAndRevokeTest.log
>
>
> Sometimes GrantAndRevokeTest fails with:
> {code:java}
> junit.framework.AssertionFailedError: Expected error message to contain 'User
> user has no MODIFY permission on <table user_keyspace.t1> or any of its
> parents', but got 'You do not have access to this datacenter (datacenter1)'
> at
> org.apache.cassandra.cql3.CQLTester.assertMessageContains(CQLTester.java:2472)
> at
> org.apache.cassandra.cql3.CQLTester.assertInvalidThrowMessage(CQLTester.java:2411)
> at
> org.apache.cassandra.cql3.CQLTester.assertUnauthorizedQuery(CQLTester.java:2484)
> at
> org.apache.cassandra.auth.GrantAndRevokeTest.testGrantOnAllKeyspaces(GrantAndRevokeTest.java:537)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
> log: [^Cassandra-trunk_2058_GrantAndRevokeTest.log]
> A recent example for 4.1:
> https://ci-cassandra.apache.org/view/Cassandra%204.1/job/Cassandra-4.1/556/testReport/junit/org.apache.cassandra.auth/GrantAndRevokeTest/testGrantOnAllKeyspaces/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
