[
https://issues.apache.org/jira/browse/CASSANDRA-9694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611882#comment-14611882
]
Sam Tunnicliffe commented on CASSANDRA-9694:
--------------------------------------------
Those stacktraces should not happen if the process I described is followed. As
I mentioned, the authenticator & authorizer won't attempt to actually use the
new tables until the old ones are removed, but you can see from the log that
this is happening. Because you wiped the {{system_auth}} keyspace on the single
upgraded node, when it restarted it immediately began attempting to access
them, but as the rest of the cluster (still on 2.1.7) doesn't have them you see
those errors.
I should note that when following the correct procedure, you *will* still see
some stacktraces in the non-upgraded nodes' log files as the upgraded nodes
attempt to perform the data conversion, but these are harmless. They don't
propagate back to clients and they are what inform the upgrading node that the
conversion cannot be completed yet. For reference, these are the sort of errors
to expect:
{noformat}
WARN [MessagingService-Incoming-/127.0.0.1] 2015-07-02 12:15:23,544
IncomingTcpConnection.java:97 - UnknownColumnFamilyException reading from
socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[main/:na]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[main/:na]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[main/:na]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[main/:na]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[main/:na]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[main/:na]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[main/:na]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:188)
~[main/:na]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:170)
~[main/:na]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88)
~[main/:na]
{noformat}
> system_auth not upgraded
> ------------------------
>
> Key: CASSANDRA-9694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9694
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Windows-7-32 bit, 3.2GB RAM, Java 1.7.0_55
> Reporter: Andreas Schnitzerling
> Assignee: Sam Tunnicliffe
> Fix For: 2.2.0 rc2
>
> Attachments: 9694.txt, system_exception.log
>
>
> After upgrading Authorization-Exceptions occur. I checked the system_auth
> keyspace and have seen, that tables users, credentials and permissions were
> not upgraded automatically. I upgraded them (I needed 2 times per table
> because of CASSANDRA-9566). After upgrading the system_auth tables I could
> login via cql using different users.
> {code:title=system.log}
> WARN [Thrift:14] 2015-07-01 11:38:57,748 CassandraAuthorizer.java:91 -
> CassandraAuthorizer failed to authorize #<User updateprog> for <keyspace
> logdata>
> ERROR [Thrift:14] 2015-07-01 11:41:26,210 CustomTThreadPoolServer.java:223 -
> Error occurred during processing of message.
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
> received only 0 responses.
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
> ~[guava-16.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
> ~[guava-16.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:362)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:295)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:272)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:259)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:243)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:143)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1891)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4588)
> ~[apache-cassandra-thrift-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4572)
> ~[apache-cassandra-thrift-2.2.0-rc1.jar:2.2.0-rc1]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[libthrift-0.9.2.jar:0.9.2]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[libthrift-0.9.2.jar:0.9.2]
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:204)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.7.0_55]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.7.0_55]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
