There’s a timeout happening when querying the system tables in the auth subsystem. The timeout message "Operation timed out - received only 2 responses” indicates that you’re logging in as the default “cassandra” superuser, as queries for all other users are performed at LOCAL_ONE consistency. Best practice is to use the default superuser only during initial cluster setup and disable it as soon as you’ve configured your own users and permissions. You should also check the replication options for the system_auth keyspace. In particular if you have a multi-dc setup, make sure your replication options reflect that (i.e. using NetworkTopologyStrategy).
I would suggest doing that as soon as possible, but in the meantime you can also: * Check for down or unresponsive nodes which are causing the queries for the auth data to time out. Reads for the default superuser are done at QUORUM, so you may be crossing WAN boundaries if you have a multi-dc setup. * Increase the validity period of the various auth caches. These can be set permanently in cassandra.yaml, or temporarily via JMX - see http://cassandra.apache.org/doc/latest/operating/security.html <http://cassandra.apache.org/doc/latest/operating/security.html> The actual timeout is happening when the superuser status is being queried. In 2.1, this status isn’t cached but once permissions for a user have been cached, it won’t be queried again until the cached permissions expire. Thanks, Sam > On 18 Jan 2019, at 14:20, rabii lamriq <lam...@gmail.com> wrote: > > Hi, > > We have some Timeout frequently in our node cassandra, can any one help me to > analyse what can be the cause. > > > INFO [SlabPoolCleaner] 2018-12-27 05:07:01,836 ColumnFamilyStore.java:1197 - > Flushing largest CFS(Keyspace='db_cass_agregats', > ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, > live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 > INFO [SlabPoolCleaner] 2018-12-27 05:07:01,836 ColumnFamilyStore.java:905 - > Enqueuing flush of customers_metrics: 239025519 (11%) on-heap, 0 (0%) off-heap > INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:01,853 Memtable.java:347 - > Writing Memtable-customers_metrics@2086879045(189.254MiB serialized bytes, > 163048 ops, 11%/0% of on/off-heap limit) > INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:02,398 Memtable.java:382 - > Completed flushing > /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140634-Data.db > (21.863MiB) for commitlog position ReplayPosition(segmentId=1525681566111, > position=14599365) > INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:02,912 Memtable.java:382 - > Completed flushing > /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140635-Data.db > (21.556MiB) for commitlog position ReplayPosition(segmentId=1525681566117, > position=19478541) > INFO [SlabPoolCleaner] 2018-12-27 05:07:03,034 ColumnFamilyStore.java:1197 - > Flushing largest CFS(Keyspace='db_cass_agregats', > ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, > live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 > INFO [SlabPoolCleaner] 2018-12-27 05:07:03,035 ColumnFamilyStore.java:905 - > Enqueuing flush of customers_metrics: 239127661 (11%) on-heap, 0 (0%) off-heap > INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:03,047 Memtable.java:347 - > Writing Memtable-customers_metrics@962624348(189.403MiB serialized bytes, > 163122 ops, 11%/0% of on/off-heap limit) > INFO [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnFamilyStore.java:1197 - > Flushing largest CFS(Keyspace='db_cass_agregats', > ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, > live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 > INFO [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnFamilyStore.java:905 - > Enqueuing flush of customers_metrics: 239049872 (11%) on-heap, 0 (0%) off-heap > INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:04,123 Memtable.java:347 - > Writing Memtable-customers_metrics@219594445(189.231MiB serialized bytes, > 163174 ops, 11%/0% of on/off-heap limit) > INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:04,210 Memtable.java:382 - > Completed flushing > /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140636-Data.db > (21.873MiB) for commitlog position ReplayPosition(segmentId=1525681566123, > position=24516704) > INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:05,168 Memtable.java:382 - > Completed flushing > /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140637-Data.db > (21.613MiB) for commitlog position ReplayPosition(segmentId=1525681566129, > position=29370537) > ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,349 QueryMessage.java:135 - > Unexpected error during query > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > ERROR [SharedPool-Worker-2] 2018-12-27 05:07:10,349 QueryMessage.java:135 - > Unexpected error during query > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > ERROR [SharedPool-Worker-4] 2018-12-27 05:07:10,349 QueryMessage.java:135 - > Unexpected error during query > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > ERROR [SharedPool-Worker-5] 2018-12-27 05:07:10,350 QueryMessage.java:135 - > Unexpected error during query > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,350 ErrorMessage.java:251 - > Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > ERROR [SharedPool-Worker-2] 2018-12-27 05:07:10,350 ErrorMessage.java:251 - > Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > ERROR [SharedPool-Worker-4] 2018-12-27 05:07:10,351 ErrorMessage.java:251 - > Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > ERROR [SharedPool-Worker-5] 2018-12-27 05:07:10,351 ErrorMessage.java:251 - > Unexpected exception during request > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.13.jar:2.1.13] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_75] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.13.jar:2.1.13] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 2 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) > ~[guava-16.0.jar:na] > ... 23 common frames omitted > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 2 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) > ~[apache-cassandra-2.1.13.jar:2.1.13] > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) > ~[apache-cassandra-2.1.13.jar:2.1.13] > ... 32 common frames omitted > INFO [SlabPoolCleaner] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:1197 - > Flushing largest CFS(Keyspace='db_cass_agregats', > ColumnFamily='customers_metrics') to free up room. Used total: 0.33/0.00, > live: 0.33/0.00, flushing: 0.00/0.00, this: 0.11/0.11 > INFO [SlabPoolCleaner] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:905 - > Enqueuing flush of customers_metrics: 239164849 (11%) on-heap, 0 (0%) off-heap > INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:13,216 Memtable.java:347 - > Writing Memtable-customers_metrics@1858853610(189.322MiB serialized bytes, > 163202 ops, 11%/0% of on/off-heap limit) > INFO [SlabPoolCleaner] 2018-12-27 05:07:13,801 ColumnFamilyStore.java:1197 - > Flushing largest CFS(Keyspace='db_cass_agregats', > ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, > live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 > INFO [SlabPoolCleaner] 2018-12-27 05:07:13,801 ColumnFamilyStore.java:905 - > Enqueuing flush of customers_metrics: 239050988 (11%) on-heap, 0 (0%) off-heap > > > Best > Rabii
