Hi, We have some Timeout frequently in our node cassandra, can any one help me to analyse what can be the cause.
INFO [SlabPoolCleaner] 2018-12-27 05:07:01,836 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:01,836 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239025519 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:01,853 Memtable.java:347 - Writing Memtable-customers_metrics@2086879045(189.254MiB serialized bytes, 163048 ops, 11%/0% of on/off-heap limit) INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:02,398 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140634-Data.db (21.863MiB) for commitlog position ReplayPosition(segmentId=1525681566111, position=14599365) INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:02,912 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140635-Data.db (21.556MiB) for commitlog position ReplayPosition(segmentId=1525681566117, position=19478541) INFO [SlabPoolCleaner] 2018-12-27 05:07:03,034 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:03,035 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239127661 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:03,047 Memtable.java:347 - Writing Memtable-customers_metrics@962624348(189.403MiB serialized bytes, 163122 ops, 11%/0% of on/off-heap limit) INFO [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:04,113 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239049872 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:04,123 Memtable.java:347 - Writing Memtable-customers_metrics@219594445(189.231MiB serialized bytes, 163174 ops, 11%/0% of on/off-heap limit) INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:04,210 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140636-Data.db (21.873MiB) for commitlog position ReplayPosition(segmentId=1525681566123, position=24516704) INFO [MemtableFlushWriter:55844] 2018-12-27 05:07:05,168 Memtable.java:382 - Completed flushing /var/opt/data/flat/bdfcas/files/data/db_cass_agregats/customers_metrics-af4be81092db11e78b314125c9bc7cec/db_cass_agregats-customers_metrics-tmp-ka-140637-Data.db (21.613MiB) for commitlog position ReplayPosition(segmentId=1525681566129, position=29370537) ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,349 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-2] 2018-12-27 05:07:10,349 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-4] 2018-12-27 05:07:10,349 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-5] 2018-12-27 05:07:10,350 QueryMessage.java:135 - Unexpected error during query com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-1] 2018-12-27 05:07:10,350 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-2] 2018-12-27 05:07:10,350 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-4] 2018-12-27 05:07:10,351 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted ERROR [SharedPool-Worker-5] 2018-12-27 05:07:10,351 ErrorMessage.java:251 - Unexpected exception during request com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na] at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.authorize(ClientState.java:353) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:251) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:245) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:229) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:132) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:235) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:260) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:122) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.13.jar:2.1.13] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.13.jar:2.1.13] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na] ... 23 common frames omitted Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:146) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1397) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1324) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1247) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:224) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 32 common frames omitted INFO [SlabPoolCleaner] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.33/0.00, live: 0.33/0.00, flushing: 0.00/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:13,204 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239164849 (11%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:55845] 2018-12-27 05:07:13,216 Memtable.java:347 - Writing Memtable-customers_metrics@1858853610(189.322MiB serialized bytes, 163202 ops, 11%/0% of on/off-heap limit) INFO [SlabPoolCleaner] 2018-12-27 05:07:13,801 ColumnFamilyStore.java:1197 - Flushing largest CFS(Keyspace='db_cass_agregats', ColumnFamily='customers_metrics') to free up room. Used total: 0.45/0.00, live: 0.33/0.00, flushing: 0.11/0.00, this: 0.11/0.11 INFO [SlabPoolCleaner] 2018-12-27 05:07:13,801 ColumnFamilyStore.java:905 - Enqueuing flush of customers_metrics: 239050988 (11%) on-heap, 0 (0%) off-heap Best Rabii
