Hello everyone,
i have a 3 node cluster with Cassandra 2.0.14 on centos in the same Data
center with RF=3 and i am using CL=Local_Quorum by default for the read and
write operations. I have given about 5 GB of heap space to cassandra.
I have 40 core machines with 3 separate SATA disks with commitlog on one
and data directories on the other two.
I am doing a read + write at the same with about 4000 QPS.
I am getting Read failures on the client where one of the replica didnot
respond. When i look at the cassandra logs i see a lot failures as i have
attached (read_failures.txt).
Am i overloading the system too much ? 4000 QPS doesnt seem too much at
first glance.
Please let me know if any other details are required.
DataModel :
partition_key, clustering_key, col1
Regards,
Anishek
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,135 MessagingService.java (line
875) 1482 READ messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,136 StatusLogger.java (line 55)
Pool Name Active Pending Completed Blocked All
Time Blocked
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,136 StatusLogger.java (line 70)
ReadStage 32 4676 2438994 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,137 StatusLogger.java (line 70)
RequestResponseStage 0 0 4624954 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,137 StatusLogger.java (line 70)
ReadRepairStage 0 1 162868 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,137 StatusLogger.java (line 70)
MutationStage 0 0 3794927 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,137 StatusLogger.java (line 70)
ReplicateOnWriteStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,138 StatusLogger.java (line 70)
GossipStage 0 0 5339 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,138 StatusLogger.java (line 70)
CacheCleanupExecutor 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,138 StatusLogger.java (line 70)
MigrationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,138 StatusLogger.java (line 70)
MemoryMeter 0 0 38 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,139 StatusLogger.java (line 70)
ValidationExecutor 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,139 StatusLogger.java (line 70)
FlushWriter 0 0 31 0
9
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,139 StatusLogger.java (line 70)
InternalResponseStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,139 StatusLogger.java (line 70)
AntiEntropyStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,139 StatusLogger.java (line 70)
MemtablePostFlusher 0 0 64 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,140 StatusLogger.java (line 70)
MiscStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,140 StatusLogger.java (line 70)
PendingRangeCalculator 0 0 3 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,140 StatusLogger.java (line 70)
commitlog_archiver 0 0 0 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,140 StatusLogger.java (line 70)
CompactionExecutor 0 0 1149 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,141 StatusLogger.java (line 70)
HintedHandoff 0 1 4 0
0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,141 StatusLogger.java (line 79)
CompactionManager 0 0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,141 StatusLogger.java (line 81)
Commitlog n/a 0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,141 StatusLogger.java (line 93)
MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,141 StatusLogger.java (line 103)
Cache Type Size Capacity
KeysToSave
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,141 StatusLogger.java (line 105)
KeyCache 59330812 104857600
all
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 111)
RowCache 0 0
all
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 118)
ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 121)
system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 121)
system.local 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 121)
system.peers 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 121)
system.batchlog 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 121)
system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 121)
system.compactions_in_progress 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,142 StatusLogger.java (line 121)
system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.compaction_history 56,18390
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.paxos 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.schema_columns 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.range_xfers 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.peer_events 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,143 StatusLogger.java (line 121)
system.hints 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,144 StatusLogger.java (line 121)
system.sstable_activity 120,14400
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,144 StatusLogger.java (line 121)
system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,144 StatusLogger.java (line 121)
system_traces.events 0,0
INFO [ScheduledTasks:1] 2015-05-12 15:30:01,144 StatusLogger.java (line 121)
activity_log.user_stay_points 875888,204472320
