[ https://issues.apache.org/jira/browse/CASSANDRA-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074829#comment-14074829 ]
Ryan McGuire commented on CASSANDRA-7567: ----------------------------------------- {code} cqlsh> select activity, source, source_elapsed, thread from system_traces.events where session_id = f0c14390-1434-11e4-bbc2-1511583f4f50; activity | source | source_elapsed | thread ------------------------------------------+----------------+----------------+----------------------- Message received from /10.109.137.124 | 10.111.173.249 | 50 | Thread-11 Determining replicas for mutation | 10.109.137.124 | 158 | Thrift:116 Sending message to /10.111.173.249 | 10.109.137.124 | 303 | WRITE-/10.111.173.249 Sending message to /10.180.41.228 | 10.109.137.124 | 303 | WRITE-/10.180.41.228 Message received from /10.109.137.124 | 10.180.41.228 | 36 | Thread-28 Appending to commitlog | 10.111.173.249 | 94484 | SharedPool-Worker-29 Adding to Standard1 memtable | 10.111.173.249 | 94569 | SharedPool-Worker-29 Enqueuing response to /10.109.137.124 | 10.111.173.249 | 94636 | SharedPool-Worker-29 Sending message to /10.109.137.124 | 10.111.173.249 | 94752 | WRITE-/10.109.137.124 Message received from /10.111.173.249 | 10.109.137.124 | 95898 | Thread-4 Processing response from /10.111.173.249 | 10.109.137.124 | 96001 | SharedPool-Worker-11 (11 rows) {code} > when the commit_log disk for a single node is overwhelmed the entire cluster > slows down > --------------------------------------------------------------------------------------- > > Key: CASSANDRA-7567 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7567 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: debian 7.5, bare metal, 14 nodes, 64CPUs, 64GB RAM, > commit_log disk sata, data disk SSD, vnodes, leveled compaction strategy > Reporter: David O'Dell > Assignee: Brandon Williams > Attachments: 7567.logs.bz2, write_request_latency.png > > > We've run into a situation where a single node out of 14 is experiencing high > disk io. This can happen when a node is being decommissioned or after it > joins the ring and runs into the bug cassandra-6621. > When this occurs the write latency for the entire cluster spikes. > From 0.3ms to 170ms. > To simulate this simply run dd on the commit_log disk (dd if=/dev/zero > of=/tmp/foo bs=1024) and you will see that instantly all nodes in the cluster > have slowed down. > BTW overwhelming the data disk does not have this same effect. > Also I've tried this where the overwhelmed node isn't being connected > directly from the client and it still has the same effect. -- This message was sent by Atlassian JIRA (v6.2#6252)