[
https://issues.apache.org/jira/browse/CASSANDRA-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074829#comment-14074829
]
Ryan McGuire commented on CASSANDRA-7567:
-----------------------------------------
{code}
cqlsh> select activity, source, source_elapsed, thread from
system_traces.events where session_id = f0c14390-1434-11e4-bbc2-1511583f4f50;
activity | source | source_elapsed |
thread
------------------------------------------+----------------+----------------+-----------------------
Message received from /10.109.137.124 | 10.111.173.249 | 50 |
Thread-11
Determining replicas for mutation | 10.109.137.124 | 158 |
Thrift:116
Sending message to /10.111.173.249 | 10.109.137.124 | 303 |
WRITE-/10.111.173.249
Sending message to /10.180.41.228 | 10.109.137.124 | 303 |
WRITE-/10.180.41.228
Message received from /10.109.137.124 | 10.180.41.228 | 36 |
Thread-28
Appending to commitlog | 10.111.173.249 | 94484 |
SharedPool-Worker-29
Adding to Standard1 memtable | 10.111.173.249 | 94569 |
SharedPool-Worker-29
Enqueuing response to /10.109.137.124 | 10.111.173.249 | 94636 |
SharedPool-Worker-29
Sending message to /10.109.137.124 | 10.111.173.249 | 94752 |
WRITE-/10.109.137.124
Message received from /10.111.173.249 | 10.109.137.124 | 95898 |
Thread-4
Processing response from /10.111.173.249 | 10.109.137.124 | 96001 |
SharedPool-Worker-11
(11 rows)
{code}
> when the commit_log disk for a single node is overwhelmed the entire cluster
> slows down
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-7567
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7567
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: debian 7.5, bare metal, 14 nodes, 64CPUs, 64GB RAM,
> commit_log disk sata, data disk SSD, vnodes, leveled compaction strategy
> Reporter: David O'Dell
> Assignee: Brandon Williams
> Attachments: 7567.logs.bz2, write_request_latency.png
>
>
> We've run into a situation where a single node out of 14 is experiencing high
> disk io. This can happen when a node is being decommissioned or after it
> joins the ring and runs into the bug cassandra-6621.
> When this occurs the write latency for the entire cluster spikes.
> From 0.3ms to 170ms.
> To simulate this simply run dd on the commit_log disk (dd if=/dev/zero
> of=/tmp/foo bs=1024) and you will see that instantly all nodes in the cluster
> have slowed down.
> BTW overwhelming the data disk does not have this same effect.
> Also I've tried this where the overwhelmed node isn't being connected
> directly from the client and it still has the same effect.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
