Yadong Chen created CASSANDRA-15932: ---------------------------------------
Summary: CommitLogSegmentManager#sync is doing sync one by one need to be improved Key: CASSANDRA-15932 URL: https://issues.apache.org/jira/browse/CASSANDRA-15932 Project: Cassandra Issue Type: Improvement Components: Local/Commit Log Reporter: Yadong Chen Assignee: Yadong Chen Attachments: image-2020-07-08-12-05-07-746.png, image-2020-07-08-12-21-10-369.png test machine settings: # 24 cores # 45G memory # maximum disk thoughput can over 500MB/s I was running PeriodicCommitLogStressTest#testFixedSize in that test machine and notice both cpu and io can't be full utilized PeriodicCommitLogStressTest#testFixedSize was running with CommitLogSyncPeriod set to 10000 (I changed it to match production environment setting) and no compression and no encryption !image-2020-07-08-12-05-07-746.png! After some digging I find out it's the sync process can't keep up and when lastSyncedAt is far behind all thread doing CommitLog#add will wait until it catch up. Once catch up, all threads go back continue writing and so on. The overall IO utilization I guess is about 60% more or less. After introduce a thread pool to do sync parallel instead of doing it one by one. The IO utilization is improved !image-2020-07-08-12-21-10-369.png! the IO throughput is more steady (almost always above 400MB) and need less time to reach that throughput -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org