[jira] [Commented] (CASSANDRA-15932) CommitLogSegmentManager#sync is doing sync one by one need to be improved
[ https://issues.apache.org/jira/browse/CASSANDRA-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153255#comment-17153255 ] Yadong Chen commented on CASSANDRA-15932: - I create a pull request here: [https://github.com/apache/cassandra/pull/668] Since it's my very first involved in this project. Advise is more than welcome. [~benedict] [~djoshi] [~jwest] Could you spare a moment for this one > CommitLogSegmentManager#sync is doing sync one by one need to be improved > - > > Key: CASSANDRA-15932 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15932 > Project: Cassandra > Issue Type: Improvement > Components: Local/Commit Log >Reporter: Yadong Chen >Assignee: Yadong Chen >Priority: Normal > Attachments: image-2020-07-08-12-05-07-746.png, > image-2020-07-08-12-21-10-369.png > > > test machine settings: > # 24 cores > # 45G memory > # maximum disk thoughput can over 500MB/s > > I was running PeriodicCommitLogStressTest#testFixedSize in that test machine > and notice both cpu and io can't be full utilized > PeriodicCommitLogStressTest#testFixedSize was running with > CommitLogSyncPeriod set to 1 (I changed it to match production > environment setting) and no compression and no encryption > !image-2020-07-08-12-05-07-746.png! > After some digging I find out it's the sync process can't keep up and when > lastSyncedAt is far behind all thread doing CommitLog#add will wait until it > catch up. > Once catch up, all threads go back continue writing and so on. > The overall IO utilization I guess is about 60% more or less. After introduce > a thread pool to do sync parallel instead of doing it one by one. The IO > utilization is improved > !image-2020-07-08-12-21-10-369.png! > the IO throughput is more steady (almost always above 400MB) and need less > time to reach that throughput -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15932) CommitLogSegmentManager#sync is doing sync one by one need to be improved
Yadong Chen created CASSANDRA-15932: --- Summary: CommitLogSegmentManager#sync is doing sync one by one need to be improved Key: CASSANDRA-15932 URL: https://issues.apache.org/jira/browse/CASSANDRA-15932 Project: Cassandra Issue Type: Improvement Components: Local/Commit Log Reporter: Yadong Chen Assignee: Yadong Chen Attachments: image-2020-07-08-12-05-07-746.png, image-2020-07-08-12-21-10-369.png test machine settings: # 24 cores # 45G memory # maximum disk thoughput can over 500MB/s I was running PeriodicCommitLogStressTest#testFixedSize in that test machine and notice both cpu and io can't be full utilized PeriodicCommitLogStressTest#testFixedSize was running with CommitLogSyncPeriod set to 1 (I changed it to match production environment setting) and no compression and no encryption !image-2020-07-08-12-05-07-746.png! After some digging I find out it's the sync process can't keep up and when lastSyncedAt is far behind all thread doing CommitLog#add will wait until it catch up. Once catch up, all threads go back continue writing and so on. The overall IO utilization I guess is about 60% more or less. After introduce a thread pool to do sync parallel instead of doing it one by one. The IO utilization is improved !image-2020-07-08-12-21-10-369.png! the IO throughput is more steady (almost always above 400MB) and need less time to reach that throughput -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15779) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide
[ https://issues.apache.org/jira/browse/CASSANDRA-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Chen updated CASSANDRA-15779: Bug Category: Parent values: Correctness(12982)Level 1 values: Test Failure(12990) Component/s: Test/unit Status: Open (was: Triage Needed) > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > fail when run with intellij ide > > > Key: CASSANDRA-15779 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15779 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Yadong Chen >Assignee: Yadong Chen >Priority: Low > Labels: test > > I run > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > using ide intellij with mac > and it fails with following message: > Thread #13: AsyncAppender-Worker-ASYNC > Thread #12: Attach Listener > Thread #10: logback-1 > Thread #5: Monitor Ctrl-Break > Thread #4: Signal Dispatcher > Thread #3: Finalizer > Thread #2: Reference Handler > Thread #1: main > java.lang.AssertionError: thread started in clientInitialization > Expected :5 > Actual :8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15779) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide
[ https://issues.apache.org/jira/browse/CASSANDRA-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Chen updated CASSANDRA-15779: Test and Documentation Plan: no need Status: Patch Available (was: Open) > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > fail when run with intellij ide > > > Key: CASSANDRA-15779 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15779 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Yadong Chen >Assignee: Yadong Chen >Priority: Low > Labels: test > > I run > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > using ide intellij with mac > and it fails with following message: > Thread #13: AsyncAppender-Worker-ASYNC > Thread #12: Attach Listener > Thread #10: logback-1 > Thread #5: Monitor Ctrl-Break > Thread #4: Signal Dispatcher > Thread #3: Finalizer > Thread #2: Reference Handler > Thread #1: main > java.lang.AssertionError: thread started in clientInitialization > Expected :5 > Actual :8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15779) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide
[ https://issues.apache.org/jira/browse/CASSANDRA-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17097816#comment-17097816 ] Yadong Chen commented on CASSANDRA-15779: - patch is available: [https://github.com/apache/cassandra/pull/579] > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > fail when run with intellij ide > > > Key: CASSANDRA-15779 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15779 > Project: Cassandra > Issue Type: Bug >Reporter: Yadong Chen >Assignee: Yadong Chen >Priority: Low > Labels: test > > I run > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > using ide intellij with mac > and it fails with following message: > Thread #13: AsyncAppender-Worker-ASYNC > Thread #12: Attach Listener > Thread #10: logback-1 > Thread #5: Monitor Ctrl-Break > Thread #4: Signal Dispatcher > Thread #3: Finalizer > Thread #2: Reference Handler > Thread #1: main > java.lang.AssertionError: thread started in clientInitialization > Expected :5 > Actual :8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15779) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide
[ https://issues.apache.org/jira/browse/CASSANDRA-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Chen updated CASSANDRA-15779: Description: I run org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef using ide intellij with mac and it fails with following message: Thread #13: AsyncAppender-Worker-ASYNC Thread #12: Attach Listener Thread #10: logback-1 Thread #5: Monitor Ctrl-Break Thread #4: Signal Dispatcher Thread #3: Finalizer Thread #2: Reference Handler Thread #1: main java.lang.AssertionError: thread started in clientInitialization Expected :5 Actual :8 was: I run org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef using ide intellij with mac and it fails with following message: Thread #13: AsyncAppender-Worker-ASYNC Thread #12: Attach Listener Thread #10: logback-1 Thread #5: Monitor Ctrl-Break Thread #4: Signal Dispatcher Thread #3: Finalizer Thread #2: Reference Handler Thread #1: main java.lang.AssertionError: thread started in clientInitialization Expected :5 Actual :8 > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > fail when run with intellij ide > > > Key: CASSANDRA-15779 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15779 > Project: Cassandra > Issue Type: Bug >Reporter: Yadong Chen >Assignee: Yadong Chen >Priority: Low > Labels: test > > I run > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > using ide intellij with mac > and it fails with following message: > Thread #13: AsyncAppender-Worker-ASYNC > Thread #12: Attach Listener > Thread #10: logback-1 > Thread #5: Monitor Ctrl-Break > Thread #4: Signal Dispatcher > Thread #3: Finalizer > Thread #2: Reference Handler > Thread #1: main > java.lang.AssertionError: thread started in clientInitialization > Expected :5 > Actual :8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15779) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide
[ https://issues.apache.org/jira/browse/CASSANDRA-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Chen updated CASSANDRA-15779: Complexity: Low Hanging Fruit Discovered By: Unit Test Severity: Low Labels: test (was: ) > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > fail when run with intellij ide > > > Key: CASSANDRA-15779 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15779 > Project: Cassandra > Issue Type: Bug >Reporter: Yadong Chen >Assignee: Yadong Chen >Priority: Low > Labels: test > > I run > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > using ide intellij with mac > > and it fails with following message: > Thread #13: AsyncAppender-Worker-ASYNC > Thread #12: Attach Listener > Thread #10: logback-1 > Thread #5: Monitor Ctrl-Break > Thread #4: Signal Dispatcher > Thread #3: Finalizer > Thread #2: Reference Handler > Thread #1: main > java.lang.AssertionError: thread started in clientInitialization > Expected :5 > Actual :8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15779) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide
[ https://issues.apache.org/jira/browse/CASSANDRA-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Chen updated CASSANDRA-15779: Description: I run org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef using ide intellij with mac and it fails with following message: Thread #13: AsyncAppender-Worker-ASYNC Thread #12: Attach Listener Thread #10: logback-1 Thread #5: Monitor Ctrl-Break Thread #4: Signal Dispatcher Thread #3: Finalizer Thread #2: Reference Handler Thread #1: main java.lang.AssertionError: thread started in clientInitialization Expected :5 Actual :8 was: I run org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef using ide intellij with mac and it fails with following message: Thread #13: AsyncAppender-Worker-ASYNC Thread #12: Attach Listener Thread #10: logback-1 Thread #5: Monitor Ctrl-Break Thread #4: Signal Dispatcher Thread #3: Finalizer Thread #2: Reference Handler Thread #1: main java.lang.AssertionError: thread started in clientInitialization Expected :5 Actual :8 > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > fail when run with intellij ide > > > Key: CASSANDRA-15779 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15779 > Project: Cassandra > Issue Type: Bug >Reporter: Yadong Chen >Assignee: Yadong Chen >Priority: Low > Labels: test > > I run > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > using ide intellij with mac > and it fails with following message: > > Thread #13: AsyncAppender-Worker-ASYNC > Thread #12: Attach Listener > Thread #10: logback-1 > Thread #5: Monitor Ctrl-Break > Thread #4: Signal Dispatcher > Thread #3: Finalizer > Thread #2: Reference Handler > Thread #1: main > java.lang.AssertionError: thread started in clientInitialization > Expected :5 > Actual :8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15779) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide
Yadong Chen created CASSANDRA-15779: --- Summary: test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef fail when run with intellij ide Key: CASSANDRA-15779 URL: https://issues.apache.org/jira/browse/CASSANDRA-15779 Project: Cassandra Issue Type: Bug Reporter: Yadong Chen Assignee: Yadong Chen I run org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef using ide intellij with mac and it fails with following message: Thread #13: AsyncAppender-Worker-ASYNC Thread #12: Attach Listener Thread #10: logback-1 Thread #5: Monitor Ctrl-Break Thread #4: Signal Dispatcher Thread #3: Finalizer Thread #2: Reference Handler Thread #1: main java.lang.AssertionError: thread started in clientInitialization Expected :5 Actual :8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15357) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef constantly failing on trunk
[ https://issues.apache.org/jira/browse/CASSANDRA-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Chen reassigned CASSANDRA-15357: --- Assignee: (was: Yadong Chen) > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > constantly failing on trunk > > > Key: CASSANDRA-15357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15357 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: David Capwell >Priority: Normal > > See https://circleci.com/gh/dcapwell/cassandra/84#tests/containers/1 and > https://circleci.com/gh/dcapwell/cassandra/85#tests/containers/1 > I see that this test is failing on trunk with the following > junit.framework.AssertionFailedError > at > org.apache.cassandra.config.DatabaseDescriptorRefTest.checkViolations(DatabaseDescriptorRefTest.java:293) > at > org.apache.cassandra.config.DatabaseDescriptorRefTest.testDatabaseDescriptorRef(DatabaseDescriptorRefTest.java:277) > Seems related to https://issues.apache.org/jira/browse/CASSANDRA-12677 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15357) test org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef constantly failing on trunk
[ https://issues.apache.org/jira/browse/CASSANDRA-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yadong Chen reassigned CASSANDRA-15357: --- Assignee: Yadong Chen > test > org.apache.cassandra.config.DatabaseDescriptorRefTest#testDatabaseDescriptorRef > constantly failing on trunk > > > Key: CASSANDRA-15357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15357 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: David Capwell >Assignee: Yadong Chen >Priority: Normal > > See https://circleci.com/gh/dcapwell/cassandra/84#tests/containers/1 and > https://circleci.com/gh/dcapwell/cassandra/85#tests/containers/1 > I see that this test is failing on trunk with the following > junit.framework.AssertionFailedError > at > org.apache.cassandra.config.DatabaseDescriptorRefTest.checkViolations(DatabaseDescriptorRefTest.java:293) > at > org.apache.cassandra.config.DatabaseDescriptorRefTest.testDatabaseDescriptorRef(DatabaseDescriptorRefTest.java:277) > Seems related to https://issues.apache.org/jira/browse/CASSANDRA-12677 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
