[jira] [Commented] (CASSANDRA-9347) Manually run CommitLogStress for 2.2 release
[ https://issues.apache.org/jira/browse/CASSANDRA-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572666#comment-14572666 ] Alan Boudreault commented on CASSANDRA-9347: [~aweisberg], FYI, the archiving has been added in our long CommitLogStress job on jenkins. The commit logs are archived very often and produce a lot of data, so I've setup some background task to clean that directory periodically. I always keep 5-10G of commit log archives, which are restored on the next run. Manually run CommitLogStress for 2.2 release Key: CASSANDRA-9347 URL: https://issues.apache.org/jira/browse/CASSANDRA-9347 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Alan Boudreault Fix For: 2.2.x Commitlog stress runs each test for 10 seconds based on a constant. Might be worth raising that to get the CL doing a little bit more work. Then run it in a loop on something with a fast SSD and something with a slow disk for a few days and see if it fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9347) Manually run CommitLogStress for 2.2 release
[ https://issues.apache.org/jira/browse/CASSANDRA-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14556369#comment-14556369 ] Alan Boudreault commented on CASSANDRA-9347: [~aweisberg] Did you get a chance to take a look about enabling archiving with CommitLogStressTest? Manually run CommitLogStress for 2.2 release Key: CASSANDRA-9347 URL: https://issues.apache.org/jira/browse/CASSANDRA-9347 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Alan Boudreault Fix For: 2.2.x Commitlog stress runs each test for 10 seconds based on a constant. Might be worth raising that to get the CL doing a little bit more work. Then run it in a loop on something with a fast SSD and something with a slow disk for a few days and see if it fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9347) Manually run CommitLogStress for 2.2 release
[
https://issues.apache.org/jira/browse/CASSANDRA-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550663#comment-14550663
]
Alan Boudreault commented on CASSANDRA-9347:
[~aweisberg] I've setup the jenkins project to run CommitLogStressTest each
week. The job is set to run commit log stress for 2 days on a ec2 machine,
which has SSD disk and on a GCE machine (which has a SAN disk).
I've been trying to see if/how I could set the archiving in those test but it
seems that the archiving is not triggered the way it would be in a normal node
stop/start. If you have any hints on that, it would be appreciated. It might
also require some modifications in that CommitLogStressTest so it could archive
and restore correctly while running. This is the error I see when enabling
archiving:
{code}
[junit] Caused by: java.io.IOException: Exception while executing the
command: /bin/ln
build/test/cassandra/commitlog/stress/CommitLog-5-1432049832433.log
/backup/CommitLog-5-1432049832433.log, command error Code: 1, command output:
/bin/ln: failed to create hard link ‘/backup/CommitLog-5-1432049832433.log’ =
‘build/test/cassandra/commitlog/stress/CommitLog-5-1432049832433.log’: No such
file or directory
{code}
Manually run CommitLogStress for 2.2 release
Key: CASSANDRA-9347
URL: https://issues.apache.org/jira/browse/CASSANDRA-9347
Project: Cassandra
Issue Type: Test
Reporter: Ariel Weisberg
Assignee: Alan Boudreault
Fix For: 2.2.x
Commitlog stress runs each test for 10 seconds based on a constant. Might be
worth raising that to get the CL doing a little bit more work.
Then run it in a loop on something with a fast SSD and something with a slow
disk for a few days and see if it fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9347) Manually run CommitLogStress for 2.2 release
[ https://issues.apache.org/jira/browse/CASSANDRA-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542529#comment-14542529 ] Ariel Weisberg commented on CASSANDRA-9347: --- I think you have to do option #2 right now. It's not designed to run indefinitely because it doesn't discard log segments as it goes. That is something I would like to fix. Maybe also look into exercising commit log archiving as well? I am still on the fence as to whether this is the right way to test this subsystem, but we made changes for 2.2 so we should do what we can now. Manually run CommitLogStress for 2.2 release Key: CASSANDRA-9347 URL: https://issues.apache.org/jira/browse/CASSANDRA-9347 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ryan McGuire Fix For: 2.2.x Commitlog stress runs each test for 10 seconds based on a constant. Might be worth raising that to get the CL doing a little bit more work. Then run it in a loop on something with a fast SSD and something with a slow disk for a few days and see if it fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9347) Manually run CommitLogStress for 2.2 release
[ https://issues.apache.org/jira/browse/CASSANDRA-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542551#comment-14542551 ] Alan Boudreault commented on CASSANDRA-9347: ah, I see. Ok I'll check to also include commitlog archiving. +1 to do what we can now and we'll improve over time. Manually run CommitLogStress for 2.2 release Key: CASSANDRA-9347 URL: https://issues.apache.org/jira/browse/CASSANDRA-9347 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Alan Boudreault Fix For: 2.2.x Commitlog stress runs each test for 10 seconds based on a constant. Might be worth raising that to get the CL doing a little bit more work. Then run it in a loop on something with a fast SSD and something with a slow disk for a few days and see if it fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9347) Manually run CommitLogStress for 2.2 release
[ https://issues.apache.org/jira/browse/CASSANDRA-9347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542512#comment-14542512 ] Alan Boudreault commented on CASSANDRA-9347: [~aweisberg] I'm looking to automate that rather than running it manually for 2.2. I will be similar to the DurationTest and could be run weekly or monthly on jenkins. This is what I'm working on: 1. Provide a patch for CommitLogStressTest.java to allow us to specify the run time. (for longer run). 2. Create a new Jenkins CommitLogTest project that will automate the run of this test in a ctool instance. 3. Configure the Jenkins project to launch 2 ctool instances (SSD and NON-SSD) and run a long test for ~2 days. Does this plan sound good? Let me know if you have anything else in mind. Question: what would you prefer for the long run tests (2 days): 1. Start only 1 CommitLogStressTest process with very-long run time, which mean that the test will spend many hours on the same compressor test, then pass to the next one. 2. Start many CommitLogStressTest processes *synchronously* with shorter run time. This is mainly just a loop executing the same test multiple times. Manually run CommitLogStress for 2.2 release Key: CASSANDRA-9347 URL: https://issues.apache.org/jira/browse/CASSANDRA-9347 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ryan McGuire Fix For: 2.2.x Commitlog stress runs each test for 10 seconds based on a constant. Might be worth raising that to get the CL doing a little bit more work. Then run it in a loop on something with a fast SSD and something with a slow disk for a few days and see if it fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
