[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607516#comment-15607516 ] Stefania commented on CASSANDRA-12802: -- No. In 2.2+ we already have CASSANDRA-11743. As for the deadlock, it cannot be reproduced in later versions and I believe this is because of CASSANDRA-6809, which fixed the logic in commit log segments, and CASSANDRA-8308 which changed the way the CL is restarted during tests. So we don't need the test hack in later versions. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.17 > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607431#comment-15607431 ] Nate McCall commented on CASSANDRA-12802: - Fair enough. I appreciate ya'll being thorough on it regardless. Thanks! > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Testing >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.17 > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606967#comment-15606967
]
Stefania commented on CASSANDRA-12802:
--
Thank you for the review and for the feedback. Committed to 2.1 as
079029a44290f78121e3577da60cec93a8ca9f10 and merged upwards with {{-s ours}}.
> RecoveryManagerTruncateTest fails in 2.1
>
>
> Key: CASSANDRA-12802
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12802
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> Example failure:
> http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604451#comment-15604451
]
Branimir Lambov commented on CASSANDRA-12802:
-
Code LGTM
[~zznate]: I would say that this patch removes risk as it replaces racy logic
in {{recover}} with a very simple filter based on a value that:
- was there before the patch;
- could cause serious trouble if it was wrong regardless of the patch.
> RecoveryManagerTruncateTest fails in 2.1
>
>
> Key: CASSANDRA-12802
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12802
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> Example failure:
> http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604267#comment-15604267 ] Stefania commented on CASSANDRA-12802: -- Noted, and if this is too risky then I suggest ignoring this test in 2.1. However, it is not just a test scenario, CASSANDRA-11743 could happen in 2.1 as well. The fix has been released in 2.2.7 and so far it has not caused problems that I know of, but let's hear [~blambov]'s opinion since I am still not that familiar with CL internals. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604120#comment-15604120 ] Nate McCall commented on CASSANDRA-12802: - [~Stefania] I'm super nervous about changing the behavior of CL.recover in 2.1 to address a (admittedly annoying) test scenario. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604047#comment-15604047
]
Stefania commented on CASSANDRA-12802:
--
Thanks for the feedback [~blambov]. Would you mind reviewing the full patch?
It's basically CASSANDRA-11743 plus the hack:
|2.1|[patch|https://github.com/stef1927/cassandra/tree/12802-2.1]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12802-2.1-testall/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12802-2.1-dtest/]|
I then plan to do a {{merge -s ours}} into 2.2 and upwards.
I'm also multiplexing one more time with the final patch
[here|http://cassci.datastax.com/job/stef1927-testall-multiplex-2/9/].
> RecoveryManagerTruncateTest fails in 2.1
>
>
> Key: CASSANDRA-12802
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12802
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> Example failure:
> http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601968#comment-15601968 ] Branimir Lambov commented on CASSANDRA-12802: - I don't think this is critical enough to fix in 2.1, the changes to commit log we do for that will be a risk in themselves. I would go with the hacky test fix. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601432#comment-15601432
]
Stefania commented on CASSANDRA-12802:
--
I've just completed a successful multiplexed run
[here|http://cassci.datastax.com/job/stef1927-testall-multiplex-2/8/] with this
[patch|https://github.com/apache/cassandra/compare/trunk...stef1927:12802-2.1-simple#diff-05c1e4fd86fea19b8e0552b1f289be85R382].
It's really a hack in the test code, basically because the race is caused by
the mutations inserted by the sstable {{GlobalTidy}}, we can just wait for
those tasks to complete before resetting the CL.
To fix the deadlock properly, as it is already done in 2.2, I believe we have
to change {{CommitLogSegment}}. I think the problem is that we are still
creating allocations after the segment has been closed. I think we would need
to backport the relevant parts of CASSANDRA-6809: {{endOfBuffer}} in
{{CommitLogSegement}} and possibly something else. [~blambov] is this an issue
likely to happen in production that is critical enough to fix in 2.1, or is the
hack that fixes the test sufficient?
> RecoveryManagerTruncateTest fails in 2.1
>
>
> Key: CASSANDRA-12802
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12802
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> Example failure:
> http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594723#comment-15594723
]
Stefania commented on CASSANDRA-12802:
--
I've reproduced at least one timeout failure with stack traces
[here|http://cassci.datastax.com/job/stef1927-testall-multiplex-2/7/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest_41/testTruncatePointInTimeReplayList/].
I'm still analyzing it, but I think I need some of the commits of
CASSANDRA-8308, the ones made by [~snazy] in order to fix the unit tests.
There are also different failures, sample
[here|http://cassci.datastax.com/job/stef1927-testall-multiplex-2/7/testReport/org.apache.cassandra.db/RecoveryManagerTruncateTest_139/testTruncatePointInTimeReplayList/],
with NPEs in {{CommitLogSegmentManager.allocate}} and file not found errors,
although these may be a consequence of the interrupted exceptions injected by
the junit timeouts.
> RecoveryManagerTruncateTest fails in 2.1
>
>
> Key: CASSANDRA-12802
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12802
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> Example failure:
> http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594345#comment-15594345 ] Stefania commented on CASSANDRA-12802: -- This multiplexed run [here|https://cassci.datastax.com/job/stef1927-testall-multiplex-2/7/] should print all thread's stack traces if a test is still running after 2 minutes. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594215#comment-15594215 ] Stefania commented on CASSANDRA-12802: -- Unfortunately there were 2 timeouts in the multiplexed run above. Given that each test normally completes in about 8 seconds, I'm guessing these were two deadlocks. I need to find a way to extract a stack trace from Jenkins to debug further. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593756#comment-15593756 ] Stefania commented on CASSANDRA-12802: -- Following Branimir's advice that he gave me offline, I've backported CASSANDRA-11743 on [this branch|https://github.com/stef1927/cassandra/tree/12802-2.1-simple] and that seems to have fixed the failures locally. Multiplexing on Jenkins [here|https://cassci.datastax.com/job/stef1927-testall-multiplex-2/6/]. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15591356#comment-15591356
]
Stefania commented on CASSANDRA-12802:
--
The flake disappears if we reset the commitlog after every test, but this
introduced 2 deadlocks on Jenkins, which I cannot reproduce locally and
therefore I don't know why that is, see
[here|https://cassci.datastax.com/job/stef1927-testall-multiplex-2/2/].
The flake also disappears if we run individual tests in isolation, see
[here|https://cassci.datastax.com/job/stef1927-testall-multiplex-2/4/] and
[here|https://cassci.datastax.com/job/stef1927-testall-multiplex-2/5/].
Regardless, I am concerned we may have a race in the CL of 2.1, although it
would involve CL.recover, which is only called on startup and from tests. With
this [branch|https://github.com/stef1927/cassandra/tree/12802-2.1], I can
reproduce the failure locally, and I've verified that the assertion fails
because no mutation is replayed during the recover. [~blambov] does this look
like segment 1476951270428 is deleted before a mutation is added to it or is it
normal?
{code}
[junit] DEBUG 08:14:33 (Unopened) segment
build/test/cassandra/commitlog:0/CommitLog-4-1476951270428.log is no longer
needed and will be deleted now
[junit] DEBUG 08:14:33 Deleting CommitLog-4-1476951270428.log
[junit] DEBUG 08:14:33 No segments in reserve; creating a fresh one
[junit] DEBUG 08:14:33 Creating new commit log segment
build/test/cassandra/commitlog:0/CommitLog-4-1476951270429.log
[junit] DEBUG 08:14:33 Total active commitlog segment space used is 20971520
[junit] DEBUG 08:14:33 Mutation Mutation(keyspace='Keyspace1',
key='6b65796d756c7469', modifications=[ColumnFamily(Standard1
[636f6c32:false:4@1,])]) added to CL at ReplayPosition(segmentId=1476951270428,
position=110)
[junit] DEBUG 08:14:33 truncating Standard1
[junit] INFO 08:14:33 Enqueuing flush of Standard1: 201 (0%) on-heap, 39
(0%) off-heap
[junit] INFO 08:14:33 Writing Memtable-Standard1@46454610(0.023KiB
serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
[junit] INFO 08:14:33 Completed flushing
build/test/cassandra/data:0/Keyspace1/Standard1-39c5e040969d11e695c777a7fe43be1f/Keyspace1-Standard1-tmp-ka-2-Data.db
(0.046KiB) for commitlog position ReplayPosition(segmentId=1476951270428,
position=118)
{code}
>From what I can see, there is a small window in CLSM.advanceAllocatingFrom()
>where a new segment doesn't exist in either the available or active segments,
>and that section of the code is synchronized. Therefore, the filter used by
>CL.recover(), CLSM.manages(String name) doesn't look safe to me, however
>synchronizing CLSM.manages() did not solve the problem.
[~blambov], I don't want to take too much of your time as I've kept you busy
already in the last few days, so don't spend time investigating this just yet,
but if you know of any known issues or limitations with the CL of 2.1 that
might explain this, your insight is welcome.
> RecoveryManagerTruncateTest fails in 2.1
>
>
> Key: CASSANDRA-12802
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12802
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
>
> Example failure:
> http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588116#comment-15588116 ] Stefania commented on CASSANDRA-12802: -- I've noticed that I can no longer reproduce the problem locally if I reset the commit log after every test, trying a multiplexed run here: https://cassci.datastax.com/job/stef1927-testall-multiplex-2/. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12802) RecoveryManagerTruncateTest fails in 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588065#comment-15588065 ] Stefania commented on CASSANDRA-12802: -- I've multiplexed this and reproduced 18 failures out of 300 tests: https://cassci.datastax.com/job/stef1927-testall-multiplex-2/1/testReport/. I can also reproduce it locally, approximately once out of 50 attempts, but I need to use a slightly modified [code|https://github.com/stef1927/cassandra/commits/12802-2.1]. When the test fails, the mutations are not replayed but I am not sure why. > RecoveryManagerTruncateTest fails in 2.1 > > > Key: CASSANDRA-12802 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12802 > Project: Cassandra > Issue Type: Bug >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x > > > Example failure: > http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/388/testReport/junit/org.apache.cassandra.db/RecoveryManagerTruncateTest/testTruncatePointInTime/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
