[jira] [Comment Edited] (CASSANDRA-15557) Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest testDropListAndAddListWithSameName
[
https://issues.apache.org/jira/browse/CASSANDRA-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055682#comment-17055682
]
Ryan Svihla edited comment on CASSANDRA-15557 at 3/10/20, 8:15 AM:
---
I borrowed the code from 15303 and updated the PR. Some notes:
* There is an additional fix in there which is related to comparison of
relations. This doesn't appear to be related to either Jira, but it makes sense
at a glance, either way [~jasonstack] added it and he can comment
* Made ClientState.getTimestatmp() static and call it directly instead of
FBUtilities.timestampMicros()
But I'm also ok with someone first merging in 15303, then I can do another PR
based on that code base just with the changes that use
ClientState.getTimetamp() instead of FBUtilities.timestampMicros().
was (Author: rssvihla):
I borrowed the code from 15303 and updated the PR. Some notes:
* There is an additional fix in there which is related to comparison of
relations. This doesn't appear to be related to either Jira, but it makes sense
at a glance, either way [~jasonstack] added it and he can comment
* Made ClientState.getTimestatmp() static and call it directly instead of
FBUtilities.timestampMicros()
But I'm also ok with someone first merging in 15303, then I can do another PR
based on that code base just with the changes that use
ClientState.getTimetamp() instead of FBUtilities.timestampMicros().
> Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest
> testDropListAndAddListWithSameName
> ---
>
> Key: CASSANDRA-15557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15557
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Ryan Svihla
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> https://app.circleci.com/jobs/github/dcapwell/cassandra/482/tests
> {code}
> junit.framework.AssertionFailedError: Invalid value for row 0 column 2
> (mycollection of type list), expected but got <[first element]>
> at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)
> at
> org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15557) Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest testDropListAndAddListWithSameName
[
https://issues.apache.org/jira/browse/CASSANDRA-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054887#comment-17054887
]
Ryan Svihla edited comment on CASSANDRA-15557 at 3/9/20, 11:56 AM:
---
So digging into the actual failure, this took a few tries as wrapping logging,
or flushing sstables seemed to make it hard to reproduce, I've confirmed it's
time based errors at least in this case:
row ts: 1583753957613001
dropped time: {{1583753957613000}}
{{[junit-timeout] Testcase:
testDropListAndAddListWithSameName(org.apache.cassandra.cql3.validation.operations.AlterTest):
FAILED }}
{{[junit-timeout] Dropped column: {java.nio.HeapByteBuffer[pos=0 lim=12
cap=12]=DroppedColumn{column=mycollection, droppedTime=1583753957613000}} Row
timestamp: 1583753957613001 }}
{{[junit-timeout] junit.framework.AssertionFailedError: Dropped column:
{java.nio.HeapByteBuffer[pos=0 lim=12
cap=12]=DroppedColumn{column=mycollection, droppedTime=1583753957613000}} Row
timestamp: 1583753957613001}}
{{[junit-timeout] at
org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:102)
}}
{{[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
}}
{{[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
}}
{{[junit-timeout] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
}}
{{[junit-timeout] Caused by: java.lang.AssertionError: Invalid value for row 0
column 2 (mycollection of type list), expected but got <[first
element]> }}
{{[junit-timeout] at
org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)}}
{{[junit-timeout] at
org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:98)
}}
{{[junit-timeout]}}
{{[junit-timeout]}}
was (Author: rssvihla):
So digging into the actual failure, this took a few tries as wrapping logging,
or flushing sstables seemed to make it hard to reproduce, I've confirmed it's
time based errors at least in this case:
row ts: {{1583753957613001 }}
dropped time: {{1583753957613000}}
{{[junit-timeout] Testcase:
testDropListAndAddListWithSameName(org.apache.cassandra.cql3.validation.operations.AlterTest):
FAILED }}
{{[junit-timeout] Dropped column: \{java.nio.HeapByteBuffer[pos=0 lim=12
cap=12]=DroppedColumn{column=mycollection, droppedTime=1583753957613000}} Row
timestamp: 1583753957613001 }}
{{[junit-timeout] junit.framework.AssertionFailedError: Dropped column:
\{java.nio.HeapByteBuffer[pos=0 lim=12
cap=12]=DroppedColumn{column=mycollection, droppedTime=1583753957613000}} Row
timestamp: 1583753957613001}}
{{[junit-timeout] at
org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:102)
}}
{{[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
}}
{{[junit-timeout] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
}}
{{[junit-timeout] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
}}
{{[junit-timeout] Caused by: java.lang.AssertionError: Invalid value for row 0
column 2 (mycollection of type list), expected but got <[first
element]> }}
{{[junit-timeout] at
org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)}}
{{[junit-timeout] at
org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:98)
}}
{{[junit-timeout]}}
{{[junit-timeout]}}
> Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest
> testDropListAndAddListWithSameName
> ---
>
> Key: CASSANDRA-15557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15557
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Ryan Svihla
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> https://app.circleci.com/jobs/github/dcapwell/cassandra/482/tests
> {code}
> junit.framework.AssertionFailedError: Invalid value for row 0 column 2
> (mycollection of type list), expected but got <[first element]>
> at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)
> at
> org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (CASSANDRA-15557) Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest testDropListAndAddListWithSameName
[
https://issues.apache.org/jira/browse/CASSANDRA-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1705#comment-1705
]
Ryan Svihla edited comment on CASSANDRA-15557 at 3/5/20, 10:27 AM:
---
Yes silly me, easy enough to prove when setting the timestamps to the same
value the tests still pass, and just to totally eliminate weird side effects,
setting the timestamp in reverse breaks the test as expected so I don't think
there is anything specific about setting the timestamp to any value that makes
it pass.
So I'm not sure I have a good theory as to why breaks in the first place. It
does then act as if somehow the drop is happening "before" the add. I'll keep
digging thanks for the review.
was (Author: rssvihla):
Yes silly me, easy enough to prove when setting the timestamps to the same
value the tests still pass, and just to totally eliminate weird side effects,
setting the timestamp in reverse breaks the test as expected so I don't think
there is anything specific about setting the timestamp to any value that makes
it pass.
So I'm not sure I have a good theory as to why breaks in the first place. It
does then act as if somehow the drop is happening "before" the insert. I'll
keep digging thanks for the review.
> Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest
> testDropListAndAddListWithSameName
> ---
>
> Key: CASSANDRA-15557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15557
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Ryan Svihla
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> https://app.circleci.com/jobs/github/dcapwell/cassandra/482/tests
> {code}
> junit.framework.AssertionFailedError: Invalid value for row 0 column 2
> (mycollection of type list), expected but got <[first element]>
> at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)
> at
> org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15557) Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest testDropListAndAddListWithSameName
[
https://issues.apache.org/jira/browse/CASSANDRA-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1705#comment-1705
]
Ryan Svihla edited comment on CASSANDRA-15557 at 3/5/20, 10:27 AM:
---
Yes silly me, easy enough to prove when setting the timestamps to the same
value the tests still pass, and just to totally eliminate weird side effects,
setting the timestamp in reverse breaks the test as expected so I don't think
there is anything specific about setting the timestamp to any value that makes
it pass.
So I'm not sure I have a good theory as to why breaks in the first place. It
does then act as if somehow the drop is happening "before" the insert. I'll
keep digging thanks for the review.
was (Author: rssvihla):
Yes silly me, easy enough to prove when setting the timestamps to the same
value the tests still pass, and just to totally eliminate weird side effects,
setting the timestamp in reverse breaks the test as expected so I don't think
there is anything specific about setting the timestamp to any value that makes
it pass.
So I'm not sure I have a good theory as to why breaks in the first place. It
does then act as if somehow the drop is happening "before" the add. I'll keep
digging thanks for the review.
> Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest
> testDropListAndAddListWithSameName
> ---
>
> Key: CASSANDRA-15557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15557
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Ryan Svihla
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> https://app.circleci.com/jobs/github/dcapwell/cassandra/482/tests
> {code}
> junit.framework.AssertionFailedError: Invalid value for row 0 column 2
> (mycollection of type list), expected but got <[first element]>
> at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)
> at
> org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15557) Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest testDropListAndAddListWithSameName
[
https://issues.apache.org/jira/browse/CASSANDRA-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051100#comment-17051100
]
Benjamin Lerer edited comment on CASSANDRA-15557 at 3/4/20, 11:51 AM:
--
I believe that the test is fine and should not be changed. The {{DroppedColum}}
should filter out every row with a timestamp less or equal to the
{{DroppedColum.timestamp}}. So even if the row and the {{DroppedColum}} have
the same timestamp, it should be ignored.
Now the interesting question is why if you fix the timestamps for the row and
the dropped column, the test is not failing anymore?
was (Author: blerer):
I believe that the test is fine and should not be changed. The {{DroppedColum}}
should filter out every row with a timestamp less or equal to the
{{DroppedColum.timestamp}}. So even if the row and the {{DroppedColum}} have
the same timestamp something it should be ignored.
Now the interesting question is why if you fix the timestamps for the row and
the dropped column, the test is not failing anymore?
> Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest
> testDropListAndAddListWithSameName
> ---
>
> Key: CASSANDRA-15557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15557
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Ryan Svihla
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> https://app.circleci.com/jobs/github/dcapwell/cassandra/482/tests
> {code}
> junit.framework.AssertionFailedError: Invalid value for row 0 column 2
> (mycollection of type list), expected but got <[first element]>
> at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)
> at
> org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15557) Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest testDropListAndAddListWithSameName
[
https://issues.apache.org/jira/browse/CASSANDRA-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051025#comment-17051025
]
Ryan Svihla edited comment on CASSANDRA-15557 at 3/4/20, 9:22 AM:
--
Not sure what's going on there, maybe some Jira macro run amok. Take 2
Code as it was when the test was originally fixed
[https://github.com/pcmanus/cassandra/commit/020960d727b882349aff00c17a1c46bf3c7f7a24#diff-43b2d530031e2f7ad4286bd05fed4ca0R253]
EDIT dunno why the highlight isn't being added to the
links..[src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java|https://github.com/pcmanus/cassandra/commit/020960d727b882349aff00c17a1c46bf3c7f7a24#diff-43b2d530031e2f7ad4286bd05fed4ca0]
line 253
Change below that I think stripped out the client state timestamp
[https://github.com/apache/cassandra/commit/207c80c1fd63dfbd8ca7e615ec8002ee8983c5d6#diff-50a7bf78bc4f943f42ce60c8768484a6R400]
EDIT (
[src/java/org/apache/cassandra/cql3/statements/schema/AlterTableStatement.java|https://github.com/apache/cassandra/commit/207c80c1fd63dfbd8ca7e615ec8002ee8983c5d6#diff-50a7bf78bc4f943f42ce60c8768484a6]
line 400)
was (Author: rssvihla):
Not sure what's going on there, maybe some Jira macro run amok. Take 2
Code as it was when the test was originally fixed
[https://github.com/pcmanus/cassandra/commit/020960d727b882349aff00c17a1c46bf3c7f7a24#diff-43b2d530031e2f7ad4286bd05fed4ca0R253]
Change below that I think stripped out the client state timestamp
[https://github.com/apache/cassandra/commit/207c80c1fd63dfbd8ca7e615ec8002ee8983c5d6#diff-50a7bf78bc4f943f42ce60c8768484a6R400]
> Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest
> testDropListAndAddListWithSameName
> ---
>
> Key: CASSANDRA-15557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15557
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Ryan Svihla
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> https://app.circleci.com/jobs/github/dcapwell/cassandra/482/tests
> {code}
> junit.framework.AssertionFailedError: Invalid value for row 0 column 2
> (mycollection of type list), expected but got <[first element]>
> at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)
> at
> org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15557) Fix flaky test org.apache.cassandra.cql3.validation.operations.AlterTest testDropListAndAddListWithSameName
[
https://issues.apache.org/jira/browse/CASSANDRA-15557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048991#comment-17048991
]
Ryan Svihla edited comment on CASSANDRA-15557 at 3/2/20 10:14 AM:
--
Seems real as I've been able to reproduce a failure in a few environments just
running the testsuite in a loop, it varies how frequently it happens. So far it
seems the slower the env (fewer cores, less ram, etc) the more likely it
happens.
Using _while ant test -Dtest.name=AlterTest; do :; done_
{{[junit-timeout] DEBUG [PerDiskMemtableFlushWriter_0:1] 2020-03-02
09:39:35,759 Memtable.java:483 - Completed flushing
/home/debian/code/cassandra/build/test/cassandra/data:0/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/na-194-big-Data.db
(0.035KiB) for commitlog position CommitLogPosition(segmentId=1583141964092,
position=7807)}}
{{[junit-timeout] DEBUG [MemtableFlushWriter:2] 2020-03-02 09:39:35,764
ColumnFamilyStore.java:1144 - Flushed to
[BigTableReader(path='/home/debian/code/cassandra/build/test/cassandra/data:0/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/na-194-big-Data.db')]
(1 sstables, 4.830KiB), biggest 4.830KiB, smallest 4.830KiB}}
{{[junit-timeout] - ---}}
{{[junit-timeout] Testcase:
testDropListAndAddListWithSameName(org.apache.cassandra.cql3.validation.operations.AlterTest):
FAILED}}
{{[junit-timeout] Invalid value for row 0 column 2 (mycollection of type
list), expected but got <[first element]>}}
{{[junit-timeout] junit.framework.AssertionFailedError: Invalid value for row
0 column 2 (mycollection of type list), expected but got <[first
element]>}}
{{[junit-timeout] at
org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)}}
{{[junit-timeout] at
org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)}}
{{[junit-timeout] }}
{{[junit-timeout] }}
{{[junit-timeout] Test
org.apache.cassandra.cql3.validation.operations.AlterTest FAILED}}
{{ [delete] Deleting directory
/home/debian/code/cassandra/build/test/cassandra/commitlog:0}}
{{ [delete] Deleting directory
/home/debian/code/cassandra/build/test/cassandra/data:0}}
{{ [delete] Deleting directory
/home/debian/code/cassandra/build/test/cassandra/saved_caches:0}}
{{ [delete] Deleting directory
/home/debian/code/cassandra/build/test/cassandra/hints:0}}
{{[junitreport] Processing
/home/debian/code/cassandra/build/test/TESTS-TestSuites.xml to
/tmp/null291777528}}
{{[junitreport] Loading stylesheet
jar:[file:/usr/share/ant/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl|file:///usr/share/ant/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl]}}
{{[junitreport] Transform time: 483ms}}
{{[junitreport] Deleting: /tmp/null291777528}}{{BUILD FAILED}}
{{/home/debian/code/cassandra/build.xml:1930: The following error occurred
while executing this line:}}
{{/home/debian/code/cassandra/build.xml:1831: Some test(s) failed.}}
was (Author: rssvihla):
Seems real as I've been able to reproduce a failure in a few environments just
running the testsuite in a loop, it varies how frequently it happens. So far it
seems the slower the env the more likely it happens.
Using _while ant test -Dtest.name=AlterTest; do :; done_
{{[junit-timeout] DEBUG [PerDiskMemtableFlushWriter_0:1] 2020-03-02
09:39:35,759 Memtable.java:483 - Completed flushing
/home/debian/code/cassandra/build/test/cassandra/data:0/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/na-194-big-Data.db
(0.035KiB) for commitlog position CommitLogPosition(segmentId=1583141964092,
position=7807)}}
{{[junit-timeout] DEBUG [MemtableFlushWriter:2] 2020-03-02 09:39:35,764
ColumnFamilyStore.java:1144 - Flushed to
[BigTableReader(path='/home/debian/code/cassandra/build/test/cassandra/data:0/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/na-194-big-Data.db')]
(1 sstables, 4.830KiB), biggest 4.830KiB, smallest 4.830KiB}}
{{[junit-timeout] - ---}}
{{[junit-timeout] Testcase:
testDropListAndAddListWithSameName(org.apache.cassandra.cql3.validation.operations.AlterTest):
FAILED}}
{{[junit-timeout] Invalid value for row 0 column 2 (mycollection of type
list), expected but got <[first element]>}}
{{[junit-timeout] junit.framework.AssertionFailedError: Invalid value for row 0
column 2 (mycollection of type list), expected but got <[first
element]>}}
{{[junit-timeout] at
org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1070)}}
{{[junit-timeout] at
org.apache.cassandra.cql3.validation.operations.AlterTest.testDropListAndAddListWithSameName(AlterTest.java:91)}}
{{[junit-timeout] }}
{{[junit-timeout] }}
{{[junit-timeout] Test
org.apache.cassandra.cql3.validation.operations.AlterTest FAILED}}
{{
