[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521284#comment-17521284
]
Yifan Cai commented on CASSANDRA-17217:
---
Starting commit
CI Results (pending):
||Branch||Source||Circle CI||
|trunk|[branch|https://github.com/yifan-c/cassandra/tree/commit_remote_branch/CASSANDRA-17217-trunk-1710429A-1133-4BAE-8E13-41C9CCD27A4F]|[build|https://app.circleci.com/pipelines/github/yifan-c/cassandra?branch=commit_remote_branch%2FCASSANDRA-17217-trunk-1710429A-1133-4BAE-8E13-41C9CCD27A4F]|
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521273#comment-17521273
]
Yifan Cai commented on CASSANDRA-17217:
---
Hi [~sathyakplm], the question is not related with the ticket. For those
questions, I'd encourage to post them on the user mailing list / slack instead.
See [https://cassandra.apache.org/_/community.html] on how to join, just in
case not known yet.
A brief answer to your question... I do not think that _at the node level_
Cassandra ensures monotonic read, i.e. R1 reads state prior to the one R2
reads. The read commands are served in parallel within a node.
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520544#comment-17520544
]
Andres de la Peña commented on CASSANDRA-17217:
---
The changes look good to me, here are 1000 successful iterations on CircleCI:
*
[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1468/workflows/60322f29-e684-4171-b81b-b6856accc9c6]
*
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1468/workflows/30f2a556-9008-400a-aba4-a2d5947a1f9f]
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519801#comment-17519801
]
Bernardo Botella Corbi commented on CASSANDRA-17217:
Besides the question for [~yifanc], can please someone take a look for a second
+1? Thanks!
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515324#comment-17515324
]
Sathyanarayanan Saravanamuthu commented on CASSANDRA-17217:
---
Hi [~yifanc] , one small query regarding the inflight read instructions. Let's
say I make first read query to the cluster and the nodes 0 and 1 respond with
failure and the client returns failure. The read to node 2 is in-flight; let's
call this read as R1. Once the first call returns, I make second read query to
the cluster. In the second query also nodes 0 and 1 respond with failure and
the client returns failure. The read to node 2 is in-flight; let's call this
read as R2. Does Cassandra assure that R1 is served first before R2?
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510918#comment-17510918
]
Bernardo Botella Corbi commented on CASSANDRA-17217:
I just updated the PR following [~yifanc] suggestion and splitting the tests.
I've also run the tests over 1000 times with no errors (similarly to what I did
with my previous commit). Hopefully this fixes the flakiness.
Thanks a lot for checking [~adelapena]
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509087#comment-17509087
]
Yifan Cai commented on CASSANDRA-17217:
---
The flaky test is likely cause by the prior read query fails earlier than the
completion of the read command execution in each replica. A read is failed as
soon as one of the read command fails, and a failure is returned to client.
However, there could be still inflight read commands in the other replicas.
They will eventually fail and increment the counter.
For the test in question, once the size tracking is turned off, new queries
should not update the metrics, but the flakiness is cause by the prior read.
There is nothing wrong with the behavior. Probably either wait for some time
between each read query to make sure all replicas complete their read command.
Or even more clean, test without changing the size tracking in between. It
basically breaks the existing test into 2, read with size tracking and w/o.
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17508753#comment-17508753
]
Andres de la Peña commented on CASSANDRA-17217:
---
[~bernardo.botella] thanks for looking into this. Unfortunately, it seems that
[this repeated
run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1373/workflows/c64980f8-4f6a-439c-ac20-3f376e341f2c/jobs/13945]
of {{LocalReadSizeWarningTest}} with 500 iterations still hits the same
failure twice.
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17508453#comment-17508453
]
Bernardo Botella Corbi commented on CASSANDRA-17217:
It looks like the previous assertWarnAborts method call may create a warn in
the metrics under certain conditions. Making sure that we have a histogram
checkpoint after disabling the warn tracking ensures that we are in the
expected state before executing the non failing query.
||PR||
|[trunk\|https://github.com/apache/cassandra/pull/1509]|
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502501#comment-17502501
]
Bernardo Botella Corbi commented on CASSANDRA-17217:
Thanks for the heads up! Will take that into account from now on :)
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502499#comment-17502499
]
Ekaterina Dimitrova commented on CASSANDRA-17217:
-
Thanks [~bernardo.botella], I noticed your assignment and that the ticket was
not reflecting the right status, so I did it for you.
Also, I see people who start contributing to the project, working on Jira
tickets but not moving them to the right status as they think only committers
can do that. I guess we have to update our docs as committer is just a
contributor with write permissions to the Cassandra repo. Normally whoever
works on the ticket propagates the ticket to respective status to ensure the
right people are triggered (in case a committer involvement is needed or just
patch available for review for example.)
Thank you for your work, let us know if you have any questions :)
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17217) LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502494#comment-17502494
]
Bernardo Botella Corbi commented on CASSANDRA-17217:
Will start taking a look
> LocalReadSizeWarningTest#failThresholdSinglePartition is flaky
> --
>
> Key: CASSANDRA-17217
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17217
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Andres de la Peña
>Assignee: Bernardo Botella Corbi
>Priority: Normal
> Fix For: 4.x
>
>
> The test
> {{org.apache.cassandra.distributed.test.trackwarnings.LocalReadSizeWarningTest#failThresholdSinglePartition}}
> seems to be flaky.
> It was discovered during CASSANDRA-17195 with [this CI
> run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1217/workflows/b366e352-0862-485a-acdc-5b75fe1ef575/jobs/11188].
> The failure can be reproduced running the test repeatedly for trunk, as it
> can be seen
> [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1219/workflows/a0c1fb7f-8a07-4fcf-97a4-7caf207dad78].
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
