[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390116#comment-17390116 ] Johannes Weißl commented on CASSANDRA-16465: [~nvest] now that Cassandra 4.0 is out, it would be very interesting for me if you still have worse performance than 3.11.6 ...? > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: Screenshot from 2021-04-24 20-50-47.png, > image-2021-04-28-02-11-27-298.png > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334381#comment-17334381 ] Nicolai Vest commented on CASSANDRA-16465: -- This is showing the impact of the performance regression for the different versions for a selected TWCS table in our test setup (three C* nodes in one cluster and with just little load). I marked the dates I installed the different versions. For versions 3.11.7 to 3.11.10 SSTable reads are related to the number of total SSTables for that table. For us the impact becomes even more dramatic when the load increases. !image-2021-04-28-02-11-27-298.png|width=486,height=460! [~stefan.miklosovic] Regarding -CASSANDRA-15690- versions affected are [3.0.21|https://issues.apache.org/jira/issues/?jql=project+%3D+CASSANDRA+AND+fixVersion+%3D+3.0.21], [3.11.7|https://issues.apache.org/jira/issues/?jql=project+%3D+CASSANDRA+AND+fixVersion+%3D+3.11.7], [4.0|https://issues.apache.org/jira/issues/?jql=project+%3D+CASSANDRA+AND+fixVersion+%3D+4.0], [4.0-beta1|https://issues.apache.org/jira/issues/?jql=project+%3D+CASSANDRA+AND+fixVersion+%3D+4.0-beta1], ... and newer I guess. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: Screenshot from 2021-04-24 20-50-47.png, > image-2021-04-28-02-11-27-298.png > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332982#comment-17332982 ] Stefan Miklosovic commented on CASSANDRA-16465: --- Hey guys, is this acutally an issue for 4.0 as well? I am just wondering if we see this in 3, why could not it be in 4 as well? > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: Screenshot from 2021-04-24 20-50-47.png > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332621#comment-17332621 ] Jai Bheemsen Rao Dhanwada commented on CASSANDRA-16465: --- Thank you [~nvest] [~AhmedElJAMI] I tried upgrading from 3.11.6 to 3.11.9 and didn't see any perf drop with my application use-case and also the cassandra-stress too. I use LCS in all my tables (system* tables uses default shipped with Cassandra). Do you mind sharing your test scenarios where you see latency, so I can make sure I am not missing anything. [~nvest] please share your finding with 3.11.10 as well, so it will be useful for me to assess which version to upgrade. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: Screenshot from 2021-04-24 20-50-47.png > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331874#comment-17331874 ] Ahmed ELJAMI commented on CASSANDRA-16465: -- [~jaid] in our case, regression was observed for the LCS. The same behaviour as mentioned by [~nvest], SStable reads increases after upgrading from 3.11.3 to 3.11.9. [~cscetbon]We have only tested the 3.11.9 but I have refered to [~nvest] mail's when I opened a new Jira who have observed the regression even with 3.11.7 > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: Screenshot from 2021-04-24 20-50-47.png > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331866#comment-17331866 ] Nicolai Vest commented on CASSANDRA-16465: -- [~jaid] We use only TWCS and LCS with frequent read and write operations in our setup and regression can only be seen for the TWCS. However, STCS might be effected too but you should do your own testing to find out. What we observe are increased SSTable reads, especially for 95th percentile and higher, and increased read latency up to twice as high as before. We observe the same regression for 3.11.7, 3.11.8, 3.11.9. We did not test 3.11.10 yet but could do that easily, if you ask for. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: Screenshot from 2021-04-24 20-50-47.png > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331304#comment-17331304 ] Stefan Miklosovic commented on CASSANDRA-16465: --- I am adding a screenshot of what we see: !Screenshot from 2021-04-24 20-50-47.png! > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > Attachments: Screenshot from 2021-04-24 20-50-47.png > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331302#comment-17331302 ] Stefan Miklosovic commented on CASSANDRA-16465: --- We (Instaclustr) also spotted this issue, for us I think we saw this from 3.11.8 (excluded), but we do not know yet if the cause is in 3.11.9 or 3.11.10. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.x, 4.0.x > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331078#comment-17331078 ] Jai Bheemsen Rao Dhanwada commented on CASSANDRA-16465: --- [~nvest] I am planning to upgrade the Cassandra version from 3.11.6 to 3.11.9 and I don't have any tables with TWCS. have you noticed this performance drop with all the Compaction Strategies or just TWCS? > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330788#comment-17330788 ] Nicolai Vest commented on CASSANDRA-16465: -- I was just curious and did run the comparison already. The branch version provided does not show the performance regression and performs just like 3.11.6. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330400#comment-17330400 ] Nicolai Vest commented on CASSANDRA-16465: -- Great, I'll give it a try during the next week. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[
https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17329095#comment-17329095
]
Paulo Motta commented on CASSANDRA-16465:
-
bq. I do have some doubt if the behavior is related to CASSANDRA-15690 because
the impact on the read latency can be observed immediately after switching the
Cassandra version. No need to wait for partitions being deleted or not.
If this is caused by CASSANDRA-15690 the performance regression should be
observed immediately after switching version, even if there are no partition
deletions.
bq. I have to object. In our setup it does happen with 3.11.7. With 3.11.6
performance is still ok.
In order to rule out CASSANDRA-15690 would you be willing to test a jar of
[this
branch|https://github.com/pauloricardomg/cassandra/tree/cassandra-3.11.7-without-CASSANDRA-15690]
? This is basically 3.11.7 without CASSANDRA-15690. You can clone this repo
and then generate the jar with {{mvn jar}} and then replace your install
{{cassandra-all*.jar}} with the newly created jar. If the performance issue is
fixed than we can safely assume it was caused by the removal of the
optimization from CASSANDRA-15690.
> Increased Read Latency With Cassandra >= 3.11.7
> ---
>
> Key: CASSANDRA-16465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16465
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Ahmed ELJAMI
>Priority: Normal
> Fix For: 3.11.11
>
>
> After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99%
> increased significantly. Getting back to 3.11.3 immediately fixed the issue.
> I have observed "SStable reads" increases after upgrading to 3.11.9.
> The same behavior was observed by some other users:
> [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html]
> According to Paulo Motta's comment, this behavior may be caused by
> https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on
> 3.11.7 and removed an optimization that may cause a correctness issue when
> there are partition deletions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17327163#comment-17327163 ] Nicolai Vest commented on CASSANDRA-16465: -- I have to object. In our setup it does happen with 3.11.7. With 3.11.6 performance is still ok. The tables affected use TWCS and impact can already be observed on our test environment with just little load (3 nodes, 15GB Data with TTL of one week, and ~30k commit logs at the last 2 minutes). > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17327082#comment-17327082 ] Cyril Scetbon commented on CASSANDRA-16465: --- Any news on that ticket. [~AhmedElJAMI] confirmed that it doesn't happen with 3.11.7 and [~skandyla] said the same for 3.11.8, so the title is misleading I think > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292296#comment-17292296 ] Yifan Cai commented on CASSANDRA-16465: --- The regression could be from CASSANDRA-15833, which is included in 3.11.9. There was a wrong condition in ColumnFilter that can lead to digest mismatch and full data requests being made for all reads. So read latency increases There is a fix (ready to commit) in CASSANDRA-16415. You can try to apply the patch or wait for a new 3.11 release. (Sent from my phone) > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292179#comment-17292179 ] Sergey Kandyla commented on CASSANDRA-16465: Also have an increased read latency after upgrading from 3.11.8 to 3.11.9 (no issues with 3.11.7 and 3.11.8) I think it's related to ColumnFilter fix, so posted a detailed explanation here: https://issues.apache.org/jira/browse/CASSANDRA-15977?focusedCommentId=17279741&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17279741 > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290926#comment-17290926 ] Aleksey Yeschenko commented on CASSANDRA-16465: --- [~paulo] Yes, we could easily - whether or not it is the cause of this JIRA. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290916#comment-17290916 ] Nicolai Vest commented on CASSANDRA-16465: -- I do have some doubt if the behavior is related to CASSANDRA-15690 because the impact on the read latency can be observed immediately after switching the Cassandra version. No need to wait for partitions being deleted or not. I did initially report of the problem at the Cassandra user forum and can still reproduce the it at both of our environments, testing (less load) and production (heavy load). During the last months I was not able to investigate any further but being able now to provide further information if required. > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16465) Increased Read Latency With Cassandra >= 3.11.7
[ https://issues.apache.org/jira/browse/CASSANDRA-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290118#comment-17290118 ] Paulo Motta commented on CASSANDRA-16465: - I'm not very familiar with CASSANDRA-15690, but perhaps we could retain the optimization for sstables without partition deletions if we include additional metadata about the presence of partition deletions on ? (cc [~samt], [~aleksey]) > Increased Read Latency With Cassandra >= 3.11.7 > --- > > Key: CASSANDRA-16465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16465 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ahmed ELJAMI >Priority: Normal > Fix For: 3.11.11 > > > After upgrading Cassandra from 3.11.3 to 3.11.9, Cassandra read latency 99% > increased significantly. Getting back to 3.11.3 immediately fixed the issue. > I have observed "SStable reads" increases after upgrading to 3.11.9. > The same behavior was observed by some other users: > [https://www.mail-archive.com/user@cassandra.apache.org/msg61247.html] > According to Paulo Motta's comment, this behavior may be caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on > 3.11.7 and removed an optimization that may cause a correctness issue when > there are partition deletions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
