Re: Increased read latency with Cassandra >= 3.11.7
Dear community, is someone working on this ticket? This is clearly performance regression and we stuck with 3.11.6 and could not upgrade to latest version. Regards, Maxim. On Mon, Feb 22, 2021 at 10:37 AM Ahmed Eljami wrote: > Hey, > I have created the issue, here => > https://issues.apache.org/jira/browse/CASSANDRA-16465 > > Le ven. 19 févr. 2021 à 10:10, Ahmed Eljami a > écrit : > >> Hi folks, >> >> If this can help, we encountered the same behaviour with 3.11.9. We are >> using LCS. >> After upgrading from 3.11.3 to 3.11.9 in Bench Environnement, Cassandra >> read latency 99% is multiplied by ~3 >> >> We are planning a second test with 3.11.6, I'll send you the results when >> it's done. >> Cheers, >> >> >> >> Le lun. 15 févr. 2021 à 19:55, Jai Bheemsen Rao Dhanwada < >> jaibheem...@gmail.com> a écrit : >> >>> Any update on this? Any idea if this is already tracked under JIRA >>> issue, so we can follow for updates >>> >>> On Wednesday, February 10, 2021, Johannes Weißl wrote: >>> Hi Nico, On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > after upgrading my Cassandra nodes from version 3.11.6 to either > 3.11.7 or 3.11.8 I experience a significant increase in read latency Any update here? Does version 3.11.10 provide any improvement? Thanks, Johannes - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org >> >> -- >> Cordialement; >> >> Ahmed ELJAMI >> > > > -- > Cordialement; > > Ahmed ELJAMI >
Re: Increased read latency with Cassandra >= 3.11.7
Hey, I have created the issue, here => https://issues.apache.org/jira/browse/CASSANDRA-16465 Le ven. 19 févr. 2021 à 10:10, Ahmed Eljami a écrit : > Hi folks, > > If this can help, we encountered the same behaviour with 3.11.9. We are > using LCS. > After upgrading from 3.11.3 to 3.11.9 in Bench Environnement, Cassandra > read latency 99% is multiplied by ~3 > > We are planning a second test with 3.11.6, I'll send you the results when > it's done. > Cheers, > > > > Le lun. 15 févr. 2021 à 19:55, Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> a écrit : > >> Any update on this? Any idea if this is already tracked under JIRA issue, >> so we can follow for updates >> >> On Wednesday, February 10, 2021, Johannes Weißl wrote: >> >>> Hi Nico, >>> >>> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: >>> > after upgrading my Cassandra nodes from version 3.11.6 to either >>> > 3.11.7 or 3.11.8 I experience a significant increase in read latency >>> >>> Any update here? Does version 3.11.10 provide any improvement? >>> >>> Thanks, >>> Johannes >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: user-h...@cassandra.apache.org >>> >>> > > -- > Cordialement; > > Ahmed ELJAMI > -- Cordialement; Ahmed ELJAMI
Re: Increased read latency with Cassandra >= 3.11.7
Hi folks, If this can help, we encountered the same behaviour with 3.11.9. We are using LCS. After upgrading from 3.11.3 to 3.11.9 in Bench Environnement, Cassandra read latency 99% is multiplied by ~3 We are planning a second test with 3.11.6, I'll send you the results when it's done. Cheers, Le lun. 15 févr. 2021 à 19:55, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> a écrit : > Any update on this? Any idea if this is already tracked under JIRA issue, > so we can follow for updates > > On Wednesday, February 10, 2021, Johannes Weißl wrote: > >> Hi Nico, >> >> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: >> > after upgrading my Cassandra nodes from version 3.11.6 to either >> > 3.11.7 or 3.11.8 I experience a significant increase in read latency >> >> Any update here? Does version 3.11.10 provide any improvement? >> >> Thanks, >> Johannes >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> -- Cordialement; Ahmed ELJAMI
Re: Increased read latency with Cassandra >= 3.11.7
Any update on this? Any idea if this is already tracked under JIRA issue, so we can follow for updates On Wednesday, February 10, 2021, Johannes Weißl wrote: > Hi Nico, > > On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > > after upgrading my Cassandra nodes from version 3.11.6 to either > > 3.11.7 or 3.11.8 I experience a significant increase in read latency > > Any update here? Does version 3.11.10 provide any improvement? > > Thanks, > Johannes > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: Increased read latency with Cassandra >= 3.11.7
Hi Nico, On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > after upgrading my Cassandra nodes from version 3.11.6 to either > 3.11.7 or 3.11.8 I experience a significant increase in read latency Any update here? Does version 3.11.10 provide any improvement? Thanks, Johannes - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Re: Increased read latency with Cassandra >= 3.11.7
Hi, I'm not sure if this will help, but I tried today to change one node to 3.11.9 from 3.11.6. We are NOT using TWCS. Very heavy read pattern, almost no writes, with constant performance test load. Cassandra read latency 99% increased significantly, but NOT on the node where I changed version (top line 90ms vs normal 2ms).. Getting back to 3.11.6 immediately fixed the issue. Interesting would be to check all nodes on the new version, but I cannot do such a test for several weeks. At least in a mixed cluster this is no go. Regards, Maxim. [image: image.png]
Antwort: Re: Increased read latency with Cassandra >= 3.11.7
Thanks Paulo! I'll give it a try and let you know. -"Paulo Motta" schrieb: - An: user@cassandra.apache.org Von: "Paulo Motta" Datum: 03.12.2020 13:39 Betreff: Re: Increased read latency with Cassandra >= 3.11.7 As a workaround if your TWCS table is append-only (no updates or deletions) you could try enabling unsafe_aggressive_sstable_expiration (see https://cassandra.apache.org/doc/latest/operating/compaction/twcs.html) so inactive sstables will be expired faster and will not be included during reads. Em qui., 3 de dez. de 2020 às 09:33, Paulo Motta escreveu: I think this could've been caused by https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on 3.11.7 and removed an optimization that may cause a correctness issue when there are partition deletions. I'd suggest you to open an issue at https://issues.apache.org/jira/projects/CASSANDRA/issues to call attention to this regression to see if there's anything that can be done when the workload has no partition deletions. Make sure to include details about your workload (schema, replication settings, CL, query, etc) to facilitate investigation. Em qua., 2 de dez. de 2020 às 12:44, Nicolai Lune Vest escreveu: Hi, we performed a test run with 3.11.9. Unfortunately we do not experience a difference regarding increased read latency at some of our tables. Do others experience the same or similar behavior and probably do have a solution? Kind regards, Nico -"Nicolai Lune Vest" schrieb: - An: user@cassandra.apache.org Von: "Nicolai Lune Vest" Datum: 06.11.2020 09:14 Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7 Hi, thanks for your interests! We use TWCS and LCS and see only TWCS effected by the update regarding higher latencies (as well as increased SSTable reads and bloom filter false positives). We did not update to 3.11.9 as of now but probably will give it a try within the next two or tree weeks. I'll let you know about the results. Cheers Nico -"Jai Bheemsen Rao Dhanwada" schrieb: - An: "user@cassandra.apache.org" Von: "Jai Bheemsen Rao Dhanwada" Datum: 05.11.2020 17:36 Betreff: Increased read latency with Cassandra >= 3.11.7 Hello Nicolai, I am also looking to upgrade to 3.11.8. Is the latency you noticed is only with TWCS? And others looks good? On Thursday, November 5, 2020, Ahmed Eljami wrote: +1 I would also be interested to know if the 3.11.9 provide any improvement for your case. Cheers. Le mer. 4 nov. 2020 à 14:53, Johannes Weißl a écrit : Hi Nico, On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > after upgrading my Cassandra nodes from version 3.11.6 to either > 3.11.7 or 3.11.8 I experience a significant increase in read latency. Have you (or others) any update on that? Does the version 3.11.9 released today provide any improvement? Thanks, Johannes - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org -- Cordialement; Ahmed ELJAMI LANCOM Systems GmbH Adenauerstr. 20 / B2 52146 Würselen Deutschland Tel: +49 2405 49936-0 Fax: +49 2405 49936-99 Web: https://www.lancom-systems.de Geschäftsführer: Ralf Koenzen, Stefan Herrlich Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Increased read latency with Cassandra >= 3.11.7
As a workaround if your TWCS table is append-only (no updates or deletions) you could try enabling unsafe_aggressive_sstable_expiration (see https://cassandra.apache.org/doc/latest/operating/compaction/twcs.html) so inactive sstables will be expired faster and will not be included during reads. Em qui., 3 de dez. de 2020 às 09:33, Paulo Motta escreveu: > I think this could've been caused by > https://issues.apache.org/jira/browse/CASSANDRA-15690 which was > introduced on 3.11.7 and removed an optimization that may cause a > correctness issue when there are partition deletions. I'd suggest you to > open an issue at https://issues.apache.org/jira/projects/CASSANDRA/issues > to call attention to this regression to see if there's anything that can be > done when the workload has no partition deletions. Make sure to include > details about your workload (schema, replication settings, CL, query, etc) > to facilitate investigation. > > Em qua., 2 de dez. de 2020 às 12:44, Nicolai Lune Vest < > nicolai.v...@lancom.de> escreveu: > >> Hi, >> >> we performed a test run with 3.11.9. Unfortunately we do not experience a >> difference regarding increased read latency at some of our tables. >> >> Do others experience the same or similar behavior and probably do have a >> solution? >> >> Kind regards, >> Nico >> >> -"Nicolai Lune Vest" schrieb: ----- >> An: user@cassandra.apache.org >> Von: "Nicolai Lune Vest" >> Datum: 06.11.2020 09:14 >> Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7 >> >> >> >> Hi, >> >> thanks for your interests! >> >> We use TWCS and LCS and see only TWCS effected by the update regarding >> higher latencies (as well as increased SSTable reads and bloom filter false >> positives). >> >> We did not update to 3.11.9 as of now but probably will give it a try >> within the next two or tree weeks. I'll let you know about the results. >> >> Cheers >> Nico >> >> -"Jai Bheemsen Rao Dhanwada" schrieb: - >> An: "user@cassandra.apache.org" >> Von: "Jai Bheemsen Rao Dhanwada" >> Datum: 05.11.2020 17:36 >> Betreff: Increased read latency with Cassandra >= 3.11.7 >> >> Hello Nicolai, >> >> >> I am also looking to upgrade to 3.11.8. Is the latency you noticed is >> only with TWCS? And others looks good? >> >> On Thursday, November 5, 2020, Ahmed Eljami >> wrote: >> >> +1 >> I would also be interested to know if the 3.11.9 provide any improvement >> for your case. >> Cheers. >> >> >> >> Le mer. 4 nov. 2020 à 14:53, Johannes Weißl a écrit : >> >> Hi Nico, >> >> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: >> > after upgrading my Cassandra nodes from version 3.11.6 to either >> > 3.11.7 or 3.11.8 I experience a significant increase in read latency. >> >> Have you (or others) any update on that? Does the version 3.11.9 >> released today provide any improvement? >> >> Thanks, >> Johannes >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> >> >> -- >> >> Cordialement; >> >> >> Ahmed ELJAMI >> >> >> >> >> >> >> >> LANCOM Systems GmbH >> Adenauerstr. 20 / B2 >> 52146 Würselen >> Deutschland >> >> Tel: +49 2405 49936-0 >> Fax: +49 2405 49936-99 >> >> Web: https://www.lancom-systems.de >> >> >> >> Geschäftsführer: Ralf Koenzen, Stefan Herrlich >> Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 >> >> >> >> >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >>
Re: Increased read latency with Cassandra >= 3.11.7
I think this could've been caused by https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on 3.11.7 and removed an optimization that may cause a correctness issue when there are partition deletions. I'd suggest you to open an issue at https://issues.apache.org/jira/projects/CASSANDRA/issues to call attention to this regression to see if there's anything that can be done when the workload has no partition deletions. Make sure to include details about your workload (schema, replication settings, CL, query, etc) to facilitate investigation. Em qua., 2 de dez. de 2020 às 12:44, Nicolai Lune Vest < nicolai.v...@lancom.de> escreveu: > Hi, > > we performed a test run with 3.11.9. Unfortunately we do not experience a > difference regarding increased read latency at some of our tables. > > Do others experience the same or similar behavior and probably do have a > solution? > > Kind regards, > Nico > > -"Nicolai Lune Vest" schrieb: - > An: user@cassandra.apache.org > Von: "Nicolai Lune Vest" > Datum: 06.11.2020 09:14 > Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7 > > > > Hi, > > thanks for your interests! > > We use TWCS and LCS and see only TWCS effected by the update regarding > higher latencies (as well as increased SSTable reads and bloom filter false > positives). > > We did not update to 3.11.9 as of now but probably will give it a try > within the next two or tree weeks. I'll let you know about the results. > > Cheers > Nico > > -"Jai Bheemsen Rao Dhanwada" schrieb: - > An: "user@cassandra.apache.org" > Von: "Jai Bheemsen Rao Dhanwada" > Datum: 05.11.2020 17:36 > Betreff: Increased read latency with Cassandra >= 3.11.7 > > Hello Nicolai, > > > I am also looking to upgrade to 3.11.8. Is the latency you noticed is only > with TWCS? And others looks good? > > On Thursday, November 5, 2020, Ahmed Eljami > wrote: > > +1 > I would also be interested to know if the 3.11.9 provide any improvement > for your case. > Cheers. > > > > Le mer. 4 nov. 2020 à 14:53, Johannes Weißl a écrit : > > Hi Nico, > > On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > > after upgrading my Cassandra nodes from version 3.11.6 to either > > 3.11.7 or 3.11.8 I experience a significant increase in read latency. > > Have you (or others) any update on that? Does the version 3.11.9 > released today provide any improvement? > > Thanks, > Johannes > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > > > > -- > > Cordialement; > > > Ahmed ELJAMI > > > > > > > > LANCOM Systems GmbH > Adenauerstr. 20 / B2 > 52146 Würselen > Deutschland > > Tel: +49 2405 49936-0 > Fax: +49 2405 49936-99 > > Web: https://www.lancom-systems.de > > > > Geschäftsführer: Ralf Koenzen, Stefan Herrlich > Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 > > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Antwort: Increased read latency with Cassandra >= 3.11.7
Hi, we performed a test run with 3.11.9. Unfortunately we do not experience a difference regarding increased read latency at some of our tables. Do others experience the same or similar behavior and probably do have a solution? Kind regards, Nico -"Nicolai Lune Vest" schrieb: - An: user@cassandra.apache.org Von: "Nicolai Lune Vest" Datum: 06.11.2020 09:14 Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7 Hi, thanks for your interests! We use TWCS and LCS and see only TWCS effected by the update regarding higher latencies (as well as increased SSTable reads and bloom filter false positives). We did not update to 3.11.9 as of now but probably will give it a try within the next two or tree weeks. I'll let you know about the results. Cheers Nico -"Jai Bheemsen Rao Dhanwada" schrieb: - An: "user@cassandra.apache.org" Von: "Jai Bheemsen Rao Dhanwada" Datum: 05.11.2020 17:36 Betreff: Increased read latency with Cassandra >= 3.11.7 Hello Nicolai, I am also looking to upgrade to 3.11.8. Is the latency you noticed is only with TWCS? And others looks good? On Thursday, November 5, 2020, Ahmed Eljami wrote: +1 I would also be interested to know if the 3.11.9 provide any improvement for your case. Cheers. Le mer. 4 nov. 2020 à 14:53, Johannes Weißl a écrit : Hi Nico, On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > after upgrading my Cassandra nodes from version 3.11.6 to either > 3.11.7 or 3.11.8 I experience a significant increase in read latency. Have you (or others) any update on that? Does the version 3.11.9 released today provide any improvement? Thanks, Johannes - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org -- Cordialement; Ahmed ELJAMI LANCOM Systems GmbH Adenauerstr. 20 / B2 52146 Würselen Deutschland Tel: +49 2405 49936-0 Fax: +49 2405 49936-99 Web: https://www.lancom-systems.de Geschäftsführer: Ralf Koenzen, Stefan Herrlich Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Antwort: Increased read latency with Cassandra >= 3.11.7
Hi, thanks for your interests! We use TWCS and LCS and see only TWCS effected by the update regarding higher latencies (as well as increased SSTable reads and bloom filter false positives). We did not update to 3.11.9 as of now but probably will give it a try within the next two or tree weeks. I'll let you know about the results. Cheers Nico -"Jai Bheemsen Rao Dhanwada" schrieb: - An: "user@cassandra.apache.org" Von: "Jai Bheemsen Rao Dhanwada" Datum: 05.11.2020 17:36 Betreff: Increased read latency with Cassandra >= 3.11.7 Hello Nicolai, I am also looking to upgrade to 3.11.8. Is the latency you noticed is only with TWCS? And others looks good? On Thursday, November 5, 2020, Ahmed Eljami wrote: +1 I would also be interested to know if the 3.11.9 provide any improvement for your case. Cheers. Le mer. 4 nov. 2020 à 14:53, Johannes Weißl a écrit : Hi Nico, On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > after upgrading my Cassandra nodes from version 3.11.6 to either > 3.11.7 or 3.11.8 I experience a significant increase in read latency. Have you (or others) any update on that? Does the version 3.11.9 released today provide any improvement? Thanks, Johannes - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org -- Cordialement; Ahmed ELJAMI LANCOM Systems GmbH Adenauerstr. 20 / B2 52146 Würselen Deutschland Tel: +49 2405 49936-0 Fax: +49 2405 49936-99 Web: https://www.lancom-systems.de Geschäftsführer: Ralf Koenzen, Stefan Herrlich Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
Increased read latency with Cassandra >= 3.11.7
Hello Nicolai, I am also looking to upgrade to 3.11.8. Is the latency you noticed is only with TWCS? And others looks good? On Thursday, November 5, 2020, Ahmed Eljami wrote: > +1 > I would also be interested to know if the 3.11.9 provide any improvement > for your case. > Cheers. > > Le mer. 4 nov. 2020 à 14:53, Johannes Weißl a écrit : > >> Hi Nico, >> >> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: >> > after upgrading my Cassandra nodes from version 3.11.6 to either >> > 3.11.7 or 3.11.8 I experience a significant increase in read latency. >> >> Have you (or others) any update on that? Does the version 3.11.9 >> released today provide any improvement? >> >> Thanks, >> Johannes >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> > > -- > Cordialement; > > Ahmed ELJAMI >
Re: Increased read latency with Cassandra >= 3.11.7
+1 I would also be interested to know if the 3.11.9 provide any improvement for your case. Cheers. Le mer. 4 nov. 2020 à 14:53, Johannes Weißl a écrit : > Hi Nico, > > On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > > after upgrading my Cassandra nodes from version 3.11.6 to either > > 3.11.7 or 3.11.8 I experience a significant increase in read latency. > > Have you (or others) any update on that? Does the version 3.11.9 > released today provide any improvement? > > Thanks, > Johannes > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- Cordialement; Ahmed ELJAMI
Re: Increased read latency with Cassandra >= 3.11.7
Hi Nico, On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote: > after upgrading my Cassandra nodes from version 3.11.6 to either > 3.11.7 or 3.11.8 I experience a significant increase in read latency. Have you (or others) any update on that? Does the version 3.11.9 released today provide any improvement? Thanks, Johannes - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Antwort: Re: Re: Increased read latency with Cassandra >= 3.11.7
Hi Maxim, unfortunately no. We decided to stay at 3.11.6 for production until we found a solution because our system is latency sensitive too. >From what I can see the most impact on read-latency is on the tables using time windowed compaction strategy. Tables with e.g. leveled compaction strategy are not affected that much. I'm suspecting some change in the behavior of the bloom filter. Bloom filter false positive rate hast increased too with 3.11.7, probably causing more SSTables to be read on request unnecessarily. Reading more SSTables probably causes a higher read-latency. However, the way leveled compaction strategy is organizing the SSTables bloom filter changes might not have such a big impact on the number of SSTables to be read on request. In case you have a test environment you could try the version update and compare bloom filter and SSTable reads on your tables. Would be interested to see your results too. Regards Nico -"Maxim Parkachov" schrieb: - An: user@cassandra.apache.org Von: "Maxim Parkachov" Datum: 23.09.2020 08:15 Betreff: Re: Re: Increased read latency with Cassandra >= 3.11.7 Hi Nico, we wanted to upgrade to 3.11.8 (from .6), but now I'm very concerned as our load is mostly read-only and very latency sensitive. Did you figure out the reason for such behaviour with the new version ? Regards, Maxim. LANCOM Systems GmbH Adenauerstr. 20 / B2 52146 Würselen Deutschland Tel: +49 2405 49936-0 Fax: +49 2405 49936-99 Web: https://www.lancom-systems.de Geschäftsführer: Ralf Koenzen, Stefan Herrlich Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
Re: Re: Increased read latency with Cassandra >= 3.11.7
Hi Nico, we wanted to upgrade to 3.11.8 (from .6), but now I'm very concerned as our load is mostly read-only and very latency sensitive. Did you figure out the reason for such behaviour with the new version ? Regards, Maxim. On Tue, Sep 15, 2020 at 5:24 AM Sagar Jambhulkar wrote: > That is odd. I thought maybe lower version had a bigger custom cache or > some other setting missed in the upgrade. > > On Mon, 14 Sep 2020, 23:00 Nicolai Lune Vest, > wrote: > >> While cache configuration stays the same, cache hit rate for both, the >> chunk cache and the key cache are worse (but still ok) after the update. >> You can see the change in the attached screenshot (update was performed at >> September the 7th ~4pm). >> >> @Sagar Any specific reason you are suspecting the cache? >> >> >> >> -"Sagar Jambhulkar" schrieb: - >> An: user@cassandra.apache.org >> Von: "Sagar Jambhulkar" >> Datum: 14.09.2020 16:25 >> Betreff: Re: Increased read latency with Cassandra >= 3.11.7 >> >> >> Maybe compare the cache size see if anything different in two versions? >> >> >> On Mon, 14 Sep 2020, 19:21 Nicolai Lune Vest, >> wrote: >> >> >> Dear Cassandra community, >> >> after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7 >> or 3.11.8 I experience a significant increase in read latency. >> With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read >> latency increases to ~2.9 ms, almost 10 times worse! This behavior applies >> to all my environments, testing and productive. >> >> I do also observe that the "bloom-filter false positive rate" and >> "SSTable reads" increases significantly with version >= 3.11.7 >> >> I'm wondering if someone experienced the same problem when updating >> Cassandra from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out. >> >> I already tried nodetool scrub, sstablescrub, manual compaction, and >> repair on selected tables without impact. When downgrading back to 3.11.6 >> read latency recovers immediately. >> >> My update routine is very simple. I'm just stopping the service, install >> the new version (using apt), and start the service again. One node after >> the other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a >> patch update and I expect this routine should do well, but maybe I'm >> missing some important steps here. >> >> I can provide more information about my system or configuration if >> required. >> >> Appreciate your support! >> >> Kind regards >> Nico >> >> >> >> >> >> >> >> LANCOM Systems GmbH >> Adenauerstr. 20 / B2 >> 52146 Würselen >> Deutschland >> >> Tel: +49 2405 49936-0 >> Fax: +49 2405 49936-99 >> >> Web: https://www.lancom-systems.de >> >> >> >> Geschäftsführer: Ralf Koenzen, Stefan Herrlich >> Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 >> >> >> >> >> >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: Re: Increased read latency with Cassandra >= 3.11.7
That is odd. I thought maybe lower version had a bigger custom cache or some other setting missed in the upgrade. On Mon, 14 Sep 2020, 23:00 Nicolai Lune Vest, wrote: > While cache configuration stays the same, cache hit rate for both, the > chunk cache and the key cache are worse (but still ok) after the update. > You can see the change in the attached screenshot (update was performed at > September the 7th ~4pm). > > @Sagar Any specific reason you are suspecting the cache? > > > > -"Sagar Jambhulkar" schrieb: - > An: user@cassandra.apache.org > Von: "Sagar Jambhulkar" > Datum: 14.09.2020 16:25 > Betreff: Re: Increased read latency with Cassandra >= 3.11.7 > > > Maybe compare the cache size see if anything different in two versions? > > > On Mon, 14 Sep 2020, 19:21 Nicolai Lune Vest, > wrote: > > > Dear Cassandra community, > > after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7 or > 3.11.8 I experience a significant increase in read latency. > With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read > latency increases to ~2.9 ms, almost 10 times worse! This behavior applies > to all my environments, testing and productive. > > I do also observe that the "bloom-filter false positive rate" and "SSTable > reads" increases significantly with version >= 3.11.7 > > I'm wondering if someone experienced the same problem when updating > Cassandra from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out. > > I already tried nodetool scrub, sstablescrub, manual compaction, and > repair on selected tables without impact. When downgrading back to 3.11.6 > read latency recovers immediately. > > My update routine is very simple. I'm just stopping the service, install > the new version (using apt), and start the service again. One node after > the other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a > patch update and I expect this routine should do well, but maybe I'm > missing some important steps here. > > I can provide more information about my system or configuration if > required. > > Appreciate your support! > > Kind regards > Nico > > > > > > > > LANCOM Systems GmbH > Adenauerstr. 20 / B2 > 52146 Würselen > Deutschland > > Tel: +49 2405 49936-0 > Fax: +49 2405 49936-99 > > Web: https://www.lancom-systems.de > > > > Geschäftsführer: Ralf Koenzen, Stefan Herrlich > Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 > > > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Increased read latency with Cassandra >= 3.11.7
Maybe compare the cache size see if anything different in two versions? On Mon, 14 Sep 2020, 19:21 Nicolai Lune Vest, wrote: > Dear Cassandra community, > > after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7 > or 3.11.8 I experience a significant increase in read latency. > With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read > latency increases to ~2.9 ms, almost 10 times worse! This behavior > applies to all my environments, testing and productive. > > I do also observe that the "bloom-filter false positive rate" and "SSTabl > e reads" increases significantly with version >= 3.11.7 > > I'm wondering if someone experienced the same problem when updating Cassandra > from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out. > > I already tried nodetool scrub, sstablescrub, manual compaction, and re > pair on selected tables without impact. When downgrading back to 3.11.6 > read latency recovers immediately. > > My update routine is very simple. I'm just stopping the service, install the > new version (using apt), and start the service again. One node after the > other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a patch > update and I expect this routine should do well, but maybe I'm missing > some important steps here. > > I can provide more information about my system or configuration if > required. > > Appreciate your support! > > Kind regards > Nico > > > -- > > > > LANCOM Systems GmbH > Adenauerstr. 20 / B2 > 52146 Würselen > Deutschland > > Tel: +49 2405 49936-0 > Fax: +49 2405 49936-99 > > Web: https://www.lancom-systems.de > > > > Geschäftsführer: Ralf Koenzen, Stefan Herrlich > Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976 > -- > > > > >
Increased read latency with Cassandra >= 3.11.7
Dear Cassandra community, after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7 or 3.11.8 I experience a significant increase in read latency. With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read latency increases to ~2.9 ms, almost 10 times worse! This behavior applies to all my environments, testing and productive. I do also observe that the "bloom-filter false positive rate" and "SSTable reads" increases significantly with version >= 3.11.7 I'm wondering if someone experienced the same problem when updating Cassandra from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out. I already tried nodetool scrub, sstablescrub, manual compaction, and re pair on selected tables without impact. When downgrading back to 3.11.6 read latency recovers immediately. My update routine is very simple. I'm just stopping the service, install the new version (using apt), and start the service again. One node after the other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a patch update and I expect this routine should do well, but maybe I'm missing some important steps here. I can provide more information about my system or configuration if required. Appreciate your support! Kind regards Nico LANCOM Systems GmbH Adenauerstr. 20 / B2 52146 Würselen Deutschland Tel: +49 2405 49936-0 Fax: +49 2405 49936-99 Web: https://www.lancom-systems.de Geschäftsführer: Ralf Koenzen, Stefan Herrlich Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
