Re: Is my range read query behaving strange ?

Raised a ticket https://issues.apache.org/jira/browse/CASSANDRA-15159 for the same. On Thu, Jun 13, 2019 at 3:55 PM Laxmikant Upadhyay wrote: > This issue is reproducible on *3.11.4 and 2.1.21* as well. (not yet > checked on 3.0) > > Range query could be : select * from test.table1; (In this

Re: Recover lost node from backup or evict/re-add?

> On Jun 13, 2019, at 6:29 AM, Oleksandr Shulgin > wrote: > >> On Thu, Jun 13, 2019 at 3:16 PM Jeff Jirsa wrote: > >> On Jun 13, 2019, at 2:52 AM, Oleksandr Shulgin >> wrote: >> On Wed, Jun 12, 2019 at 4:02 PM Jeff Jirsa wrote: To avoid violating consistency guarantees, you have to r

Re: Recover lost node from backup or evict/re-add?

On Thu, Jun 13, 2019 at 3:16 PM Jeff Jirsa wrote: > On Jun 13, 2019, at 2:52 AM, Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > On Wed, Jun 12, 2019 at 4:02 PM Jeff Jirsa wrote: > > To avoid violating consistency guarantees, you have to repair the replicas >> while the lost node is

Re: Recover lost node from backup or evict/re-add?

> On Jun 13, 2019, at 2:52 AM, Oleksandr Shulgin > wrote: > >> On Wed, Jun 12, 2019 at 4:02 PM Jeff Jirsa wrote: > >> To avoid violating consistency guarantees, you have to repair the replicas >> while the lost node is down > > How do you suggest to trigger it? Potentially replicas of the

Re: very slow repair

On Thu, Jun 13, 2019 at 2:09 PM Léo FERLIN SUTTON wrote: > Last, but not least: are you using the default number of vnodes, 256? The >> overhead of large number of vnodes (times the number of nodes), can be >> quite significant. We've seen major improvements in repair runtime after >> switching

Re: Speed up compaction

Hi Léo, Major compactions in LCS (and minor as well) are very slow indeed and I'm afraid there's not much you can do to speed things up. There are lots of synchronized sections in the LCS code and it has to do a lot of comparisons between sstables to make sure a partition won't end up in two sstab

Re: Speed up compaction

On Thu, Jun 13, 2019 at 2:44 PM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Thu, Jun 13, 2019 at 2:07 PM Léo FERLIN SUTTON > wrote: > >> >> Overall we are talking about a 1.08TB table, using LCS. >> >> SSTable count: 1047 >>> SSTables in each level: [15/4, 10, 103/100, 918, 0,

Re: Speed up compaction

On Thu, Jun 13, 2019 at 2:07 PM Léo FERLIN SUTTON wrote: > > Overall we are talking about a 1.08TB table, using LCS. > > SSTable count: 1047 >> SSTables in each level: [15/4, 10, 103/100, 918, 0, 0, 0, 0, 0] > > SSTable Compression Ratio: 0.5192269874287099 > > Number of partitions (estimate): 7

Re: very slow repair

> > Last, but not least: are you using the default number of vnodes, 256? The > overhead of large number of vnodes (times the number of nodes), can be > quite significant. We've seen major improvements in repair runtime after > switching from 256 to 16 vnodes on Cassandra version 3.0. Is there

Re: Speed up compaction

On Thu, Jun 13, 2019 at 12:09 PM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Thu, Jun 13, 2019 at 11:28 AM Léo FERLIN SUTTON > wrote: > >> >> ## Cassandra configuration : >> 4 concurrent_compactors >> Current compaction throughput: 150 MB/s >> Concurrent reads/write are both set

Re: Is my range read query behaving strange ?

This issue is reproducible on *3.11.4 and 2.1.21* as well. (not yet checked on 3.0) Range query could be : select * from test.table1; (In this case, Read repair actually sending old mutation to the node which has tombstone ) I also ran normal read and it also returns a row in this case instead

Re: Speed up compaction

On Thu, Jun 13, 2019 at 11:28 AM Léo FERLIN SUTTON wrote: > > ## Cassandra configuration : > 4 concurrent_compactors > Current compaction throughput: 150 MB/s > Concurrent reads/write are both set to 128. > > I have also temporarily stopped every repair operations. > > Any ideas about how I can s

Re: very slow repair

On Thu, Jun 13, 2019 at 10:36 AM R. T. wrote: > > Well, actually by running cfstats I can see that the totaldiskspaceused is > about ~ 1.2 TB per node in the DC1 and ~ 1 TB per node in DC2. DC2 was off > for a while thats why there is a difference in space. > > I am using Cassandra 3.0.6 and > my

Re: Recover lost node from backup or evict/re-add?

On Wed, Jun 12, 2019 at 4:02 PM Jeff Jirsa wrote: > To avoid violating consistency guarantees, you have to repair the replicas > while the lost node is down > How do you suggest to trigger it? Potentially replicas of the primary range for the down node are all over the local DC, so I would go w

Speed up compaction

I am currently noticing very very slow compactions on my cluster and wondering if there is any way to speed things up. Right now I have this compaction currently running : 60c1cfc0-8da7-11e9-bc08-3546c703a280Compaction keyspace1 > table1 8.77 GB 1.71 TB bytes 0.50% The

Re: very slow repair

Hi, Thank you for your reply, Well, actually by running cfstats I can see that the totaldiskspaceused is about ~ 1.2 TB per node in the DC1 and ~ 1 TB per node in DC2. DC2 was off for a while thats why there is a difference in space. I am using Cassandra 3.0.6 and my stream_throughput_outbound

Re: Is my range read query behaving strange ?

HI Michael, Thanks for your reply. I don't think this issue is related to CASSANDRA-12765 as in my case the sstable which has tombstone does not have maxLocalDeletionTime == Integer.MAX_VALUE . I am able to reproduce this issue on 2.1.17 as