Re: insertion rate change after stopping node

[Osman Yozgatlıoğlu]

Thanks, i'll check it out.
Regards,
Osman

On Wed, 5 Aug 2020 at 21:15, Jeff Jirsa  wrote:
>
> Potentially slow machine / bad cpu / bad disk / bad dimms.
>
>
> On Wed, Aug 5, 2020 at 10:50 AM Osman Yozgatlıoğlu 
>  wrote:
>>
>> Hello,
>>
>> I have 2 DC 8+7 nodes Cassandra cluster with RF:2:2.
>> Insertion rate dropped recently.
>> If I stop an exact node, insertion rate goes normal.
>> What should be the problem?
>>
>> Regards,
>> Osman
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Cassandra.Link Knowledge Base - v. 0.7 - Jobs Section

[Rahul Singh]

Folks,

Quick update. We added a jobs section that's aggregating jobs from a few
different job markets but only those that relate to Cassandra. Right now
it's just a "lucene" based filter from a larger data set, but our team is
working to put some ML action to do NLP based classification.

Would love to get your feedback. It's not indeed but it is interesting to
see how people are using Cassandra out there and who.

Best,


rahul.xavier.si...@gmail.com

http://cassandra.link

Re: insertion rate change after stopping node

[Jeff Jirsa]

Potentially slow machine / bad cpu / bad disk / bad dimms.


On Wed, Aug 5, 2020 at 10:50 AM Osman Yozgatlıoğlu <
osman.yozgatlio...@gmail.com> wrote:

> Hello,
>
> I have 2 DC 8+7 nodes Cassandra cluster with RF:2:2.
> Insertion rate dropped recently.
> If I stop an exact node, insertion rate goes normal.
> What should be the problem?
>
> Regards,
> Osman
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

insertion rate change after stopping node

[Osman Yozgatlıoğlu]

Hello,

I have 2 DC 8+7 nodes Cassandra cluster with RF:2:2.
Insertion rate dropped recently.
If I stop an exact node, insertion rate goes normal.
What should be the problem?

Regards,
Osman

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Re: streaming stuck on joining a node with TBs of data

[onmstester onmstester]

OK. Thanks 




I'm using STCS.



Anyway, IMHO, this is one of the main bottlenecks for using big/dense node in 
Cassandra (which reduces cluster size and data center costs) and it could be 
almost solved (at least for me), if we could reduce number of sstables at 
receiver side (either by sending bigger sstables at sending side or by merging 
sstables in memtable at receiving side)



(Just fixed a wrong word in my previous question)


 On Wed, 05 Aug 2020 10:02:51 +0430 onmstester onmstester 
 wrote 


OK. Thanks

I'm using STCS.

Anyway, IMHO, this is one of the main bottlenecks for using big/dense node in 
Cassandra (which reduces cluster size and data center costs) and it could be 
almost solved (at least for me), if we could eliminate number of sstables at 
receiver side (either by sending bigger sstables at sending side or by merging 
sstables in memtable at receiving side)


Sent using https://www.zoho.com/mail/




 On Mon, 03 Aug 2020 19:17:33 +0430 Jeff Jirsa  
wrote 


Memtable really isn't involved here, each data file is copied over as-is and 
turned into a new data file, it doesn't read into the memtable (though it does 
deserialize and re-serialize, which temporarily has it in memory, but isn't in 
the memtable itself).



You can cut down on the number of data files copied in by using fewer vnodes, 
or by changing your compaction parameters (e.g. if you're using LCS, change 
sstable size from 160M to something higher), but there's no magic to join / 
compact those data files on the sending side before sending.




On Mon, Aug 3, 2020 at 4:15 AM onmstester onmstester 
 wrote:

IMHO (reading system.log) each streamed-in file from any node would be write 
down as a separate sstable to the disk and won't be wait in memtable until 
enough amount of memtable has been created inside memory, so there would be 
more compactions because of multiple small sstables. Is there any configuration 
in cassandra to force streamed-in to pass memtable-sstable cycle, to have 
bigger sstables at first place?



Sent using https://www.zoho.com/mail/






 Forwarded message 
From: onmstester onmstester 
To: "user"
Date: Sun, 02 Aug 2020 08:35:30 +0430
Subject: Re: streaming stuck on joining a node with TBs of data
 Forwarded message 



Thanks Jeff,



Already used netstats and it only shows that streaming from a single node 
remained and stuck and bunch of dropped messages, next time i will check 
tpstats too.

Currently i stopped the joining/stucked node, make the auto_bootstrap false and 
started the node and its UN now, is this OK too?



What about streaming tables one by one, any idea?



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 21:44:09 +0430 Jeff Jirsa  
wrote 





Nodetool tpstats and netstats should give you a hint why it’s not joining



If you don’t care about consistency and you just want it joined in its current 
form (which is likely strictly incorrect but I get it), “nodetool disablegossip 
&& nodetool enablegossip” in rapid succession (must be less than 30 seconds in 
between commands) will PROBABLY change it from joining to normal (unclean, 
unsafe, do this at your own risk).





On Jul 31, 2020, at 11:46 PM, onmstester onmstester 
 wrote:





No Secondary index, No SASI, No materialized view



Sent using https://www.zoho.com/mail/






 On Sat, 01 Aug 2020 11:02:54 +0430 Jeff Jirsa  
wrote 



Are there secondary indices involved? 



On Jul 31, 2020, at 10:51 PM, onmstester onmstester 
 wrote:





Hi,



I'm going to join multiple new nodes to already existed and running cluster. 
Each node should stream in >2TB of data, and it took a few days (with 500Mb 
streaming) to almost get finished. But it stuck on streaming-in from one final 
node, but i can not see any bottleneck on any side (source or destination 
node), the only problem is 400 pending compactions on joining node, which i 
disabled auto_compaction, but no improvement.



1. How can i safely stop streaming/joining the new node and make it UN, then 
run repair on the node?

2. On bootstrap a new node, multiple tables would be streamed-in simultaneously 
and i think that this would increase number of compactions in compare with a 
scenario that "the joining node first stream-in one table then switch to 
another one and etc". Am i right and this would decrease compactions? If so, is 
there a config or hack in cassandra to force that?





Sent using https://www.zoho.com/mail/