Re: A Question About Hints

[shalom sagges]

Everyone really should move off of the 2.x versions just like you are doing.

Tell me about it... But since there are a lot of groups involved, these
things take time unfortunately.


Thanks for your assistance Kenneth


On Mon, Mar 4, 2019 at 11:04 PM Kenneth Brotman
 wrote:

> Since you are in the process of upgrading, I’d do nothing on the settings
> right now.  But if you wanted to do something on the settings in the
> meantime, based on my read of the information available, I’d maybe double
> the default settings. The upgrade will help a lot of things as you know.
>
>
>
> Everyone really should move off of the 2.x versions just like you are
> doing.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 12:34 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> See my comments inline.
>
>
>
> Do the 8 nodes clusters have the problem too?
>
> Yes
>
>
>
> To the same extent?
>
> It depends on the throughput, but basically the smaller clusters get low
> throughput, so the problem is naturally smaller.
>
>
>
> Is it any cluster across multi-DC’s?
>
> Yes
>
>
>
> Do all the clusters use nodes with similar specs?
>
> All nodes have similar specs within a cluster but different specs on
> different clusters.
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
> Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the
> process of upgrading to 3.11.4
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> I guess you can say it about almost any parameter you change :)
>
>
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
> The entire article is very good, but I wonder if it's still valid since it
> was created around 4 years ago.
>
>
>
> Thanks!
>
>
>
>
>
>
>
>
>
> On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman 
> wrote:
>
> Makes sense  If you have time and don’t mind, could you answer the
> following:
>
> Do the 8 nodes clusters have the problem too?
>
> To the same extent?
>
> Is it just the clusters with the large node count?
>
> Is it any cluster across multi-DC’s?
>
> Do all the clusters use nodes with similar specs?
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
>
>
>
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 11:04 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> It varies...
>
> Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
>
> Both settings are on default.
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
> That's the plan, but I thought I might first get some valuable information
> from someone in the community that has already experienced in this type of
> change.
>
>
>
> Thanks!
>
>
>
> On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman
>  wrote:
>
> It sounds like your use case might be appropriate for tuning those two
> settings some.
>
>
>
> How many nodes are in the cluster?
>
> Are both settings definitely on the default values currently?
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
>
>
> Then of course share your results with us.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 9:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> Hi Kenneth,
>
>
>
> The concern is that in some cases, hints accumulate on nodes, and it takes
> a while until they are delivered (multi DCs).
>
> I see that whenever there are  a lot of hints in play,like after a rolling
> restart, the cluster works harder. That's why I want to decrease the hints
> delivery time.
>
> I didn't want to change the configuration blindly and thought the
> community might have some experience on this subject.
>
>
>
> I went over the cassandra.yaml file but didn't find any information on
> optimizing these attributes, just that the max_throttle is divided between
> nodes in the cluster and that I should increase the
> max_hints_delivery_threads because I have multi-dc deployments.
>
>
>
> # Maximum throttle in KBs per second, per delivery thread.  This will be
> # reduced proportionally to the number of nodes in the cluster.  (If there
> # are two nodes in the cluster, each delivery thread will use the maximum

RE: A Question About Hints

[Kenneth Brotman]

Since you are in the process of upgrading, I’d do nothing on the settings right 
now.  But if you wanted to do something on the settings in the meantime, based 
on my read of the information available, I’d maybe double the default settings. 
The upgrade will help a lot of things as you know.

 

Everyone really should move off of the 2.x versions just like you are doing.

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 12:34 PM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

See my comments inline. 

 

Do the 8 nodes clusters have the problem too?

Yes

 

To the same extent?  

It depends on the throughput, but basically the smaller clusters get low 
throughput, so the problem is naturally smaller. 

 

Is it any cluster across multi-DC’s?

Yes

 

Do all the clusters use nodes with similar specs?

All nodes have similar specs within a cluster but different specs on different 
clusters. 

 

The version of Cassandra you are on can make a difference.  What version are 
you on?

Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the process of 
upgrading to 3.11.4 

 

Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: 
https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions you can 
get into trouble if you go to fast or two slow?

I guess you can say it about almost any parameter you change :)

 

BTW, I thought the comments at the end of the article you mentioned were really 
good.

The entire article is very good, but I wonder if it's still valid since it was 
created around 4 years ago. 

 

Thanks!

 

 

 

 

On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman  
wrote:

Makes sense  If you have time and don’t mind, could you answer the following:

Do the 8 nodes clusters have the problem too? 

To the same extent?  

Is it just the clusters with the large node count? 

Is it any cluster across multi-DC’s?

Do all the clusters use nodes with similar specs?

 

The version of Cassandra you are on can make a difference.  What version are 
you on?

 

Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: 
https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions you can 
get into trouble if you go to fast or two slow?

BTW, I thought the comments at the end of the article you mentioned were really 
good.

 

 

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 11:04 AM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

It varies...

Some clusters have 48 nodes, others 24 nodes and some 8 nodes. 

Both settings are on default. 

 

I’d try making a single conservative change to one or the other, measure and 
reassess.  Then do same to other setting.

That's the plan, but I thought I might first get some valuable information from 
someone in the community that has already experienced in this type of change. 

 

Thanks!

 

On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman  
wrote:

It sounds like your use case might be appropriate for tuning those two settings 
some. 

 

How many nodes are in the cluster?

Are both settings definitely on the default values currently?

 

I’d try making a single conservative change to one or the other, measure and 
reassess.  Then do same to other setting.

 

Then of course share your results with us.

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 9:54 AM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

Hi Kenneth, 

 

The concern is that in some cases, hints accumulate on nodes, and it takes a 
while until they are delivered (multi DCs). 

I see that whenever there are  a lot of hints in play,like after a rolling 
restart, the cluster works harder. That's why I want to decrease the hints 
delivery time. 

I didn't want to change the configuration blindly and thought the community 
might have some experience on this subject. 

 

I went over the cassandra.yaml file but didn't find any information on 
optimizing these attributes, just that the max_throttle is divided between 
nodes in the cluster and that I should increase the max_hints_delivery_threads 
because I have multi-dc deployments.   

 

# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously)
hinted_handoff_throttle_in_kb: 1024

# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2

 

 

Thanks for your help!

 

 

On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman  
wrote:

What is the concern?  Why are you looking there?  

Re: A Question About Hints

[shalom sagges]

See my comments inline.

Do the 8 nodes clusters have the problem too?
Yes

To the same extent?

It depends on the throughput, but basically the smaller clusters get low
throughput, so the problem is naturally smaller.


Is it any cluster across multi-DC’s?

Yes


Do all the clusters use nodes with similar specs?

All nodes have similar specs within a cluster but different specs on
different clusters.


The version of Cassandra you are on can make a difference.  What version
are you on?

Currently I'm on various versions, 2.0.14, 2.1.15 and 3.0.12. In the
process of upgrading to 3.11.4


Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
you can get into trouble if you go to fast or two slow?

I guess you can say it about almost any parameter you change :)


BTW, I thought the comments at the end of the article you mentioned were
really good.

The entire article is very good, but I wonder if it's still valid since it
was created around 4 years ago.


Thanks!





On Mon, Mar 4, 2019 at 9:37 PM Kenneth Brotman 
wrote:

> Makes sense.  If you have time and don’t mind, could you answer the
> following:
>
> Do the 8 nodes clusters have the problem too?
>
> To the same extent?
>
> Is it just the clusters with the large node count?
>
> Is it any cluster across multi-DC’s?
>
> Do all the clusters use nodes with similar specs?
>
>
>
> The version of Cassandra you are on can make a difference.  What version
> are you on?
>
>
>
> Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video
> at: https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions
> you can get into trouble if you go to fast or two slow?
>
> BTW, I thought the comments at the end of the article you mentioned were
> really good.
>
>
>
>
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 11:04 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> It varies...
>
> Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
>
> Both settings are on default.
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
> That's the plan, but I thought I might first get some valuable information
> from someone in the community that has already experienced in this type of
> change.
>
>
>
> Thanks!
>
>
>
> On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman
>  wrote:
>
> It sounds like your use case might be appropriate for tuning those two
> settings some.
>
>
>
> How many nodes are in the cluster?
>
> Are both settings definitely on the default values currently?
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
>
>
> Then of course share your results with us.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 9:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> Hi Kenneth,
>
>
>
> The concern is that in some cases, hints accumulate on nodes, and it takes
> a while until they are delivered (multi DCs).
>
> I see that whenever there are  a lot of hints in play,like after a rolling
> restart, the cluster works harder. That's why I want to decrease the hints
> delivery time.
>
> I didn't want to change the configuration blindly and thought the
> community might have some experience on this subject.
>
>
>
> I went over the cassandra.yaml file but didn't find any information on
> optimizing these attributes, just that the max_throttle is divided between
> nodes in the cluster and that I should increase the
> max_hints_delivery_threads because I have multi-dc deployments.
>
>
>
> # Maximum throttle in KBs per second, per delivery thread.  This will be
> # reduced proportionally to the number of nodes in the cluster.  (If there
> # are two nodes in the cluster, each delivery thread will use the maximum
> # rate; if there are three, each will throttle to half of the maximum,
> # since we expect two nodes to be delivering hints simultaneously)
> hinted_handoff_throttle_in_kb: 1024
>
> # Number of threads with which to deliver hints;
> # Consider increasing this number when you have multi-dc deployments, since
> # cross-dc handoff tends to be slower
> max_hints_delivery_threads: 2
>
>
>
>
>
> Thanks for your help!
>
>
>
>
>
> On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman
>  wrote:
>
> What is the concern?  Why are you looking there?  The casssandra.yml file
> has some notes about it.  Did you read them?
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 7:22 AM
> *To:* user@cassandra.apache.org
> *Subject:* A Question About Hints
>
>
>
> Hi All,
>
>
>
> Does anyone know what is the most optimal hints configuration (multiple
> DCs) in terms of
>
> max_hints_delivery_threads and hinted_handoff_throt

RE: A Question About Hints

[Kenneth Brotman]

Makes sense.  If you have time and don’t mind, could you answer the following:

Do the 8 nodes clusters have the problem too? 

To the same extent?  

Is it just the clusters with the large node count? 

Is it any cluster across multi-DC’s?

Do all the clusters use nodes with similar specs?

 

The version of Cassandra you are on can make a difference.  What version are 
you on?

 

Did you see Edward Capriolo’s presentation at 26:19 into the YouTube video at: 
https://www.youtube.com/watch?v=uN4FtAjYmLU where he briefly mentions you can 
get into trouble if you go to fast or two slow?



BTW, I thought the comments at the end of the article you mentioned were really 
good.

 

 

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 11:04 AM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

It varies...

Some clusters have 48 nodes, others 24 nodes and some 8 nodes. 

Both settings are on default. 

 

I’d try making a single conservative change to one or the other, measure and 
reassess.  Then do same to other setting.

That's the plan, but I thought I might first get some valuable information from 
someone in the community that has already experienced in this type of change. 

 

Thanks!

 

On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman  
wrote:

It sounds like your use case might be appropriate for tuning those two settings 
some. 

 

How many nodes are in the cluster?

Are both settings definitely on the default values currently?

 

I’d try making a single conservative change to one or the other, measure and 
reassess.  Then do same to other setting.

 

Then of course share your results with us.

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 9:54 AM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

Hi Kenneth, 

 

The concern is that in some cases, hints accumulate on nodes, and it takes a 
while until they are delivered (multi DCs). 

I see that whenever there are  a lot of hints in play,like after a rolling 
restart, the cluster works harder. That's why I want to decrease the hints 
delivery time. 

I didn't want to change the configuration blindly and thought the community 
might have some experience on this subject. 

 

I went over the cassandra.yaml file but didn't find any information on 
optimizing these attributes, just that the max_throttle is divided between 
nodes in the cluster and that I should increase the max_hints_delivery_threads 
because I have multi-dc deployments.   

 

# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously)
hinted_handoff_throttle_in_kb: 1024

# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2

 

 

Thanks for your help!

 

 

On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman  
wrote:

What is the concern?  Why are you looking there?  The casssandra.yml file has 
some notes about it.  Did you read them?

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 7:22 AM
To: user@cassandra.apache.org
Subject: A Question About Hints

 

Hi All,

 

Does anyone know what is the most optimal hints configuration (multiple DCs) in 
terms of 

max_hints_delivery_threads and hinted_handoff_throttle_in_kb? 

If it's different for various use cases, is there a rule of thumb I can work 
with?

 

I found this post but it's quite old:

http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/

 

Thanks!

Re: A Question About Hints

[shalom sagges]

It varies...
Some clusters have 48 nodes, others 24 nodes and some 8 nodes.
Both settings are on default.

I’d try making a single conservative change to one or the other, measure
and reassess.  Then do same to other setting.

That's the plan, but I thought I might first get some valuable information
from someone in the community that has already experienced in this type of
change.


Thanks!


On Mon, Mar 4, 2019 at 8:27 PM Kenneth Brotman 
wrote:

> It sounds like your use case might be appropriate for tuning those two
> settings some.
>
>
>
> How many nodes are in the cluster?
>
> Are both settings definitely on the default values currently?
>
>
>
> I’d try making a single conservative change to one or the other, measure
> and reassess.  Then do same to other setting.
>
>
>
> Then of course share your results with us.
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 9:54 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: A Question About Hints
>
>
>
> Hi Kenneth,
>
>
>
> The concern is that in some cases, hints accumulate on nodes, and it takes
> a while until they are delivered (multi DCs).
>
> I see that whenever there are  a lot of hints in play,like after a rolling
> restart, the cluster works harder. That's why I want to decrease the hints
> delivery time.
>
> I didn't want to change the configuration blindly and thought the
> community might have some experience on this subject.
>
>
>
> I went over the cassandra.yaml file but didn't find any information on
> optimizing these attributes, just that the max_throttle is divided between
> nodes in the cluster and that I should increase the
> max_hints_delivery_threads because I have multi-dc deployments.
>
>
>
> # Maximum throttle in KBs per second, per delivery thread.  This will be
> # reduced proportionally to the number of nodes in the cluster.  (If there
> # are two nodes in the cluster, each delivery thread will use the maximum
> # rate; if there are three, each will throttle to half of the maximum,
> # since we expect two nodes to be delivering hints simultaneously.)
> hinted_handoff_throttle_in_kb: 1024
>
> # Number of threads with which to deliver hints;
> # Consider increasing this number when you have multi-dc deployments, since
> # cross-dc handoff tends to be slower
> max_hints_delivery_threads: 2
>
>
>
>
>
> Thanks for your help!
>
>
>
>
>
> On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman
>  wrote:
>
> What is the concern?  Why are you looking there?  The casssandra.yml file
> has some notes about it.  Did you read them?
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 7:22 AM
> *To:* user@cassandra.apache.org
> *Subject:* A Question About Hints
>
>
>
> Hi All,
>
>
>
> Does anyone know what is the most optimal hints configuration (multiple
> DCs) in terms of
>
> max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
>
> If it's different for various use cases, is there a rule of thumb I can
> work with?
>
>
>
> I found this post but it's quite old:
>
> http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/
>
>
>
> Thanks!
>
>

RE: A Question About Hints

[Kenneth Brotman]

It sounds like your use case might be appropriate for tuning those two settings 
some. 

 

How many nodes are in the cluster?

Are both settings definitely on the default values currently?

 

I’d try making a single conservative change to one or the other, measure and 
reassess.  Then do same to other setting.

 

Then of course share your results with us.

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 9:54 AM
To: user@cassandra.apache.org
Subject: Re: A Question About Hints

 

Hi Kenneth, 

 

The concern is that in some cases, hints accumulate on nodes, and it takes a 
while until they are delivered (multi DCs). 

I see that whenever there are  a lot of hints in play,like after a rolling 
restart, the cluster works harder. That's why I want to decrease the hints 
delivery time. 

I didn't want to change the configuration blindly and thought the community 
might have some experience on this subject. 

 

I went over the cassandra.yaml file but didn't find any information on 
optimizing these attributes, just that the max_throttle is divided between 
nodes in the cluster and that I should increase the max_hints_delivery_threads 
because I have multi-dc deployments.   

 

# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously.)
hinted_handoff_throttle_in_kb: 1024

# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2

 

 

Thanks for your help!

 

 

On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman  
wrote:

What is the concern?  Why are you looking there?  The casssandra.yml file has 
some notes about it.  Did you read them?

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 7:22 AM
To: user@cassandra.apache.org
Subject: A Question About Hints

 

Hi All,

 

Does anyone know what is the most optimal hints configuration (multiple DCs) in 
terms of 

max_hints_delivery_threads and hinted_handoff_throttle_in_kb? 

If it's different for various use cases, is there a rule of thumb I can work 
with?

 

I found this post but it's quite old:

http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/

 

Thanks!

Re: A Question About Hints

[shalom sagges]

Hi Kenneth,

The concern is that in some cases, hints accumulate on nodes, and it takes
a while until they are delivered (multi DCs).
I see that whenever there are  a lot of hints in play,like after a rolling
restart, the cluster works harder. That's why I want to decrease the hints
delivery time.
I didn't want to change the configuration blindly and thought the community
might have some experience on this subject.

I went over the cassandra.yaml file but didn't find any information on
optimizing these attributes, just that the max_throttle is divided between
nodes in the cluster and that I should increase the
max_hints_delivery_threads because I have multi-dc deployments.

# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously.)
hinted_handoff_throttle_in_kb: 1024

# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2


Thanks for your help!


On Mon, Mar 4, 2019 at 6:44 PM Kenneth Brotman 
wrote:

> What is the concern?  Why are you looking there?  The casssandra.yml file
> has some notes about it.  Did you read them?
>
>
>
> *From:* shalom sagges [mailto:shalomsag...@gmail.com]
> *Sent:* Monday, March 04, 2019 7:22 AM
> *To:* user@cassandra.apache.org
> *Subject:* A Question About Hints
>
>
>
> Hi All,
>
>
>
> Does anyone know what is the most optimal hints configuration (multiple
> DCs) in terms of
>
> max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
>
> If it's different for various use cases, is there a rule of thumb I can
> work with?
>
>
>
> I found this post but it's quite old:
>
> http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/
>
>
>
> Thanks!
>

RE: A Question About Hints

[Kenneth Brotman]

What is the concern?  Why are you looking there?  The casssandra.yml file has 
some notes about it.  Did you read them?

 

From: shalom sagges [mailto:shalomsag...@gmail.com] 
Sent: Monday, March 04, 2019 7:22 AM
To: user@cassandra.apache.org
Subject: A Question About Hints

 

Hi All,

 

Does anyone know what is the most optimal hints configuration (multiple DCs) in 
terms of 

max_hints_delivery_threads and hinted_handoff_throttle_in_kb? 

If it's different for various use cases, is there a rule of thumb I can work 
with?

 

I found this post but it's quite old:

http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/

 

Thanks!

A Question About Hints

[shalom sagges]

Hi All,

Does anyone know what is the most optimal hints configuration (multiple
DCs) in terms of
max_hints_delivery_threads and hinted_handoff_throttle_in_kb?
If it's different for various use cases, is there a rule of thumb I can
work with?

I found this post but it's quite old:
http://www.uberobert.com/bandwidth-cassandra-hinted-handoff/

Thanks!

RE: MV's stuck in build state

[Kenneth Brotman]

Hi Dipan,

 

What is the state of the dev and production clusters now?  

How big is the production cluster?

How many nodes on each cluster spin out of control?

 

On the production cluster, the data is 67 MB, you'd have use a value at
least twice that size as the commitlog_segment_size.  Of course you don't
want to leave it really high if you do change it.  

On the dev cluster the data is 18 MB, you used a value way over twice size
when you bumped the commitlog_sement_size to 128 MB, but in doing so you are
wasting a lot of memory capacity of course, but you say it didn't fix the
problem on that cluster, so. is the message you are getting reflecting this
change by showing  to be 128 MB now or is it still a different value?

 

Is there another problem too perhaps you were running low on capacity on the
node?  Could low capacity be a problem on either cluster? 

 

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Monday, March 04, 2019 12:52 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Hello Kenneth,

 

Apologies for the late reply.

 

1) On production the value of x was 67 MB and y was 16 MV as value of
commitlog_segment_size_in_mb is 32.

2) On Dev the value of x was 18 MB and y was 16 MV as value of
commitlog_segment_size_in_mb was 32 initially. I had bumped up the value of
commitlog_segment_size_in_mb to 128 when the node eventually crashed.

3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did
try "nodetool stop" and "nodetool stop VIEW_BUILD".

 

Thanks,

Dipan Shah

  _  

From: Kenneth Brotman 
Sent: Friday, March 1, 2019 8:19 PM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state 

 

Dipan,

 

On your production cluster, when you were first getting the "Mutation of 
bytes ." message, what was the value of x and y?

How about when you got the message on the Dev Cluster, what was the value of
x and y in that message?

On the Dev cluster, did you try going into JMX and directly hitting the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation?

 

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Hello Kenneth,

 

Thanks for replying.

 

I had actually tried this on a Dev environment earlier and it caused the
node to spin out of control. I'll explain what I did over there:

 

1) Found "Mutation of  bytes is too large for the maxiumum size of "
and thus increased the value of "commitlog_segment_size_in_mb" to 64

2) This worked for a few minutes and again the view started failing when it
hit the new limits and the messages now were "Mutation of  bytes is too
large for the maxiumum size of 2*"

3) So just to try I increased the value to 128

4) Now after this change the node started crashing as soon as I brought the
service online. I was not able to recover even after restoring the value of
"commitlog_segment_size_in_mb" to 32

 

Now there is a key differences to that issue and what I am facing currently:

 

The views were not dropped on the earlier environment whereas I have already
dropped the view on the current environment (and cant experiment much as the
current environment is in production).

 

I know this is a bit tricky but I'm pretty much stuck over here and thinking
of finding a non-problem creating solution over here.

 

Thanks,

Dipan Shah

  _  

From: Kenneth Brotman 
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state 

 

Hi Dipan,

 

Did you try following the advice in the referenced DataStax article called
Mutation of
  bytes is too large for the
maximum size of  as suggested in the stackoverflow.com post you cited?

 

Kenneth Brotman

 

From: Dipan Shah [mailto:dipan@hotmail.com] 
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Forgot to add version info. This is on 37.

 

[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]

 

Thanks,

Dipan Shah

  _  

From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state 

 

Hello All,

 

I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".

 



 

I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus"

 

I have tried "nodetool stop VIEW_BUILD" as suggested here:
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vie
w-build and have also reboot a few nodes in th

Re: MV's stuck in build state

[Dipan Shah]

Hello Kenneth,

Apologies for the late reply.

1) On production the value of x was 67 MB and y was 16 MV as value of 
commitlog_segment_size_in_mb is 32.
2) On Dev the value of x was 18 MB and y was 16 MV as value of 
commitlog_segment_size_in_mb was 32 initially. I had bumped up the value of 
commitlog_segment_size_in_mb to 128 when the node eventually crashed.
3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did 
try "nodetool stop" and "nodetool stop VIEW_BUILD".


Thanks,

Dipan Shah


From: Kenneth Brotman 
Sent: Friday, March 1, 2019 8:19 PM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state


Dipan,



On your production cluster, when you were first getting the “Mutation of  
bytes …” message, what was the value of x and y?

How about when you got the message on the Dev Cluster, what was the value of x 
and y in that message?

On the Dev cluster, did you try going into JMX and directly hitting the 
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction operation?





From: Dipan Shah [mailto:dipan@hotmail.com]
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state



Hello Kenneth,



Thanks for replying.



I had actually tried this on a Dev environment earlier and it caused the node 
to spin out of control. I'll explain what I did over there:



1) Found "Mutation of  bytes is too large for the maxiumum size of " and 
thus increased the value of "commitlog_segment_size_in_mb" to 64

2) This worked for a few minutes and again the view started failing when it hit 
the new limits and the messages now were "Mutation of  bytes is too large 
for the maxiumum size of 2*"

3) So just to try I increased the value to 128

4) Now after this change the node started crashing as soon as I brought the 
service online. I was not able to recover even after restoring the value of 
"commitlog_segment_size_in_mb" to 32



Now there is a key differences to that issue and what I am facing currently:



The views were not dropped on the earlier environment whereas I have already 
dropped the view on the current environment (and cant experiment much as the 
current environment is in production).



I know this is a bit tricky but I'm pretty much stuck over here and thinking of 
finding a non-problem creating solution over here.



Thanks,

Dipan Shah



From: Kenneth Brotman 
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state



Hi Dipan,



Did you try following the advice in the referenced DataStax article called 
Mutation of  bytes is too large for the maximum size of 

 as suggested in the stackoverflow.com post you cited?



Kenneth Brotman



From: Dipan Shah [mailto:dipan@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state



Forgot to add version info. This is on 3.7.



[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]



Thanks,

Dipan Shah



From: Dipan Shah 
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state



Hello All,



I have a few MV's that are stuck in build state because of a bad schema design 
and thus getting a lot of messages like this "Mutation xxx is too large for 
maximum size of 16.000MiB".



[cid:image001.png@01D4CFFA.539C41B0]



I have dropped those MV's and I can no longer see their schema in the keyspace. 
But they are visible under "system.views_build_in_progress" and "nodetool 
viewbuildstatus".



I have tried "nodetool stop VIEW_BUILD" as suggested here: 
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build
 and have also reboot a few nodes in the cluster. This has also not helped.



Is there anything else that can be done over here?

[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]


Stop Cassandra Materialized View Build - Stack 
Overflow

Its not documented, but nodetool stop actually takes any compaction type, not 
just the ones listed (which the view build is one of). So you can simply: 
nodetool stop VIEW_BUILD Or you can hit JMX directly with the 
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction 
operation.. All thats really gonna do is set a flag for the view builder to 
stop on its next loop.

stackoverflow.com






Thanks,

Dipan Shah