Re: MUTATION messages were dropped in last 5000 ms for cross nodetimeout

[Jeff Jirsa]

No

-- 
Jeff Jirsa


> On Aug 4, 2017, at 7:18 PM, Peng Xiao <2535...@qq.com> wrote:
> 
> hi，
> 
> Does message drop mean data loss?
> Thanks
> 
> 
> -- Original --
> From: Akhil Mehra 
> Date: 周五,8月 4,2017 16:00
> To: user 
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross 
> nodetimeout
> 
> Glad I could be of help :)
> 
> Hopefully the partition size resize goes smoothly.
> 
> Regards,
> Akhil
> 
>> On 4/08/2017, at 5:41 AM, ZAIDI, ASAD A  wrote:
>> 
>> Hi Akhil,
>>  
>> Thank you for your reply.
>>  
>> I kept testing different timeout numbers over last week and eventually 
>> settled at setting *_request_timeout_in_ms parameters at 1.5minutes for 
>> coordinator wait time. That is the number where I donot see any dropped 
>> mutations. 
>>  
>> Also asked developers to tweak data model where we saw bunch of tables with 
>> really large partition size , some are ranging  Partition-key size around 
>> ~6.6GB.. we’re now working to reduce the partition size of the tables. I am 
>> hoping corrected data model will help reduce coordinator wait time (get back 
>> to default number!)  again.
>>  
>> Thank again/Asad
>>  
>> From: Akhil Mehra [mailto:akhilme...@gmail.com] 
>> Sent: Friday, July 21, 2017 4:24 PM
>> To: user@cassandra.apache.org
>> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
>> timeout
>>  
>> Hi Asad,
>>  
>> The 5000 ms is not configurable 
>> (https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423).
>>  This just the time after which the number of dropped messages are reported. 
>> Thus dropped messages are reported every 5000ms. 
>>  
>> If you are looking to tweak the number of ms after which a message is 
>> considered dropped then you need to use the write_request_timeout_in_ms.  
>> The write_request_timeout_in_ms 
>> (http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html)
>>  can be used to increase the mutation timeout. By default it is set to 
>> 2000ms.
>>  
>> I hope that helps.
>>  
>> Regards,
>> Akhil
>>  
>>  
>> On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A  wrote:
>>  
>> Hi Akhil,
>>  
>> Thank you for your reply. Previously, I did ‘tune’ various timeouts – 
>> basically increased them a bit but none of those parameter listed in the 
>> link matches with that “were dropped in last 5000 ms”.
>> I was wondering from where that [5000ms] number is coming from when,  like I 
>> mentioned before, none of any timeout parameter settings matches that #!
>>  
>> Load is intermittently high but again cpu queue length never goes beyond 
>> medium depth. I wonder if there is some internal limit that I’m still not 
>> aware of.
>>  
>> Thanks/Asad
>>  
>>  
>> From: Akhil Mehra [mailto:akhilme...@gmail.com] 
>> Sent: Thursday, July 20, 2017 3:47 PM
>> To: user@cassandra.apache.org
>> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
>> timeout
>>  
>> Hi Asad,
>>  
>> http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped
>>  
>> As mentioned in the link above this is a load shedding mechanism used by 
>> Cassandra.
>>  
>> Is you cluster under heavy load?
>>  
>> Regards,
>> Akhil
>>  
>>  
>> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A  wrote:
>>  
>> Hello Folks –
>>  
>> I’m using apache-cassandra 2.2.8.
>>  
>> I see many messages like below in my system.log file. In Cassandra.yaml file 
>> [ cross_node_timeout: true] is set and NTP server is also running correcting 
>> clock drift on 16node cluster. I do not see pending or blocked HintedHandoff 
>>  in tpstats output though there are bunch of MUTATIONS dropped observed.
>>  
>> 
>> INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
>> MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
>> 2152 for cross node timeout
>> 
>>  
>> I’m seeking help here if you please let me know what I need to check in 
>> order to address these cross node timeouts.
>>  
>> Thank you,
>> Asad
> 

Re: MUTATION messages were dropped in last 5000 ms for cross nodetimeout

[Peng Xiao]

hi??


Does message drop mean data loss?

Thanks


-- Original --
From: Akhil Mehra 
Date: ,8?? 4,2017 16:00
To: user 
Subject: Re: MUTATION messages were dropped in last 5000 ms  for cross 
nodetimeout



Glad I could be of help :)

Hopefully the partition size resize goes smoothly.


Regards,
Akhil

On 4/08/2017, at 5:41 AM, ZAIDI, ASAD A  wrote:

Hi Akhil,
 
Thank you for your reply.
 
I kept testing different timeout numbers over last week and eventually settled 
at setting *_request_timeout_in_ms parameters at 1.5minutes for coordinator 
wait time. That is the number where I donot see any dropped mutations. 
 
Also asked developers to tweak data model where we saw bunch of tables with 
really large partition size , some are ranging  Partition-key size around 
~6.6GB.. we??re now working to reduce the partition size of the tables. I am 
hoping corrected data model will help reduce coordinator wait time (get back to 
default number!)  again.
 
Thank again/Asad
 
From: Akhil Mehra [mailto:akhilme...@gmail.com] 
Sent: Friday, July 21, 2017 4:24 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
timeout


 
Hi Asad,
 

The 5000 ms is not configurable 
(https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423).
 This just the time after which the number of dropped messages are reported. 
Thus dropped messages are reported every 5000ms. 

 

If you are looking to tweak the number of ms after which a message is 
considered dropped then you need to use the write_request_timeout_in_ms.  The 
write_request_timeout_in_ms 
(http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html)
 can be used to increase the mutation timeout. By default it is set to 2000ms.

 

I hope that helps.

 

Regards,

Akhil

 

 

On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A  wrote:

 
Hi Akhil,

 

Thank you for your reply. Previously, I did ??tune?? various timeouts ?C 
basically increased them a bit but none of those parameter listed in the link 
matches with that ??were dropped in last 5000 ms??.

I was wondering from where that [5000ms] number is coming from when,  like I 
mentioned before, none of any timeout parameter settings matches that #!

 

Load is intermittently high but again cpu queue length never goes beyond medium 
depth. I wonder if there is some internal limit that I??m still not aware of.

 

Thanks/Asad

 

 

From: Akhil Mehra [mailto:akhilme...@gmail.com] 
Sent: Thursday, July 20, 2017 3:47 PM
To: user@cassandra.apache.org
Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
timeout



 

Hi Asad,


 


http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped

 


As mentioned in the link above this is a load shedding mechanism used by 
Cassandra.


 


Is you cluster under heavy load?


 


Regards,


Akhil


 

 

On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A  wrote:


 

Hello Folks ?C


 


I??m using apache-cassandra 2.2.8.


 


I see many messages like below in my system.log file. In Cassandra.yaml file [ 
cross_node_timeout: true] is set and NTP server is also running correcting 
clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
in tpstats output though there are bunch of MUTATIONS dropped observed.


 





INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
2152 for cross node timeout





 


I??m seeking help here if you please let me know what I need to check in order 
to address these cross node timeouts.


 


Thank you,


Asad

Re: Upgrading from Cassandra 3.9 to Cassandra 3.11

[Igor Leão]

Hi Petrus,

Thanks!
I see it's considered a good practice to always have snapshots, but I
couldn't quite understand if it is possible to add new nodes running
Cassandra 3.11 into a Cassandra 3.9 cluster.
I tried doing so this morning, using
`-Dcassandra.force_3_0_protocol_version=true`,
and everything worked fine, but these new nodes were with 90%+ of cpu
usage, so I removed them. Do you have any idea why this happened? Should I
always upgrade every machine locally, keeping its data, instead of adding
new nodes with a newer version? Best,

2017-08-04 16:01 GMT-03:00 Petrus Gomes :

> Hi Igor,
> Have a look into that document. ;-)
> https://github.com/apache/cassandra/blob/cassandra-3.11/NEWS.txt
> Thanks,
> Petrus Silva
>
> On Fri, Aug 4, 2017 at 10:15 AM, Igor Leão  wrote:
>
>> Hi there,
>>
>> Is it possible to add a Cassandra 3.11 node into a Cassandra 3.9 cluster?
>> I wish I could upgrade this cluster without uninstalling and installing
>> Cassandra on existing nodes, just by adding and removing nodes.
>>
>> Thanks in advance!
>>
>
>


-- 
Igor Leão  Site Reliability Engineer

Mobile: +55 81 99727-1083 
Skype: *igorvpcleao*
Office: +55 81 4042-9757 
Website: inlocomedia.com 
[image: inlocomedia]

 [image: LinkedIn]

 [image: Facebook]  [image: Twitter]

Re: Upgrading from Cassandra 3.9 to Cassandra 3.11

[Petrus Gomes]

Hi Igor,
Have a look into that document. ;-)
https://github.com/apache/cassandra/blob/cassandra-3.11/NEWS.txt
Thanks,
Petrus Silva

On Fri, Aug 4, 2017 at 10:15 AM, Igor Leão  wrote:

> Hi there,
>
> Is it possible to add a Cassandra 3.11 node into a Cassandra 3.9 cluster?
> I wish I could upgrade this cluster without uninstalling and installing
> Cassandra on existing nodes, just by adding and removing nodes.
>
> Thanks in advance!
>

Upgrading from Cassandra 3.9 to Cassandra 3.11

[Igor Leão]

Hi there,

Is it possible to add a Cassandra 3.11 node into a Cassandra 3.9 cluster?
I wish I could upgrade this cluster without uninstalling and installing
Cassandra on existing nodes, just by adding and removing nodes.

Thanks in advance!

Re: MUTATION messages were dropped in last 5000 ms for cross node timeout

[Akhil Mehra]

Glad I could be of help :)

Hopefully the partition size resize goes smoothly.

Regards,
Akhil

> On 4/08/2017, at 5:41 AM, ZAIDI, ASAD A  wrote:
> 
> Hi Akhil,
>  
> Thank you for your reply.
>  
> I kept testing different timeout numbers over last week and eventually 
> settled at setting *_request_timeout_in_ms parameters at 1.5minutes for 
> coordinator wait time. That is the number where I donot see any dropped 
> mutations. 
>  
> Also asked developers to tweak data model where we saw bunch of tables with 
> really large partition size , some are ranging  Partition-key size around 
> ~6.6GB.. we’re now working to reduce the partition size of the tables. I am 
> hoping corrected data model will help reduce coordinator wait time (get back 
> to default number!)  again.
>  
> Thank again/Asad
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com] 
> Sent: Friday, July 21, 2017 4:24 PM
> To: user@cassandra.apache.org
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
> timeout
>  
> Hi Asad,
>  
> The 5000 ms is not configurable 
> (https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/net/MessagingService.java#L423
>  
> ).
>  This just the time after which the number of dropped messages are reported. 
> Thus dropped messages are reported every 5000ms. 
>  
> If you are looking to tweak the number of ms after which a message is 
> considered dropped then you need to use the write_request_timeout_in_ms.  The 
> write_request_timeout_in_ms 
> (http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html
>  
> )
>  can be used to increase the mutation timeout. By default it is set to 2000ms.
>  
> I hope that helps.
>  
> Regards,
> Akhil
>  
>  
> On 22/07/2017, at 2:46 AM, ZAIDI, ASAD A  > wrote:
>  
> Hi Akhil,
>  
> Thank you for your reply. Previously, I did ‘tune’ various timeouts – 
> basically increased them a bit but none of those parameter listed in the link 
> matches with that “were dropped in last 5000 ms”.
> I was wondering from where that [5000ms] number is coming from when,  like I 
> mentioned before, none of any timeout parameter settings matches that #!
>  
> Load is intermittently high but again cpu queue length never goes beyond 
> medium depth. I wonder if there is some internal limit that I’m still not 
> aware of.
>  
> Thanks/Asad
>  
>  
> From: Akhil Mehra [mailto:akhilme...@gmail.com ] 
> Sent: Thursday, July 20, 2017 3:47 PM
> To: user@cassandra.apache.org 
> Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node 
> timeout
>  
> Hi Asad,
>  
> http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped 
> 
>  
> As mentioned in the link above this is a load shedding mechanism used by 
> Cassandra.
>  
> Is you cluster under heavy load?
>  
> Regards,
> Akhil
>  
>  
> On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A  > wrote:
>  
> Hello Folks –
>  
> I’m using apache-cassandra 2.2.8.
>  
> I see many messages like below in my system.log file. In Cassandra.yaml file 
> [ cross_node_timeout: true] is set and NTP server is also running correcting 
> clock drift on 16node cluster. I do not see pending or blocked HintedHandoff  
> in tpstats output though there are bunch of MUTATIONS dropped observed.
>  
> 
> INFO  [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - 
> MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 
> 2152 for cross node timeout
> 
>  
> I’m seeking help here if you please let me know what I need to check in order 
> to address these cross node timeouts.
>  
> Thank you,
> Asad