Mutations get dropped because a node can't keep up with writes. If you understand the Cassandra write path, writes are ACKed when the mutation is appended to the commitlog which is why it's very fast.
Knowing that, dropped mutations mean that the disk is not able to keep up with the IO. Another word for it is "overloaded". So what do you do? Use fast SSDs (HDDs and EBS won't do) and size your cluster correctly to deal with the peak load, i.e. add more nodes. Cheers! On Fri, Jul 21, 2017 at 1:27 AM, ZAIDI, ASAD A <az1...@att.com> wrote: > Hello Folks – > > > > I’m using apache-cassandra 2.2.8. > > > > I see many messages like below in my system.log file. In Cassandra.yaml > file [ cross_node_timeout: true] is set and NTP server is also running > correcting clock drift on 16node cluster. I do not see pending or blocked > HintedHandoff in tpstats output though there are bunch of MUTATIONS > dropped observed. > > > > <start timeout message > > > INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 > - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout > and 2152 for cross node timeout > > <end timeout message > > > > > I’m seeking help here if you please let me know what I need to check in > order to address these cross node timeouts. > > > > Thank you, > > Asad > > >
