Looks like all of this is happening because we’re using CAS operations and the driver is going to SERIAL consistency level.
SERIAL and LOCAL_SERIAL write failure scenarios¶ > > <http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html?scroll=concept_ds_umf_5xx_zj__failure-scenarios>If > one of three nodes is down, the Paxos commit fails under the following > conditions: > > - CQL query-configured consistency level of ALL > > > - Driver-configured serial consistency level of SERIAL > > > - Replication factor of 3 > > I don’t understand why this would fail.. it seems completely broken in this situation. We were having write timeout at replication factor of 2 .. and a lot of people from the list said of course , because 2 nodes with 1 node down means there’s no quorum and paxos needs a quorum. .. and not sure why I missed that :-P So we went with 3 replicas, and a quorum, but this is new and I didn’t see this documented. We set the driver to QUORUM but then I guess the driver sees that this is a CAS operation and forces it back to SERIAL? Doesn’t this mean that all decommissions result in failures of CAS? This is Cassandra 2.0.9 btw. On Wed, Jul 1, 2015 at 2:22 PM, Kevin Burton <bur...@spinn3r.com> wrote: > We get lots of write timeouts when we decommission a node. About 80% of > them are write timeout and just about 20% of them are read timeout. > > We’ve tried to adjust streamthroughput (and compaction throughput) for > that matter and that doesn’t resolve the issue. > > We’ve increased write_request_timeout_in_ms … and read timeout as well. > > Is there anything else I should be looking at? > > I can’t seem to find the documentation that explains what the heck is > happening. > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts>
