Hi, We have a small 5 node cluster of m4.xlarge clients that receives writes from ~20 clients. The clients will write as fast as they can, and the whole process is limited by the write performance of the cassandra cluster. After we have tweaked our schema to avoid large partitions, the load is going ok and we don't see any warnings or errors in the cassandra logs. But we do see quite a lot of hint handoff activity. During the load, the cassandra nodes are quite loaded, with linux reporting a load as high as 20.
I have read the available documentation on how hints works, and to my understanding hints should only be written if a node is down. But as far as I can see, none of the nodes are marked as down during the load. So I suspect I am missing something :) We have configured the servers with write_request_timeout_in_ms: 120000 and the clients with a timeout of 130000, but still get hints stored. In our case, I would like for the cluster to wait for the write to be persisted on the relevant nodes before returning an ok to the client. But I don't know which knobs to turn to accomplish this? or if it is even possible :) We are running cassandra 3.0.3, with 8Gb heap and a replication factor of 3. Thank you in advance! Yours sincerely, Bo Madsen
