Hi again Bo, Yeah, definitely wounds weird you are seeing hints.
It's definitely interesting that you aren't seeing hints for 2.1.13. Are you seeing hints when not using materialized views in 3.0.3? Cheers, Jens On Mon, Apr 25, 2016 at 10:05 AM Bo Finnerup Madsen <bo.gunder...@gmail.com> wrote: > Hi Jan, > > I don't think I expressed myself clearly :) > The load I mention is as reported by linux. It is my understanding that a > load of 20 means that the computer have enough work to saturate 20 CPU's. > Seeing as we only have 2 vCPUs in our AWS machines, it is quite a bit more > than 20% :) > > Regarding that blog, I have read it...several times by now :) > Unfortunately it does not answer the general question of exactly when a > hint is written. > > fre. 22. apr. 2016 kl. 04.33 skrev Jan <cne...@yahoo.com>: > >> HI Bo; >> >> you raised 2 questions: >> 20% system utilization >> Hints >> >> 20% system utilization: For a node or a cluster to have 20% utilization >> is Normal during peak write operation. >> Hints: hints are written when a node is unreachable; C* 3.0 has >> a complete over haul in the way hints have been implemented. >> >> Recommend reading up this blog article: >> >> http://www.datastax.com/dev/blog/whats-coming-to-cassandra-in-3-0-improved-hint-storage-and-delivery >> >> hope this helps >> Jan/ >> >> >> -------------------------------------------- >> On Thu, 4/21/16, Jens Rantil <jens.ran...@tink.se> wrote: >> >> Subject: Re: When are hints written? >> To: user@cassandra.apache.org >> Date: Thursday, April 21, 2016, 8:57 AM >> >> Hi again >> Bo, >> I assume this is the piece of >> documentation you are referring to? >> http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_about_hh_c.html?scroll=concept_ds_ifg_jqx_zj__performance >> >> > If a >> replica node is overloaded or unavailable, and the failure >> detector has not yet marked it down, then expect most or all >> writes to that node to fail after the timeout triggered by >> write_request_timeout_in_ms, >> which defaults to 10 seconds. During that time, Cassandra >> writes the hint when the timeout is reached. >> I'm not an expert on this, but >> the way I've seen is that hints are written stored as >> soon as there is _any_ issues writing a mutation >> (insert/update/delete) to a node. By "issue", that >> essentially means that a node hasn't acknowledged back >> to the coordinator that the write succeeded within >> write_request_timeout_in_ms. This includes TCP/socket >> timeouts, connection issues or that the node is down. The >> hints are stored for a maximum timespan defaulting to 3 >> hours. >> >> Cheers, >> Jens >> On Thu, Apr >> 21, 2016 at 8:06 AM Bo Finnerup Madsen <bo.gunder...@gmail.com> >> wrote: >> Hi Jens, >> Thank you >> for the tip!ALL would definitely cure our hints >> issue, but as you note, it is not optimal as we are unable >> to take down nodes without clients failing. >> I am most probably overlooking >> something in the documentation, but I cannot see any >> description of when hints are written other than when a node >> is marked as being down. And since none of our nodes have >> been marked as being down (at least according to the logs), >> I suspect that there is some timeout that governs when hints >> are written? >> Regarding >> your other post: Yes, 3.0.3 is pretty new. But we are new to >> this cassandra game, and our schema-fu is not strong enough >> for us to create a schema without using materialized views >> :) >> >> ons. 20. apr. 2016 kl. 17.09 skrev Jens Rantil >> <jens.ran...@tink.se>: >> Hi Bo, >> > In our case, I would like for the >> cluster to wait for the write to be persisted on the >> relevant nodes before returning an ok to the client. But I don't know >> which >> knobs to turn to accomplish this? or if it is even possible >> :) >> This is what write consistency >> option is for. Have a look at >> https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html >> . >> Note, however that if you use ALL, your clients will fail >> (throw exception, depending on language) as soon as a single >> partition can't be written. This means you can't do >> online maintenance of a Cassandra node (such as upgrading it >> etc.) without experiencing write issues. >> Cheers,Jens >> On Wed, Apr 20, 2016 at 3:39 PM Bo Finnerup Madsen >> <bo.gunder...@gmail.com> >> wrote: >> Hi, >> We have a >> small 5 node cluster of m4.xlarge clients that receives >> writes from ~20 clients. The clients will write as fast as >> they can, and the whole process is limited by the write >> performance of the cassandra cluster.After we have tweaked our schema to >> avoid large partitions, the load is going ok and we >> don't see any warnings or errors in the cassandra logs. >> But we do see quite a lot of hint handoff activity. During >> the load, the cassandra nodes are quite loaded, with linux >> reporting a load as high as 20. >> I have read the available >> documentation on how hints works, and to my understanding >> hints should only be written if a node is down. But as far >> as I can see, none of the nodes are marked as down during >> the load. So I suspect I am missing something >> :)We have configured the servers >> with write_request_timeout_in_ms: 120000 and the clients >> with a timeout of 130000, but still get hints >> stored. >> In our case, I >> would like for the cluster to wait for the write to be >> persisted on the relevant nodes before returning an ok to >> the client. But I don't know which knobs to turn to >> accomplish this? or if it is even possible :) >> We are running cassandra 3.0.3, with >> 8Gb heap and a replication factor of 3. >> Thank you in advance! >> Yours sincerely, Bo >> Madsen >> -- >> >> >> >> >> >> >> >> >> Jens Rantil >> Backend Developer @ Tink >> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden >> For urgent matters you can reach me at >> +46-708-84 18 32. >> -- >> >> >> >> >> >> >> >> >> Jens Rantil >> Backend Developer @ Tink >> Tink AB, Wallingatan 5, 111 60 >> Stockholm, Sweden >> For urgent matters you can >> reach me at +46-708-84 18 32. >> > -- Jens Rantil Backend Developer @ Tink Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden For urgent matters you can reach me at +46-708-84 18 32.
