Re: insertion rate change after stopping node
Thanks, i'll check it out. Regards, Osman On Wed, 5 Aug 2020 at 21:15, Jeff Jirsa wrote: > > Potentially slow machine / bad cpu / bad disk / bad dimms. > > > On Wed, Aug 5, 2020 at 10:50 AM Osman Yozgatlıoğlu > wrote: >> >> Hello, >> >> I have 2 DC 8+7 nodes Cassandra cluster with RF:2:2. >> Insertion rate dropped recently. >> If I stop an exact node, insertion rate goes normal. >> What should be the problem? >> >> Regards, >> Osman >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
insertion rate change after stopping node
Hello, I have 2 DC 8+7 nodes Cassandra cluster with RF:2:2. Insertion rate dropped recently. If I stop an exact node, insertion rate goes normal. What should be the problem? Regards, Osman - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: hints files at joining node
Thank you all, I assume, my new node is used as coordinator, since all other nodes are under heavy write load. I need to spare some space for hints folder. Regards, Osman On Thu, 7 May 2020 at 08:29, Jeff Jirsa wrote: > > Incremental bootstrap patch changed the logic here. A node can act as a > coordinator before it's fully joined. It's ... decidedly non-ideal and > probably needs to be changed. > https://issues.apache.org/jira/browse/CASSANDRA-8942 > > > > > > On Wed, May 6, 2020 at 9:57 PM Erick Ramirez > wrote: >> >> The fact that a new node is acting as a coordinator suggests that (1) you >> are adding a node to a DC that is taking traffic from the app, and (2) you >> are likely adding the node using nodetool rebuild instead of the standard >> bootstrap method. >> >> If you are adding a node using the rebuild option (with auto_bootstrap set >> to false in cassandra.yaml), the node joins the cluster as a normal node >> except it doesn't have any data to serve read requests but it will accept >> writes. This isn't a recommended way of adding nodes to a DC that is >> actively serving requests from the app. >> >>> As I understand only coordinator nodes generates hints files. Is >>> cluster uses this node as coordinator before complete join? >>> Or this process is normal for joining and seen as repair? >> >> >> The fact that the new node is storing hints is a concern because it >> indicates that other nodes in your cluster are unresponsive or down. You >> need to investigate why that is the case. >> >>> >>> By the way, files not deleted after 3 hour period. >> >> >> The node will collect hints for other nodes for 3 hours (default). If the >> replica has not come back online after 3 hours, hints will no longer be >> stored but it doesn't delete the hints but instead hand it off to the >> respective replica when it comes back online. Cheers! - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
hints files at joining node
Hello, I have 2 dc cluster and trying to add a new node to one dc. It will take time since total data is huge at every node. I see waiting hints files at that joining node. What is that means? As I understand only coordinator nodes generates hints files. Is cluster uses this node as coordinator before complete join? Or this process is normal for joining and seen as repair? By the way, files not deleted after 3 hour period. Regards, Osman - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: one node down and cluster works better
Thanks Mehmet and Erick, I don't have any monitoring other than nodetool but I manage to see some disk errors cause exceptions. I changed faulty disk and performance ok now. Regards, Osman On Sun, 5 Apr 2020 at 03:17, Erick Ramirez wrote: > > With only 2 replicas per DC, it means you're likely writing with a > consistency level of either ONE or LOCAL_ONE. Everytime you hit the > problematic node, the write performance drops. All other configurations being > equal, this indicates an issue with the commitlog disk on the node. > > Get your sysadmin to check for any issues with the disk, perhaps there's a > problem with the hardware and failure is impending. As a side note, it's > probably a good time to ensure that cluster repairs are in order in case the > node fails. Cheers! > > GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have > answers! Share your expertise on https://community.datastax.com/. > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
one node down and cluster works better
Hello, I manage one cluster with 2 dc, 7 nodes each and replication factor is 2:2 My insertion performance dropped somehow. I restarted nodes one by one and found one node degrades performance. Verified this node after problem occurs a couple of times. How can I continue to investigate? Regards, Osman - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: inter dc bandwidth calculation
Thank you. I have an insight now. Regards, Osman On Wed, 15 Jan 2020 at 19:18, Reid Pinchback wrote: > > Oh, duh. Revise that. I was forgetting that multi-dc writes are sent to a > single node in the other dc and tagged to be forwarded to other nodes within > the dc. > > So your quick-and-dirty estimate would be more like (write volume) x 2 to > leave headroom for random other mechanics. > > R > > > On 1/15/20, 11:07 AM, "Reid Pinchback" wrote: > > Message from External Sender > > I would think that it would be largely driven by the replication factor. > It isn't that the sstables are forklifted from one dc to another, it's just > that the writes being made to the memtables are also shipped around by the > coordinator nodes as the writes happen. Operations at the sstable level, > like compactions, are local to the node. > > One potential wrinkle that I'm unclear on, is related to repairs. I > don't know if merkle trees are biased to mostly bounce around only intra-dc, > versus how often they are communicated inter-dc. Note that even queries can > trigger some degree of repair traffic if you have a usage pattern of trying > to read data recently written, because at the bleeding edge of the recent > changes you'll have more cases of rows not having had time to settle to a > consistent state. > > If you want a quick-and-dirty heuristic, I'd probably take (write volume) > x (replication factor) x 2 as a guestimate so you have some headroom for C* > and TCP mechanics, but then monitor to see what your real use is. > > R > > > On 1/15/20, 4:14 AM, "Osman Yozgatlıoğlu" > wrote: > > Message from External Sender > > Hello, > > Is there any way to calculate inter dc bandwidth requirements for > proper operation? > I can't find any info about this subject. > Can we say, how much sstable collected at one dc has to be > transferred to other? > I can calculate bandwidth with generated sstable then. > I have twcs with one hour window. > > Regards, > Osman > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > > > > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
inter dc bandwidth calculation
Hello, Is there any way to calculate inter dc bandwidth requirements for proper operation? I can't find any info about this subject. Can we say, how much sstable collected at one dc has to be transferred to other? I can calculate bandwidth with generated sstable then. I have twcs with one hour window. Regards, Osman - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Facing issues while starting Cassandra
I faced similar issues with different locale settings. Could you try following command before running? export LC_ALL=C; Regards, Osman On Tue, 24 Dec 2019 at 09:01, Inquistive allen wrote: > > Hello Jeff, > > Thanks for responding. > I have validated the cassandra.yaml file with other hosts in the cluster. > There is no difference. I copied a yaml file from other node to this node and > changed the required configs. Still facing the same issue. > The server went down for patching and after coming back up, Cassandra dosent > seem to start. > Having looked for solutions on google, I found that it might be a problem > with the /tmp directory where the classes are stored. > Each time I try starting Cassandra, in the /tmp directory a new directory is > created, but nothing is inside the directory. After some time, the node goes > down. > > I believe there is something to do with the /tmp directory. > Request you to comment on the same. > > Thanks > > On Tue, 24 Dec, 2019, 3:42 AM Jeff Jirsa, wrote: >> >> Are you able to share the yaml? Almost certainly something in it that’s >> invalid. >> >> On Dec 23, 2019, at 12:51 PM, Inquistive allen wrote: >> >> >> Hello Team, >> >> I am facing issues while starting Cassandra. >> >> Caused by: org.apache.cassandra.exceptions.ConfigurationException : Invalid >> yaml: file: /path/to/yaml >> Error: null ; can't construct a java object for tag: >> yaml.org,2002:org.apache.cassandra.config.Config; exception= >> java.lang.reflect.InvocationTargetException >> >> Request to comment on how to resolve the issue. >> >> Thanks & Regards >> Allen - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
