Re: hints files at joining node
Thank you all, I assume, my new node is used as coordinator, since all other nodes are under heavy write load. I need to spare some space for hints folder. Regards, Osman On Thu, 7 May 2020 at 08:29, Jeff Jirsa wrote: > > Incremental bootstrap patch changed the logic here. A node can act as a > coordinator before it's fully joined. It's ... decidedly non-ideal and > probably needs to be changed. > https://issues.apache.org/jira/browse/CASSANDRA-8942 > > > > > > On Wed, May 6, 2020 at 9:57 PM Erick Ramirez > wrote: >> >> The fact that a new node is acting as a coordinator suggests that (1) you >> are adding a node to a DC that is taking traffic from the app, and (2) you >> are likely adding the node using nodetool rebuild instead of the standard >> bootstrap method. >> >> If you are adding a node using the rebuild option (with auto_bootstrap set >> to false in cassandra.yaml), the node joins the cluster as a normal node >> except it doesn't have any data to serve read requests but it will accept >> writes. This isn't a recommended way of adding nodes to a DC that is >> actively serving requests from the app. >> >>> As I understand only coordinator nodes generates hints files. Is >>> cluster uses this node as coordinator before complete join? >>> Or this process is normal for joining and seen as repair? >> >> >> The fact that the new node is storing hints is a concern because it >> indicates that other nodes in your cluster are unresponsive or down. You >> need to investigate why that is the case. >> >>> >>> By the way, files not deleted after 3 hour period. >> >> >> The node will collect hints for other nodes for 3 hours (default). If the >> replica has not come back online after 3 hours, hints will no longer be >> stored but it doesn't delete the hints but instead hand it off to the >> respective replica when it comes back online. Cheers! - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: hints files at joining node
Incremental bootstrap patch changed the logic here. A node can act as a coordinator before it's fully joined. It's ... decidedly non-ideal and probably needs to be changed. https://issues.apache.org/jira/browse/CASSANDRA-8942 On Wed, May 6, 2020 at 9:57 PM Erick Ramirez wrote: > The fact that a new node is acting as a coordinator suggests that (1) you > are adding a node to a DC that is taking traffic from the app, and (2) you > are likely adding the node using nodetool rebuild instead of the standard > bootstrap method. > > If you are adding a node using the rebuild option (with auto_bootstrap > set to false in cassandra.yaml), the node joins the cluster as a normal > node except it doesn't have any data to serve read requests but it will > accept writes. This isn't a recommended way of adding nodes to a DC that is > actively serving requests from the app. > > As I understand only coordinator nodes generates hints files. Is >> cluster uses this node as coordinator before complete join? >> Or this process is normal for joining and seen as repair? >> > > The fact that the new node is storing hints is a concern because it > indicates that other nodes in your cluster are unresponsive or down. You > need to investigate why that is the case. > > >> By the way, files not deleted after 3 hour period. > > > The node will collect hints for other nodes for 3 hours (default). If the > replica has not come back online after 3 hours, hints will no longer be > stored but it doesn't delete the hints but instead hand it off to the > respective replica when it comes back online. Cheers! >
Re: hints files at joining node
The fact that a new node is acting as a coordinator suggests that (1) you are adding a node to a DC that is taking traffic from the app, and (2) you are likely adding the node using nodetool rebuild instead of the standard bootstrap method. If you are adding a node using the rebuild option (with auto_bootstrap set to false in cassandra.yaml), the node joins the cluster as a normal node except it doesn't have any data to serve read requests but it will accept writes. This isn't a recommended way of adding nodes to a DC that is actively serving requests from the app. As I understand only coordinator nodes generates hints files. Is > cluster uses this node as coordinator before complete join? > Or this process is normal for joining and seen as repair? > The fact that the new node is storing hints is a concern because it indicates that other nodes in your cluster are unresponsive or down. You need to investigate why that is the case. > By the way, files not deleted after 3 hour period. The node will collect hints for other nodes for 3 hours (default). If the replica has not come back online after 3 hours, hints will no longer be stored but it doesn't delete the hints but instead hand it off to the respective replica when it comes back online. Cheers!
hints files at joining node
Hello, I have 2 dc cluster and trying to add a new node to one dc. It will take time since total data is huge at every node. I see waiting hints files at that joining node. What is that means? As I understand only coordinator nodes generates hints files. Is cluster uses this node as coordinator before complete join? Or this process is normal for joining and seen as repair? By the way, files not deleted after 3 hour period. Regards, Osman - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
