Re: live nodes list in ring
Benjamin, Yes i have seen this when adding a new node into the cluster. the new node doesnt see the complete ring through nodetool, but the strange part is that looking at the ring through jconsole shows the complete ring. it as if there is a big in nodetool publishing the actual ring. has anyone seen that scenario. while in this situation is does appear that the cluster is functioning correctly with replicating data, just cant trust nodetools ring information. Artie 2010/6/30 Benjamin Black > Does this happen after you have changed the ring topology, especially > adding nodes? > > 2010/6/30 Stephen Hamer : > > When this happens to me I have to do a full cluster restart. Even doing a > > rolling restart across the cluster doesn't seem to fix them, all of the > > nodes need to be stopped at the same time. After bringing everything back > up > > the ring is correct. > > > > > > > > Does anyone know how a cluster gets into this state? > > > > > > > > Stephen > > > > > > > > From: aaron morton [mailto:aa...@thelastpickle.com] > > Sent: Wednesday, June 30, 2010 1:42 PM > > To: user@cassandra.apache.org > > Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' > > Subject: Re: live nodes list in ring > > > > > > > > At start up do you see log lines like this > > > > > > > > Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster > > > > > > > > Are all the nodes listed? > > > > > > > > aaron > > > > On 30 Jun 2010, at 22:50, 王一锋 wrote: > > > > Hi, > > > > > > > > In a cassandra cluster, when issueing ring command on every nodes, some > can > > show all nodes in the cluster but some can only show some other nodes. > > > > All nodes share the same seed list. > > > > And even some of the nodes in the seed list have this problem. > > > > Restarting the problematic nodes won't solve it. > > > > Try closing firewalls with following commands > > > > > > > > service iptables stop > > > > > > > > Still won't work. > > > > > > > > Anyone got a clue? > > > > > > > > Thanks very much. > > > > > > > > Yifeng > > > > >
Re: RE: live nodes list in ring
problem solved with a full cluster stop and then start~ Thanks Stephen. But still wondering the reason 2010-07-01 发件人： Stephen Hamer 发送时间： 2010-07-01 08:39:13 收件人： user@cassandra.apache.org 抄送： 'huzhonghua'; 'GongJianTao（宫建涛）' 主题： RE: live nodes list in ring When this happens to me I have to do a full cluster restart. Even doing a rolling restart across the cluster doesn’t seem to fix them, all of the nodes need to be stopped at the same time. After bringing everything back up the ring is correct. Does anyone know how a cluster gets into this state? Stephen From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Wednesday, June 30, 2010 1:42 PM To: user@cassandra.apache.org Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' Subject: Re: live nodes list in ring At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
Re: Re: live nodes list in ring
To me, this happens all in a sudden. No nodes removed nor new nodes joined 2010-07-01 发件人： Benjamin Black 发送时间： 2010-07-01 09:55:59 收件人： user 抄送： 主题： Re: live nodes list in ring Does this happen after you have changed the ring topology, especially adding nodes? 2010/6/30 Stephen Hamer : > When this happens to me I have to do a full cluster restart. Even doing a > rolling restart across the cluster doesn't seem to fix them, all of the > nodes need to be stopped at the same time. After bringing everything back up > the ring is correct. > > > > Does anyone know how a cluster gets into this state? > > > > Stephen > > > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Wednesday, June 30, 2010 1:42 PM > To: user@cassandra.apache.org > Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' > Subject: Re: live nodes list in ring > > > > At start up do you see log lines like this > > > > Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster > > > > Are all the nodes listed? > > > > aaron > > On 30 Jun 2010, at 22:50, 王一锋 wrote: > > Hi, > > > > In a cassandra cluster, when issueing ring command on every nodes, some can > show all nodes in the cluster but some can only show some other nodes. > > All nodes share the same seed list. > > And even some of the nodes in the seed list have this problem. > > Restarting the problematic nodes won't solve it. > > Try closing firewalls with following commands > > > > service iptables stop > > > > Still won't work. > > > > Anyone got a clue? > > > > Thanks very much. > > > > Yifeng > >
Re: Re: live nodes list in ring
Yes, but not all nodes. Just the part that the restarting node can show on its ring. 2010-07-01 发件人： aaron morton 发送时间： 2010-07-01 04:42:56 收件人： user 抄送： 'huzhonghua'; 'GongJianTao（宫建涛 ）' 主题： Re: live nodes list in ring At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
Re: live nodes list in ring
Does this happen after you have changed the ring topology, especially adding nodes? 2010/6/30 Stephen Hamer : > When this happens to me I have to do a full cluster restart. Even doing a > rolling restart across the cluster doesn't seem to fix them, all of the > nodes need to be stopped at the same time. After bringing everything back up > the ring is correct. > > > > Does anyone know how a cluster gets into this state? > > > > Stephen > > > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Wednesday, June 30, 2010 1:42 PM > To: user@cassandra.apache.org > Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' > Subject: Re: live nodes list in ring > > > > At start up do you see log lines like this > > > > Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster > > > > Are all the nodes listed? > > > > aaron > > On 30 Jun 2010, at 22:50, 王一锋 wrote: > > Hi, > > > > In a cassandra cluster, when issueing ring command on every nodes, some can > show all nodes in the cluster but some can only show some other nodes. > > All nodes share the same seed list. > > And even some of the nodes in the seed list have this problem. > > Restarting the problematic nodes won't solve it. > > Try closing firewalls with following commands > > > > service iptables stop > > > > Still won't work. > > > > Anyone got a clue? > > > > Thanks very much. > > > > Yifeng > >
RE: live nodes list in ring
When this happens to me I have to do a full cluster restart. Even doing a rolling restart across the cluster doesn’t seem to fix them, all of the nodes need to be stopped at the same time. After bringing everything back up the ring is correct. Does anyone know how a cluster gets into this state? Stephen From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Wednesday, June 30, 2010 1:42 PM To: user@cassandra.apache.org Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' Subject: Re: live nodes list in ring At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
Re: live nodes list in ring
At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: > Hi, > > In a cassandra cluster, when issueing ring command on every nodes, some can > show all nodes in the cluster but some can only show some other nodes. > All nodes share the same seed list. > And even some of the nodes in the seed list have this problem. > Restarting the problematic nodes won't solve it. > Try closing firewalls with following commands > > service iptables stop > > Still won't work. > > Anyone got a clue? > > Thanks very much. > > Yifeng
live nodes list in ring
Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
