Re: Nodetool talking to an old IP address (and timing out)
node tool passes the host name un modified to the JMX library to connect to the host. The JMX server will, by default, bind to the ip address of the machine. If the host name was wrong, I would guess the JMX service failed to bind. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 30/05/2012, at 6:39 AM, Douglas Muth wrote: > 8 hours, 1 cup of coffee, and 4 Advil later, and I think I got the > bottom of this. Not having much of a Java or JMX background, I'll try > to explain it the best that I can. > > To recap, my machine originally had the IP address of 10.244.207.16. > Then I shutdown/restarted that EC2 instance, and it had the IP > 10.84.117.110. During this, Cassandra was fine -- I could still > connect to 127.0.0.1 with cqlsh and the Helenus node.js module. > > Things got weird only when I tried to use nodetool to manage the > instance. As best I can tell, here's the algorithm that nodetool uses > when connecting to a Cassandra instance: > > Step 1) Connect to the hostname and port specified on the command > line. "localhost" and "7199" are the defaults. > > Step 2) Cassandra, at boot time, notes the hostname of the machine, > and tells nodetool "go connect to this hostname instead!" > > After further investigation, it seems that after my instance was > rebooted, the file /etc/hostname was not updated. It still had the > value "ip-10-244-207-16.ec2.internal" in it. This means that any > attempt to connect to Cassandra involved Cassandra telling nodetool, > "Hey, go talk to 10.244.207.16 instead". And that's where things went > wrong. > > The permanent fix for this was to change the hostname to "localhost" > and to restart Cassandra. The fact that Cassandra notes the hostname > at startup was one thing that made this so difficult to track down. I > did not see the old IP anywhere in Cassandra configuration (or in > logfile output), so I did not think there was anything abnormal > happening in the instance. > > While I'm sure there's a good reason for this sort of behavior, it is > very confusing to a Cassandra newbie such as myself, and I'll bet > others have been affected by this as well. In the future, I think > some sort of logging of this sort of of logic, or perhaps a --verbose > mode for nodetool would be a really good idea. What do other folks > think? > > -- Doug > http://twitter.com/dmuth > > > On Tue, May 29, 2012 at 12:08 PM, Douglas Muth wrote: >> I'm afraid that did not work. I'm running JMX on port 7199 (the >> default) and I verified that the port is open and accepting >> connections. > [snip]
Re: Nodetool talking to an old IP address (and timing out)
Did you open the inbound port 1024 ~ 65535 at Security Group? JMX uses two connection channels, one is 7199 by default for accepting connection request, another one is a random port between 1024 ~65535 decided during run time. Nodetool runs over JMX. Patrick. -Original Message- From: Douglas Muth Sent: Tuesday, May 29, 2012 11:39 AM To: user@cassandra.apache.org Subject: Re: Nodetool talking to an old IP address (and timing out) 8 hours, 1 cup of coffee, and 4 Advil later, and I think I got the bottom of this. Not having much of a Java or JMX background, I'll try to explain it the best that I can. To recap, my machine originally had the IP address of 10.244.207.16. Then I shutdown/restarted that EC2 instance, and it had the IP 10.84.117.110. During this, Cassandra was fine -- I could still connect to 127.0.0.1 with cqlsh and the Helenus node.js module. Things got weird only when I tried to use nodetool to manage the instance. As best I can tell, here's the algorithm that nodetool uses when connecting to a Cassandra instance: Step 1) Connect to the hostname and port specified on the command line. "localhost" and "7199" are the defaults. Step 2) Cassandra, at boot time, notes the hostname of the machine, and tells nodetool "go connect to this hostname instead!" After further investigation, it seems that after my instance was rebooted, the file /etc/hostname was not updated. It still had the value "ip-10-244-207-16.ec2.internal" in it. This means that any attempt to connect to Cassandra involved Cassandra telling nodetool, "Hey, go talk to 10.244.207.16 instead". And that's where things went wrong. The permanent fix for this was to change the hostname to "localhost" and to restart Cassandra. The fact that Cassandra notes the hostname at startup was one thing that made this so difficult to track down. I did not see the old IP anywhere in Cassandra configuration (or in logfile output), so I did not think there was anything abnormal happening in the instance. While I'm sure there's a good reason for this sort of behavior, it is very confusing to a Cassandra newbie such as myself, and I'll bet others have been affected by this as well. In the future, I think some sort of logging of this sort of of logic, or perhaps a --verbose mode for nodetool would be a really good idea. What do other folks think? -- Doug http://twitter.com/dmuth On Tue, May 29, 2012 at 12:08 PM, Douglas Muth wrote: I'm afraid that did not work. I'm running JMX on port 7199 (the default) and I verified that the port is open and accepting connections. [snip]
Re: Nodetool talking to an old IP address (and timing out)
8 hours, 1 cup of coffee, and 4 Advil later, and I think I got the bottom of this. Not having much of a Java or JMX background, I'll try to explain it the best that I can. To recap, my machine originally had the IP address of 10.244.207.16. Then I shutdown/restarted that EC2 instance, and it had the IP 10.84.117.110. During this, Cassandra was fine -- I could still connect to 127.0.0.1 with cqlsh and the Helenus node.js module. Things got weird only when I tried to use nodetool to manage the instance. As best I can tell, here's the algorithm that nodetool uses when connecting to a Cassandra instance: Step 1) Connect to the hostname and port specified on the command line. "localhost" and "7199" are the defaults. Step 2) Cassandra, at boot time, notes the hostname of the machine, and tells nodetool "go connect to this hostname instead!" After further investigation, it seems that after my instance was rebooted, the file /etc/hostname was not updated. It still had the value "ip-10-244-207-16.ec2.internal" in it. This means that any attempt to connect to Cassandra involved Cassandra telling nodetool, "Hey, go talk to 10.244.207.16 instead". And that's where things went wrong. The permanent fix for this was to change the hostname to "localhost" and to restart Cassandra. The fact that Cassandra notes the hostname at startup was one thing that made this so difficult to track down. I did not see the old IP anywhere in Cassandra configuration (or in logfile output), so I did not think there was anything abnormal happening in the instance. While I'm sure there's a good reason for this sort of behavior, it is very confusing to a Cassandra newbie such as myself, and I'll bet others have been affected by this as well. In the future, I think some sort of logging of this sort of of logic, or perhaps a --verbose mode for nodetool would be a really good idea. What do other folks think? -- Doug http://twitter.com/dmuth On Tue, May 29, 2012 at 12:08 PM, Douglas Muth wrote: > I'm afraid that did not work. I'm running JMX on port 7199 (the > default) and I verified that the port is open and accepting > connections. [snip]
Re: Nodetool talking to an old IP address (and timing out)
I'm afraid that did not work. I'm running JMX on port 7199 (the default) and I verified that the port is open and accepting connections. Here's what I'm seeing: dmuth@devteam:~/cliq (production) $ nodetool --host localhost --port 7199 ring Error connection to remote JMX agent! java.rmi.ConnectException: Connection refused to host: 10.244.207.16; nested exception is: java.net.ConnectException: Connection timed out at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) [snip] I'm guessing that the old IP address of our machine is cached somewhere inside of Cassandra or something related. Anyone have suggestions on where else I can check or debugging ideas? Thanks, -- Doug On Sun, May 27, 2012 at 1:45 PM, Cyril Auburtin wrote: > specify the jmx port to nodetool, hard coded in conf/cassandra-env.sh > > nodetool -h localhost -p [jmx port] ring > > 2012/5/27 Douglas Muth >> >> Hi folks, >> >> I'm a relative newbie to Cassandra, and have been trying to get up to >> speed on it so that I can start using it at $WORK. >> >> I ran into an interesting issue the other day with nodetool. I >> currently have Cassandra running on an Amazon EC2 instance running >> Ubuntu 10.10. At one point, I rebooted the system, and it looks like >> any attempt to use nodetool to talk to the localhost instead tries to >> connect to the old IP address of the machine! (EC2 instances get a new >> IP after shutdown/startup) >> >> When I try to run nodetool now, it times out after about 10 seconds >> with an error like this: >> >> dmuth@devteam:~ $ nodetool --host localhost ring >> Error connection to remote JMX agent! >> java.rmi.ConnectException: Connection refused to host: 10.244.207.16; >> nested exception is: >> java.net.ConnectException: Connection timed out >> >> >> And I've verified that the IP of the machine does NOT in fact end in .16: >> >> dmuth@devteam:~ $ ifconfig eth0 >> eth0 Link encap:Ethernet HWaddr 12:31:3d:14:6a:84 >> inet addr:10.84.117.110 Bcast:10.84.117.255 Mask:255.255.255.0 >> >> >> I checked configuration file for Cassandra and verified that I do in >> fact have the new IP address in there. I also made sure that there >> was nothing weird in /etc/hosts. >> >> Also, cqlsh works just fine, as does the Helenus client for node.js. >> I can talk to our cassandra instance just fine through either of those >> two. >> >> I'm out of ideas at this point. Does anyone have any other >> suggestions for what I investigate on my system? >> >> Thanks, >> >> -- Doug >> http://twitter.com/dmuth > >
Re: Nodetool talking to an old IP address (and timing out)
specify the jmx port to nodetool, hard coded in conf/cassandra-env.sh nodetool -h localhost -p [jmx port] ring 2012/5/27 Douglas Muth > Hi folks, > > I'm a relative newbie to Cassandra, and have been trying to get up to > speed on it so that I can start using it at $WORK. > > I ran into an interesting issue the other day with nodetool. I > currently have Cassandra running on an Amazon EC2 instance running > Ubuntu 10.10. At one point, I rebooted the system, and it looks like > any attempt to use nodetool to talk to the localhost instead tries to > connect to the old IP address of the machine! (EC2 instances get a new > IP after shutdown/startup) > > When I try to run nodetool now, it times out after about 10 seconds > with an error like this: > > dmuth@devteam:~ $ nodetool --host localhost ring > Error connection to remote JMX agent! > java.rmi.ConnectException: Connection refused to host: 10.244.207.16; > nested exception is: > java.net.ConnectException: Connection timed out > > > And I've verified that the IP of the machine does NOT in fact end in .16: > > dmuth@devteam:~ $ ifconfig eth0 > eth0 Link encap:Ethernet HWaddr 12:31:3d:14:6a:84 > inet addr:10.84.117.110 Bcast:10.84.117.255 Mask:255.255.255.0 > > > I checked configuration file for Cassandra and verified that I do in > fact have the new IP address in there. I also made sure that there > was nothing weird in /etc/hosts. > > Also, cqlsh works just fine, as does the Helenus client for node.js. > I can talk to our cassandra instance just fine through either of those > two. > > I'm out of ideas at this point. Does anyone have any other > suggestions for what I investigate on my system? > > Thanks, > > -- Doug > http://twitter.com/dmuth >
