node tool passes the host name un modified to the JMX library to connect to the host.
The JMX server will, by default, bind to the ip address of the machine. If the host name was wrong, I would guess the JMX service failed to bind. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 30/05/2012, at 6:39 AM, Douglas Muth wrote: > 8 hours, 1 cup of coffee, and 4 Advil later, and I think I got the > bottom of this. Not having much of a Java or JMX background, I'll try > to explain it the best that I can. > > To recap, my machine originally had the IP address of 10.244.207.16. > Then I shutdown/restarted that EC2 instance, and it had the IP > 10.84.117.110. During this, Cassandra was fine -- I could still > connect to 127.0.0.1 with cqlsh and the Helenus node.js module. > > Things got weird only when I tried to use nodetool to manage the > instance. As best I can tell, here's the algorithm that nodetool uses > when connecting to a Cassandra instance: > > Step 1) Connect to the hostname and port specified on the command > line. "localhost" and "7199" are the defaults. > > Step 2) Cassandra, at boot time, notes the hostname of the machine, > and tells nodetool "go connect to this hostname instead!" > > After further investigation, it seems that after my instance was > rebooted, the file /etc/hostname was not updated. It still had the > value "ip-10-244-207-16.ec2.internal" in it. This means that any > attempt to connect to Cassandra involved Cassandra telling nodetool, > "Hey, go talk to 10.244.207.16 instead". And that's where things went > wrong. > > The permanent fix for this was to change the hostname to "localhost" > and to restart Cassandra. The fact that Cassandra notes the hostname > at startup was one thing that made this so difficult to track down. I > did not see the old IP anywhere in Cassandra configuration (or in > logfile output), so I did not think there was anything abnormal > happening in the instance. > > While I'm sure there's a good reason for this sort of behavior, it is > very confusing to a Cassandra newbie such as myself, and I'll bet > others have been affected by this as well. In the future, I think > some sort of logging of this sort of of logic, or perhaps a --verbose > mode for nodetool would be a really good idea. What do other folks > think? > > -- Doug > http://twitter.com/dmuth > > > On Tue, May 29, 2012 at 12:08 PM, Douglas Muth <doug.m...@gmail.com> wrote: >> I'm afraid that did not work. I'm running JMX on port 7199 (the >> default) and I verified that the port is open and accepting >> connections. > [snip]
