Good to hear Mark. HBase is sensitive to this in a way that Hadoop is not because HBase region servers and clients do not use a static configuration to locate the master. If you check your Hadoop config, I suspect you'll find you configured the namenode location using either a hostname that did not resolve to a localhost address, or a non-localhost IP address. Likewise perhaps the jobtracker location. So datanodes, tasktrackers, clients, etc. would use the static configuration no matter what IP address the namenode, jobtracker, etc. was self-discovering. However the HBase master publishes its location to clients via Zookeeper. This is done to support multimaster operation (failover). So if the HBase master self-discovers its location as a localhost address, then it will publish that. Region servers or clients which go to Zookeeper for the master location will get back an address in that case only useful if they happen to be co-located with the master.
- Andy ________________________________ From: Mark Vigeant <[email protected]> To: "[email protected]" <[email protected]> Sent: Tue, November 24, 2009 1:41:15 PM Subject: RE: Way to Specify HBase master? Ah, thank you so much Andy! You know, I had just gotten hadoop working again (it turns out it too was running with master 127.0.1.1 which is odd because it worked while hbase didn't...) by changing the hostnames on my machines and doing a lot of frustrated restarting but finally everything worked. And I just deleted the 127/8 things in my /etc/hosts and it looks like hbase is FINALLY running! Woo! -Mark -----Original Message----- From: Andrew Purtell [mailto:[email protected]] Sent: Tuesday, November 24, 2009 4:28 PM To: [email protected] Subject: Re: Way to Specify HBase master? If you have an entry in your /etc/hosts file on the master that causes it to resolve its hostname to a localhost address (anything in 127/8) , then this will happen: 1) The master will publish its address into zookeeper as a localhost address. 2) The region servers will attempt to contact the master on the localhost address... which is their localhost of course 3) The connection fails unless the region server and the master happen to be running on the same node. In your case, the solution is to remove all entries in /etc/hosts which cause resolution of the local host name to a localhost address and substitute them with either a functional DNS setup or a /etc/hosts file which resolves your host names to the IP addresses assigned on the shared LAN segment (i.e. eth0 interfaces). - Andy ________________________________ From: Mark Vigeant <[email protected]> To: "[email protected]" <[email protected]> Sent: Tue, November 24, 2009 9:21:01 AM Subject: RE: Way to Specify HBase master? Wait, I think there's been some confusion: the master resolves itself to 127.0.1.1, not 127.0.0.1 I've looked at the HMaster logs (http://pastebin.com/m4ecf071e) and it says that in fact what the region servers are reading is true: the master is running as "hadoop 127.0.1.1." A few interesting things to note are: the ROOT region is not set, and the /hbase/rs node is completely empty. Is this still the matter of a host-name issue? I notice there are dns properties for all aspects of hbase, should I tamper with them? -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Jean-Daniel Cryans Sent: Tuesday, November 24, 2009 11:40 AM To: [email protected] Subject: Re: Way to Specify HBase master? It seems to be the same problems isn't it? The master resolves itself to 127.0.0.1 and all region servers, when communicating with Zookeeper, read the master's znode with the local address but since there isn't a master there, it can't connect. The reason why you can't specify a master is because HBase is dynamically multi-master e.g. you can start 100 masters (one is elected), kill some backups, get new ones, etc. So each node, master and region server, in HBase has to be able to publish its own address. Like Stack said, playing with your hosts file should be enough, just make sure that the hadoop config is still sane (and try to see why it fails if so). If you have a DNS server, you can also set the following values: <property> <name>hbase.zookeeper.dns.interface</name> <value>default</value> <description>The name of the Network Interface from which a ZooKeeper server should report its IP address. </description> </property> <property> <name>hbase.zookeeper.dns.nameserver</name> <value>default</value> <description>The host name or IP address of the name server (DNS) which a ZooKeeper server should use to determine the host name used by the master for communication and display purposes. </description> </property> J-D On Tue, Nov 24, 2009 at 8:28 AM, Mark Vigeant <[email protected]> wrote: > So I tried to change the hostname and it messed up my hadoop cluster... it > appears that everything else runs fine on the host hadoop 127.0.1.1 > > I'll change strategies here: my regionservers see the host and try to connect > to it, but the connection is refused. Does anyone have any ideas on how I can > change that? > > Here's a snippet of the log from my regionserver: > http://pastebin.com/m6bbcb432 > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of stack > Sent: Monday, November 23, 2009 5:28 PM > To: [email protected] > Subject: Re: Way to Specify HBase master? > > This sounds like basic networking issue where host lookup is finding > 127.0.0.1. Can you get someone to help you setup your networking on your > machine. Else play w/ your /etc/hosts. Seems like your host name resolves > to 127.0.0.1. > St.Ack > > On Mon, Nov 23, 2009 at 12:31 PM, Mark Vigeant <[email protected] >> wrote: > >> I'm trying to play around with HBase but it hasn't been working for me. >> From the look of the logs, it appears that the master is running on >> 127.0.1.1 instead of my machine (fry) and so all of the regionservers start, >> but the logs say they cannot connect to the master. From the master logs, it >> thinks there are 0 regionservers and thus won't do anything for me. I'm >> using HBase 20.2. How can I set the IP address of the master machine? >> >> Mark Vigeant >> RiskMetrics Group, Inc. This email message and any attachments are for the sole use of the intended recipients and may contain proprietary and/or confidential information which may be privileged or otherwise protected from disclosure. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy the original message and any copies of the message as well as any attachments to the original message.
