I disabled the firewall in all the machines.Then i started hbase(bin/start-hbase.sh), after 2-3 minutesi stopped hbase on master node (bin/stop-hbase.sh). Even then HRegionserver daemon is running on region server machine. So i checked regionserver machines log, the full log is here
http://pastebin.com/FXsNLMD9 I found that regionserver machine is not able to communicate with master though it is attempting to connect to master (hbase-master:60000)for every one minute or so. so i tried to do the following as you have told. hduser@vamshi-laptop:~$ telnet hbase-master 60000 Trying 10.0.1.54... telnet: Unable to connect to remote host: Connection refused hduser@vamshi-laptop:~$ telnet hbase-master:60000 telnet: could not resolve hbase-master:60000/telnet: Name or service not known i made hbase-master to bind to 10.0.1.54 in /etc/hosts file, so i tried even following also. hduser@vamshi-laptop:~$ telnet 10.0.1.54 60000 Trying 10.0.1.54... telnet: Unable to connect to remote host: Connection refused hduser@vamshi-laptop:~$ So, i thought whether hbase-master is bound to port 60000 on the master node or not, so while hbase master is running, i checked by typing netstat -anp | grep 60000 i could not find 10.0.1.54:60000 any where on the console, rather i found 127.0.1.1:60000 in some places. Is this the correct way of representing /etc/hosts file on master node? 127.0.0.1 localhost 127.0.1.1 vamshikrishna-laptop 10.0.1.54 hbase-master hadoop-namenode vamshikrishna-desktop 10.0.1.55 hbase-regionserver1 hadoop-datanode1 10.0.1.97 hbase-regionserver2 hadoop-datanode2 arun-desktop # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts Even i checked by typing the command on both master node and region server machine $ nmap -p 60000 hbase-master then i got some thing on console like 60000/tcp closed Please help me, i dont understand what went wrong in my set up. On Thu, Dec 1, 2011 at 11:28 PM, Jean-Daniel Cryans <jdcry...@apache.org>wrote: > So since I don't see the rest of the log I'll have to assume that the > region server was never able to connect to the master. Connection > refused could be a firewall, start the master and then try to telnet > from the other machines to master:60000. > > J-D > > On Thu, Dec 1, 2011 at 6:45 AM, Vamshi Krishna <vamshi2...@gmail.com> > wrote: > > I found in the logs of region server machines, i found this error (on > both > > regionserver machines) > > > > 2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at > > hbase-master/10.0.1.54:60020 could not be reached after 1 tries, giving > up. > > *2011-11-30 14:44:37,762* WARN > > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to > > master. Retrying. Error was: > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) > > at > > > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > > at > > > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328) > > at > > > org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883) > > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) > > at > > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) > > at $Proxy5.getProtocolVersion(Unknown Source) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) > > at > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572) > > at java.lang.Thread.run(Thread.java:662) > > 2011-11-30 14:44:40,768 INFO > > org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to > > Master server at hbase-master:60000 > > *2011-11-30 14:45:40,847* WARN > > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to > > master. Retrying. Error was: > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) > > at > > > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) > > at > > > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328) > > at > > > org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883) > > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) > > at > > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) > > at $Proxy5.getProtocolVersion(Unknown Source) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) > > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) > > at > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499) > > at > > > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572) > > at java.lang.Thread.run(Thread.java:662) > > > > > > and the same error is observed in the whole log repeatedly. After seeing > it > > what i understand is that some how master started HRegionServer daemons > on > > the machines but from then onwards the RegionServer machines are not able > > to communicate with master. If we observe it is trying to communicate > with > > master for evry one minute. > > > > But i am not understanding where to check and modify the things.. please > > help. i feel all connections are OK. > > > > On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <jdcry...@apache.org > >wrote: > > > >> stop-hbase.sh only tells the master to stop, which in turn will tell > >> the region servers to stop. If they are still running, it might be > >> because of an error. Look at their logs to figure what's going on. > >> > >> J-D > >> > >> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <vamshi2...@gmail.com> > >> wrote: > >> > hey soryy for posting multiple times. > >> > J-D, As you said, i refered to my regionserver log, there i found > >> > Could not resolve the DNS name of vamshikrishna-desktop > >> > so i added an alias ' vamshikrishna-desktop ' to its corresponding IP > >> > address in /etc/hosts. So, from then master is able to run > HRegionServer > >> > daemon in the regionserver machines also. > >> > > >> > But the ONLY problem now is when i stop hbase on my master node by > >> running > >> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node but > NOT > >> on > >> > regionserver nodes.The HRegionServer daemon is still running on the > other > >> > regionserver machines. > >> > I think the HRegionServer daemons on all the regionserver machines > should > >> > also get stopped, when we execute bin/sto-hbase.sh on master node, > right? > >> > please help.. > >> > > >> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans < > >> jdcry...@apache.org>wrote: > >> > > >> >> You posted this in two threads, please refrain from doing this in the > >> >> future. > >> >> > >> >> From what I can read in there, it tried to get the master address > >> >> that's supposed to be in zookeeper but it failed because it was > >> >> missing and then died. The way it's handled is a bit ugly but the > >> >> effect is the same, take a look at the master log to see what's going > >> >> on. > >> >> > >> >> J-D > >> >> > >> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna < > vamshi2...@gmail.com> > >> >> wrote: > >> >> > Hi Lars, > >> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines. > >> >> > Finally that problem i mentioned got resolved i.e now i can see the > >> >> > following after i run bin/start-hbase.sh on my master machine, > >> >> > > >> >> > hbase-master: starting zookeeper, logging to > >> >> > > >> >> > >> > /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out > >> >> > starting master, logging to > >> >> > > >> >> > >> > /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out > >> >> > hbase-regionserver2: starting regionserver, logging to > >> >> > > >> >> > >> > /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out > >> >> > hbase-regionserver1: starting regionserver, logging to > >> >> > > >> >> > >> > /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out > >> >> > > >> >> > hbase-master: starting regionserver, logging to > >> >> > > >> >> > >> > /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out > >> >> > > >> >> > > >> >> > > >> >> > But still , if i do jps on both region server machines, > HRegionServer > >> >> > daemon is not seen.And i checked the logs on both the regionservere > >> >> > machines, i.e > >> >> > > >> >> > > >> >> > >> > /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log > >> >> > ---(hbase-regionserver1) > >> >> > and > >> >> > > >> >> > >> > /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log > >> >> > ---(hbase-regionserver2) > >> >> > > >> >> > i found the same in both .it is like this, (here > >> vamshikrishna-desktop > >> >> is > >> >> > the master and namenode machine) > >> >> > > >> >> > > >> >> > > >> >> > 2011-11-29 11:24:36,714 ERROR > org.apache.hadoop.hbase.HServerAddress: > >> >> Could > >> >> > not resolve the DNS name of vamshikrishna-desktop > >> >> > 2011-11-29 11:24:36,716 FATAL > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region > >> >> server > >> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0, > >> >> regions=0, > >> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null > >> >> > java.lang.IllegalArgumentException: hostname can't be null > >> >> > at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108) > >> >> > at > >> >> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572) > >> >> > at java.lang.Thread.run(Thread.java:662) > >> >> > 2011-11-29 11:24:36,719 INFO > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: > Unhandled > >> >> > exception: hostname can't be null > >> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer: > >> Stopping > >> >> > server on 60020 > >> >> > 2011-11-29 11:24:36,749 INFO > >> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating > >> LruBlockCache > >> >> > with maximum size 199.4m > >> >> > 2011-11-29 11:24:36,775 INFO > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server > >> at: > >> >> > vamshi-laptop,60020,1322546076276 > >> >> > 2011-11-29 11:24:36,775 DEBUG > >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog > >> tracker > >> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef > >> >> > 2011-11-29 11:24:36,776 INFO > >> org.apache.hadoop.hbase.regionserver.Leases: > >> >> > regionserver60020 closing leases > >> >> > 2011-11-29 11:24:36,776 INFO > >> org.apache.hadoop.hbase.regionserver.Leases: > >> >> > regionserver60020 closed leases > >> >> > 2011-11-29 11:24:36,776 INFO > >> >> > > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > >> >> > Closed zookeeper sessionid=0x133ede31f3c0004 > >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn: > >> EventThread > >> >> > shut down > >> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: > Session: > >> >> > 0x133ede31f3c0004 closed > >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn: > >> EventThread > >> >> > shut down > >> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: > Session: > >> >> > 0x133ede31f3c0005 closed > >> >> > 2011-11-29 11:24:36,792 INFO > >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook > >> >> starting; > >> >> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main] > >> >> > 2011-11-29 11:24:36,792 INFO > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: > Shutdown > >> >> hook > >> >> > 2011-11-29 11:24:36,792 INFO > >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs > >> shutdown > >> >> > hook thread. > >> >> > 2011-11-29 11:24:36,792 INFO > >> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook > >> >> finished. > >> >> > > >> >> > what is the cause for problem here, please help. > >> >> > > >> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George < > lars.geo...@gmail.com> > >> >> wrote: > >> >> > > >> >> >> Hi, > >> >> >> > >> >> >> Did you add the list of servers to the regionservers file in the > >> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your > >> >> >> environment? > >> >> >> > >> >> >> Lars > >> >> >> > >> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote: > >> >> >> > >> >> >> > Hi i am running hbase on 3 machines, on one node master and > >> >> regionserver, > >> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh on > >> master, > >> >> all > >> >> >> > the hbase daemons running on master node(HMaster, HQuorumpeer, > >> >> >> > HRegionServer), But on the other regionserver machines i could > not > >> >> find > >> >> >> any > >> >> >> > hbase daemons running. According to my knowledge the > HRegionserver > >> >> >> daemons > >> >> >> > should run on other machines automatically when i run > >> >> bin/start-hbase.sh > >> >> >> on > >> >> >> > master node, right? But its not happening so. > >> >> >> > More over its printing errors on my master node when i run > >> >> >> > bin/start-hbase.sh > >> >> >> > > >> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No > such > >> >> file > >> >> >> or > >> >> >> > directory > >> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No > >> such > >> >> file > >> >> >> > or directory > >> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No > such > >> >> file > >> >> >> or > >> >> >> > directory > >> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No > >> such > >> >> file > >> >> >> > or directory , > >> >> >> > > >> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both > >> >> machines. > >> >> >> > Infact the path of the {HBASE_HOME} folder on each of the > >> respective > >> >> >> > machines machines is same. i.e > >> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4 > >> >> >> > > >> >> >> > please can anybody help? > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > *Regards* > >> >> >> > * > >> >> >> > Vamshi Krishna > >> >> >> > * > >> >> >> > >> >> >> > >> >> > > >> >> > > >> >> > -- > >> >> > *Regards* > >> >> > * > >> >> > Vamshi Krishna > >> >> > * > >> >> > > >> >> > >> > > >> > > >> > > >> > -- > >> > *Regards* > >> > * > >> > Vamshi Krishna > >> > * > >> > > > > > > > > -- > > *Regards* > > * > > Vamshi Krishna > > * > -- *Regards* * Vamshi Krishna *
