Thankyou suraj, beacuase of discussing on that issue with you, i came to
know many other things also which i need to take care of during hbase
setup. Finally that problem i mentioned got resolved i.e now i can see the
following after i run bin/start-hbase.sh on my master machine,
hbase-master: starting zookeeper, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
starting master, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
hbase-regionserver2: starting regionserver, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
hbase-regionserver1: starting regionserver, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
hbase-master: starting regionserver, logging to
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
But still , if i do jps on both region server machines, HRegionServer
daemon is not seen.And i checked the logs on both the regionservere
machines, i.e
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
---(hbase-regionserver1)
and
/home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
---(hbase-regionserver2)
i found the same in both .it is like this, (here vamshikrishna-desktop is
the master and namenode machine)
2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress: Could
not resolve the DNS name of vamshikrishna-desktop
2011-11-29 11:24:36,716 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
serverName=vamshi-laptop,60020,1322546076276, load=(requests=0, regions=0,
usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
java.lang.IllegalArgumentException: hostname can't be null
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
at
org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
at
org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:662)
2011-11-29 11:24:36,719 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
exception: hostname can't be null
2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2011-11-29 11:24:36,749 INFO
org.apache.hadoop.hbase.regionserver.StoreFile: Allocating LruBlockCache
with maximum size 199.4m
2011-11-29 11:24:36,775 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
vamshi-laptop,60020,1322546076276
2011-11-29 11:24:36,775 DEBUG
org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
regionserver60020 closing leases
2011-11-29 11:24:36,776 INFO org.apache.hadoop.hbase.regionserver.Leases:
regionserver60020 closed leases
2011-11-29 11:24:36,776 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x133ede31f3c0004
2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
0x133ede31f3c0004 closed
2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
0x133ede31f3c0005 closed
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown
hook thread.
2011-11-29 11:24:36,792 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.
what is the cause for problem here, please help.
On Mon, Nov 28, 2011 at 9:00 PM, Suraj Varma <svarma...@gmail.com> wrote:
> Ok.
>
> Can you run dos2unix against both your HBASE_HOME/bin and
> HBASE_HOME/conf directory?
>
> After this, restart your cluster and see if you are getting the same issue.
> --Suraj
>
>
> On Sun, Nov 27, 2011 at 10:58 PM, Vamshi Krishna <vamshi2...@gmail.com>
> wrote:
> > Hi,
> > 1)No, hbase is running as same user i.e hduser, in all machines i have
> > hbase in hduser account only.
> >
> > 2)Yes, iset HBASE_HOME environmental variable in all machines under
> hduser
> > , in the file /.bashrc like
> > export HBASE_HOME=/home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
> > And, in all the machines, hbase is located at the same place, i.e if i
> run
> > command echo $HBASE_HOME in all machines separatley ,under hduser
> > account, three machines are printing
> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4 , So, i hope
> > HBASE_HOME is set in all machines, right?
> >
> > 3) i didn't understand " If a new bash env is created, will that env get
> > inherited?" statement. I didn't create any new bash env.
> >
> > 4) i changed all the machines to wired network only. Now all machines are
> > connected with wired connection.
> > 5)hbase is running fine in standalone mode.
> >
> > Hadoop is running all machines perfectly, i ran some example(word
> count) ,
> > its running fine on all machines, in distributed mode.
> > Hbase is perfectly running on master node ONLY.
> > This is my present situation, i am feeling like just struck with this
> > problem, please help.
> >
> >
> > On Mon, Nov 28, 2011 at 2:04 AM, Suraj Varma <svarma...@gmail.com>
> wrote:
> >
> >> So - first of all:
> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..:
> >> > No such file or
> >> > directory
> >>
> >> This indicates that the environment of the process running
> >> hbase-regionserver2 is not set correctly. It is not able to find the
> >> directory.
> >>
> >> Is hbase running as a different user?
> >>
> >> I think your first step should be to get the above "No such file or
> >> directory" errors resolved. Did you set HBASE_HOME in the environment
> >> manually? If a new bash env is created, will that env get inherited?
> >>
> >> So - start simple steps first - get hbase running stand alone, then
> >> move to pseudo / fully distributed.
> >>
> >> By introducing variables like wired/wireless etc in your network, you
> >> are introducing uncertainities which will bite you on an ongoing basis
> >> (wireless going down, increased latencies, etc, etc). So - the cluster
> >> is unlikely to be stable with such a setup.
> >>
> >> I would recommend going with a fully wired setup, if your goal is to
> >> have a stable hbase cluster. If it is a "at home test cluster", then
> >> that's fine - but be prepared for frequent cluster reboots. :)
> >>
> >> --Suraj
> >>
> >>
> >> On Sat, Nov 26, 2011 at 11:20 PM, Vamshi Krishna <vamshi2...@gmail.com>
> >> wrote:
> >> > Hi suraj,
> >> > Thank you,
> >> > i am using ubuntu-10.04 OS. actually i am using 3 machines out of
> which
> >> > two are desktops connected with wired connection(10.0.1.54 and
> >> 10.0.1.97),
> >> > other is laptop(10.0.1.55) connected with wireless router. i didnot
> >> modify
> >> > any of such interface settings in hbase-site.xml file.
> >> > Would this sort of connecting nodes (i.e using both wired and wireless
> >> > connections) create any such problem as i mentioned? If so, atleast
> two
> >> > desktop machines are connected with same wired connection, then also
> it
> >> > threw an error, like
> >> >
> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..: No such
> file
> >> or
> >> > directory
> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such
> >> > file or directory
> >> >
> >> > i will change hbase-site.xml as you have suggested and let you know
> what
> >> is
> >> > happening. But before that answer my above query please.
> >> >
> >> > One more thing is, i am running region servers on the datanodes and
> >> > master only, then i hope it would become co-located, right?
> >> >
> >> > Thank you
> >> > On Sun, Nov 27, 2011 at 2:05 AM, Suraj Varma <svarma...@gmail.com>
> >> wrote:
> >> >
> >> >> Vamshi:
> >> >> What OS are you trying this on? Is it Linux / Windows?
> >> >>
> >> >> You can use the *.dns.interface configurations to use the specific
> >> >> network interface that you want.
> >> >>
> >> >> That is, set the following configurations in your hbase-site.xml on
> >> >> all hbase nodes
> >> >> hbase.zookeeper.dns.interface
> >> >> hbase.regionserver.dns.interface
> >> >> hbase.master.dns.interface
> >> >>
> >> >> You can set it to lo or eth0 or eth1 per your environment.
> >> >>
> >> >> Restart all nodes after this and you should see them bind to the
> >> >> specified interface.
> >> >> You can do the same for hadoop by setting dfs.datanode.dns.interface
> >> >> in hdfs-site.xml for hadoop nodes.
> >> >>
> >> >> On Windows cygwin, I've found setting that it to "localhost" works.
> >> >>
> >> >> Your setup should work for test environments ... for production, the
> >> >> standard setup would be to co-locate region servers and data nodes to
> >> >> get data locality.
> >> >> --Suraj
> >> >>
> >> >> On Thu, Nov 24, 2011 at 10:51 PM, Vamshi Krishna <
> vamshi2...@gmail.com>
> >> >> wrote:
> >> >> > Hi, i checked by executing netstat -anp, i found
> >> >> > tcp 0 0 127.0.1.1:60000 0.0.0.0:*
> >> >> > LISTEN 6814/java
> >> >> > .
> >> >> > ***some other ports**
> >> >> > tcp 0 0 0.0.0.0:60010 0.0.0.0:*
> >> >> > LISTEN 6814/java .
> >> >> > ***some other ports**
> >> >> > .
> >> >> > tcp 0 0 10.0.1.54:44095 10.0.1.54:9000
> >> >> > ESTABLISHED 6814/java
> >> >> > ***some other ports**
> >> >> > tcp 0 0 10.0.1.54:55204 10.0.1.54:2222
> >> >> > ESTABLISHED 6814/java
> >> >> > ***some other ports**
> >> >> > ***some other ports**
> >> >> >
> >> >> > the port of HMaster i found when i ran jps is 6814. But the same
> port
> >> i
> >> >> > found in different places bound to different IPs (i.e some time
> >> >> 127.0.1.1 ,
> >> >> > and some times 10.0.1.54). Same is the case with all other daemons
> >> which
> >> >> > are running after i ran bin/start-hbase.sh
> >> >> > Does it cause any abnormal behavior if i have like above. All
> hadoop
> >> >> > daemons are runnong on other machines, but hbase daemon i.e
> >> HRegionServer
> >> >> > is not running on other machines. HMaster, HQuorumPeer,
> HRegionServer
> >> are
> >> >> > running on master node.
> >> >> > My /etc/hosts file has content like
> >> >> >
> >> >> > 127.0.0.1 localhost
> >> >> > 127.0.1.1 vamshikrishna-desktop
> >> >> > 10.0.1.54 hbase-master hadoop-namenode
> >> >> > 10.0.1.55 hbase-regionserver1 hadoop-datanode1
> >> >> >
> >> >> > Still what more modification i need to do to bind the hostnames to
> >> IPs?
> >> >> >
> >> >> >
> >> >> > Please some body help..
> >> >> >
> >> >> >
> >> >> > On Mon, Nov 21, 2011 at 7:42 PM, Joey Echeverria <
> j...@cloudera.com>
> >> >> wrote:
> >> >> >
> >> >> >> I suspect that the daemon's are binding to 127.0.0.1 instead of
> >> >> >> 10.0.1.54 on the master. You can check this by running netstat
> -anp.
> >> I
> >> >> >> would recommend either configuring the /etc/hosts to bind the
> >> >> >> vamshikrishna-desktop and vamshi-laptop hostnames to the 10.0.1.x
> >> >> >> addresses.
> >> >> >>
> >> >> >> -Joey
> >> >> >>
> >> >> >> On Thu, Nov 17, 2011 at 1:40 AM, Vamshi Krishna <
> >> vamshi2...@gmail.com>
> >> >> >> wrote:
> >> >> >> > hi
> >> >> >> > i am working with 2 node hbase cluster as shown below
> >> >> >> > On node1 (10.0.1.54) : master node, region server, hadoop
> >> namenode,
> >> >> >> hadoop
> >> >> >> > datanode
> >> >> >> > on node2 (10.0.1.55): region server, hadoop datanode.
> >> >> >> > When i start both hadoop then hbase, all daemons are running
> >> properly
> >> >> on
> >> >> >> > masternode i.e node1,
> >> >> >> > 2404 NameNode
> >> >> >> > 3657
> >> >> >> > 3007 TaskTracker
> >> >> >> > 2848 JobTracker
> >> >> >> > 3522 HRegionServer
> >> >> >> > 3848 Main
> >> >> >> > 3292 HQuorumPeer
> >> >> >> > 2769 SecondaryNameNode
> >> >> >> > 3345 HMaster
> >> >> >> > 2575 DataNode
> >> >> >> > 4768 Jps
> >> >> >> > but on node2, only TaskTracker and Datanode daemons are
> running,
> >> the
> >> >> >> > HRegionServer daemon is not running.
> >> >> >> > I provide some files which may be helpful in helping me. i have
> my
> >> >> >> > /etc/hosts file of node1 (vamshikrishna-desktop) has
> >> >> >> > # /etc/hosts (for master AND slave)
> >> >> >> >
> >> >> >> > 127.0.0.1 localhost
> >> >> >> > 127.0.1.1 vamshikrishna-desktop
> >> >> >> > 10.0.1.54 hbase-master hadoop-namenode
> >> >> >> > 10.0.1.55 hbase-regionserver1 hadoop-datanode1
> >> >> >> >
> >> >> >> > and file {HBASE_HOME}/conf/regionservers has
> >> >> >> >
> >> >> >> > hbase-regionserver1
> >> >> >> > hbase-master
> >> >> >> >
> >> >> >> > i have my /etc/hosts file of node2 (vamshikrishna-laptop) has
> >> >> >> > 127.0.0.1 localhost
> >> >> >> > 127.0.0.1 vamshi-laptop
> >> >> >> > 10.0.1.54 hbase-master hadoop-namenode
> >> >> >> > 10.0.1.55 hbase-regionserver1 hadoop-datanode1
> >> >> >> >
> >> >> >> > and file {HBASE_HOME}/conf/regionservers has
> >> >> >> > localhost
> >> >> >> >
> >> >> >> > Please can anybody help me in finding out what is the reason for
> >> >> >> > Regionserver not running on node2..? And moreover when i run
> >> >> >> > start-hbase.sh , it will display error like
> >> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..: No
> such
> >> >> file
> >> >> >> or
> >> >> >> > directory
> >> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No
> >> such
> >> >> file
> >> >> >> > or directory , but i could find out
> >> {HBASE_HOME}/bin/hbase-daemon.sh
> >> >> >> > clearly.. i don't know what went wrong..!
> >> >> >> > please can any body help?
> >> >> >> >
> >> >> >> > --
> >> >> >> > *Regards*
> >> >> >> > *
> >> >> >> > Vamshi Krishna
> >> >> >> > *
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Joseph Echeverria
> >> >> >> Cloudera, Inc.
> >> >> >> 443.305.9434
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > *Regards*
> >> >> > *
> >> >> > Vamshi Krishna
> >> >> > *
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > *Regards*
> >> > *
> >> > Vamshi Krishna
> >> > *
> >> >
> >>
> >
> >
> >
> > --
> > *Regards*
> > *
> > Vamshi Krishna
> > *
> >
>
--
*Regards*
*
Vamshi Krishna
*
